Using benches is broken as hell

I don't know how this wasn't fixed in the days following 0.4 launch, but since then interacting with benches or some other interactable objects is extremely broken.
2 times out of 3 my characters will use skills bound to left click instead of using the bench. This is obviously very annoying. So please fix it.
Thanks.

Any fix incoming ?
This issue also affects picking loot up and clicking on shrines.