Intermittently stops taking inputs from actions

Since the 0.3.1 patch I have an issue where the game stops taking inputs from actions for skills on the action bar. I can still move with WASD.
After 5-10 seconds the it starts working as normal again.
It's both inputs bound to the keyboard and the mouse that are affected.