Crossbow ammunition losing hotbar/mouse bindings multiple times per map

I bind my ammunitions to my mouse buttons like anyone would and then drag weapon set 1/2's Crossbow Shot over each R/L mouse button in the hotbar so that each button uses different ammunition.

Then multiple times per map it loses the binding randomly during a fight, and suddenly one of my buttons is shooting regular Crossbow Shots again without the ammunition.

The fix is really annoying... I need to press G, remove the gem and replace it, drag the skill back onto my hotbar, click the mouse button to load some ammunition, scroll up and drag either set1 or set2's crossbow shot over the hotbar's skill to bind it again.

Over and over and over and over and over again. I've probably done this 200 times now, and it happens mid-fight, super annoying! :)