Spirit/minion count accounting is bugged?

The bug seems to trigger anytime you mess with your equipped gems, or your passive tree. Allocating minions causes other minions to disappear.

I've got an equipped sceptre for skeleton warriors as well as a gem for equipped skeleton clerics. I also have two persistent gems Ravenous Swarm and Withering. Periodically when I do one of the above the clerics will stop summoning and disappear from the list of minions being summoned. To get them back I can either lower the count of skeleton warriors by one or I can deallocate all the spirit. And then reallocate them.

The order you reallocate them in seems to matter. First the persistent buff or aura gems, then the cleric and finally the skeleton.

If you only lower the warrior count you end up with some spirit you can't allocate and lose some amount of minions you can summon. If you fully deallocate and then reallocate then you can use all of your spirit.

Feels like some count somewhere is getting miscalculated.