Spirit Issue on Weapon Swap

I'm running an Abyssal Lich with Umbral Well, to get buffs from minions.

I am attempting to having two different minion buff setups on weapon swap.

1. Reavers for Actions Speed
2. Clerics for ES Recovery

Weapon 1 is dual wielding two Mjolnir for my primary config. Including the 2x soul cores with extra +15spirit each, I have a total of 195 spirit. However,g the game will not allow me to use more than 35. It acts as if the rest is already reserved, despite nothing else reserving spirit.

Weapon 2 is just a scepter, leaving that weapon set with nearly 400 spirit. This weapon set allows me to reserve as many minions as my spirit can afford, as usual.

If I swap out the mjolnirs with two other maces, I can then assign spirit and summon the Reavers. If I then re-equip the mjolnirs, the minions will stay. However, the moment I weapon swap (X), they are bricked again and won't come back.

After a bunch of testing, I thought it was the Soul Cores of Azcapa that were the problem, but after testing with some other Mjolnirs that did not have the soul cores socketed, the problem persists. There is clearly some strange interaction with Mjolnir upon weapon swap. My guess is that it's somehow tied to Thundergod's Wrath, despite it not having a spirit reservation cost.