Companions won't stay active between weapon sets

As stated in my title, when I swap weapon sets, one of my companions dissappears. I have the passive node that lets you have 2 companions, and i only have 2 active. For some reason, this issue stops happening when I turn off my other spirit reservation skills, but the game shows that I have enough spirit to have them all active at once, and lets me turn them all on. But then when I use abilities on different weapon sets, one of my companions dissappears. What's happening???

Extra info: I'm a level 52 Huntress (spirit walker)
100 dex
72 int
66 str
None of my weapons or offhands give attributes (int/dex/str)
I'm under the limit for support gems
I have 60 spirit, 160 with weapon set 1 (100 from scepter)
Weapon set 1: spear/scepter
Weapon set 2: bow/quiver
Companions: Wild Protector, Mighty Silverfist, Diretusk Boar (going to replace with the noxious behemoth when it's bug stopping it from being tamed gets fixed)
My spirit abilities:
Attrition - 38 spirit
Silverfist - 51 spirit
Boar - 35 spirit
Caltrops - 38 spirit
Malice - 0 spirit (i have vitality and clarity slotted but it isn't taking the cost from my spirit pool?)
Wild Protector - 0 spirit (ascendancy)