Spirit walker can't summon wolf pack

I'm playing Spirit Walker and had been using a boss companion and wolf pack then I tried using a different non boss companion and now when I go back and try to turn Wolfpack back on it won't swap. even if I turn off wolf pack or remove the gem completely I still can't reactivate them. Now im stuck with my other companion