Animate Weapon of Self Reflection bugged with specters

When using Animate Weapon of Self Reflection and having weapons summoned, entering a leaving instances (lab, mostly Mirage) reduces you specter from 3 to 2 and leaves one weapon summoned, instead of leaving 0 summons and all you specters. I am also using energy blade, don't know if that might cause it.