Unequipping support gems from wand triggers silver charm?

pretty mundane, but i was testing out support gems in my hideout. whenever i unequip any support gem from my chaos bolt skill granted by withered wand, my silver charm procs. i tried with 3 or 4 different gems refilling the charm at the well and it seems to be any support gem being unsocketed. doesn't happen with any other skills, can't see a reason it would cause a slow and cause the charm to proc.