Support Gem Limits for Weapon swap

So the problem I've run into is that I weapon swap for bossing but I also want to use the same supports that are in my other skill which cannot be used with my other weapon i.e. in my case I use unarmed shattering palm for my clear and rapid assault for bossing but I cannot use common supports even tho the other skill becomes unusable when I swap, kinda moots the point of having a swap in the first place