skill gem support requirements POE2

I dont really know how to word it properly so im just going to explain my specific situation that I just encountered. I dont believe it should work this way maybe it was coded this way for a purpose, but I dont see why. Im a spark sorc with my second weapon slot using a chiming staff for sigil of power. Im short on dex points for dex support gems so I went to allocate more passive tree nodes to dex and came to realize that you cant apply Set 1 or 2 passive points to apply to this. It only applies main set points to gem requirements. I dont see why, if you have a skill that ONLY applies to a certain weapon set, that those set points cant apply to the gem point requirements. Im assuming no single set points apply at all.