Spirit gems locks us out of active gems

The number of spirit gems we can use is already limited by spirit. I would like them to not take gem slots. With this, we could use more active skills making the gameplay more dynamic. Combat more engaging.

I currently have more spirit than I can use without letting go of skills I need, and I would love to fit wind dancer in at least.
I currently need to use only one main damage skill and I can't take one or two alternate skills I can choose to use situationally. The one thing I liked the most about using chayula's darkness was being able to use more skills (zero spirit). Combat was dynamic, I would dodge away and throw storm waves, etc.

Wasn't the point in poe2 to use more than one main active skill and more than one combo, instead of repeating the same combination of keys the entire time? Spirit gems occupying gem slots removes that.

Maybe I'd go further and not limit gem slots at all. This would allow me to even use the second weapon set with something else, and perhaps actually make use of the alternating passive tree weapon points.