Skill binding MKB vs controller

Skill bar changes from one setup affects other steup (mouse+kb vs gamepad).
For example Currently assigning the skill bar on mouse+kb affects the skill bar of the controller. When changing assigning the skills in the controller setup it affects (kb/mouse).

This requires me to reassign my skills every single time I switch between the two.