Weapon Set passives that I refunded are now missing unable to be used

I just refunded 4 passive points from weapon set one and they have disappeared. I am unable to use them and it shows 0 skill points available. Weapon set two shows 9 out of 24 used, but weapon set one shows 5 out of 24 used and I am unable to allocate those 4 refunded at all. I've tried exiting the game and that did not fix it.

Has anyone else experienced this and know how to fix it? I tried refunding all weapon set points available and it works for weapon set two, but it shows only 20 out of 24 available for weapon set one as those initial 4 are nowhere to be found.