Possible Lich Ascendancy Passive bug

I'm on my lich and spec'd into the Eldritch empower node and was using it for a bit of time. Then I unspec'd it and it looks like maybe it's still acting as if i have it maybe? im not sure im losing hp and energy shield each time i cast a spell and i think its maybe a bug. I looked at my uniques and I dont have any keystones on the passive tree and I looked at my support gems. So im not really sure whats going on and I feel like im losing my mind trying to figure it out.