Problem with "The Coming Calamity Hero Armor"

When I insert a support gem into the herald and then press "esc" and exit to hero selection, and then return to the character, the support gems I inserted disappear (this also happens when I enter a different location). Mostly, the gems that disappear are those that don't use spirit. If I wait long enough and insert them constantly, they eventually stop disappearing. I have to carry multiple copies of the gems I use and constantly insert them because they keep disappearing.