Unable to Interact with Death Crystals/Soul Cores in the Trials using Controller when enemies nearby

On controller, the interact button is tied to a button with a skill (typically the A button). If there are enemies nearby, I cannot disable the Death Crystals in the Trials of the Sekhemas, or pick up the Soul Cores in the Trials of Chaos.

For controller players, it would be nice if touching the Death Crystal would deactivate it, and touching the Soul Core would pick it up automatically.