Game controller - Can't interact with Verisium

I'm playing on a Steamdeck using the controller support. You have to interact with the Verisium with the green - A button but it doesn't work. It shows you the A button is activated like it does for anything else you can interact with in the game, but it doesn't actually charge up the Verisium, so there's no way to mine it with the controller.