Banishing totems does not work anymore on controller

As title says, there is no way you can banish totems on controller.

Before today's patch you could press A standing nearby totem and it would start banishing totems. It does not work this way any more. I have tried multiple different buttons and nothing seems to be working