Logging out every time I portal to town and back (Xbox)

Every time I portal to town and try to go back, the game crashes and I get completely logged out. I've cleared my cache and even started a new character, but it's still happening. Yesterday it was occasional, but today it's literally every single time. The game has become unplayable.