Latency between instances very inconsistent

Going back and forth between hideout and map sometimes yields huge variances in how much latency exists... almost like each time I join an instance it just randomly selects a gateway for me or something.

Often times to alleviate the problem I have to port to hideout, then go back into the map (often have to try several times) to have latency return to normal.

BTW on console changing the gateway doesn't seem to allow you to save your settings which is disconcerting... as if the game doesn't recognize that a change was made.

Same Issue. Entering Temple of Kopec (Level 42) from Ziggurat Encampment Network goes from a consistent 1 or 2 ms to a widely fluctuating 5 - 580 ms.

Tried things that have worked in the past:
Left and reentered zone, problem not fixed, although it did improve from the original 200 - 5000 ms issue.
Exited to character selection, problem not fixed.

Tried things advised elsewhere:
Logged into different character, the logged back to this character.
Logged out completely and restarted the game.
Neither of these improved the situation.

Have seen this issue with other zones, normally leaving the zone and reentering it fixes the issue on first or second try.

Edit: Upon entering The Drowned City, network is 2 to 3 ms. I realize that these are different instances, and are obviously on different servers or going through different routing, but these two entrances are literally next to each other in the Ziggurat Encampment.