Crazy latency spikes when joining an xbox friend in poe2

I literally cant play on the instances from my xbox friend, we are both on the same server and dont live that far apart, so that cant be it. When he joins my insance there is no problem, stable 20ms. But the second I enter his instance, my ping spikes from 20ms to 300ms and sometimes even 4k+ ms. Its almost random when it happens, sometimes I join and everythign is fine, then he opens the next portal and I cant play again cause 4k ms. Is there anything I can do?