Stuck on load screen on certain maps (PS5)

Issue happens mostly when I play with a friend who is playing on a PC.

It seems to happen mostly on the following campaign maps:
Jungle Ruins
Infested Barrens
Temple of Chaos (first floor/level)

For end game maps it seems to happen on Stone Citadel

It happens most of the time if my friend playing on a PC enters the map first and thus creating the instance.
If I enter first, the issue does not seem to happen at all.

In order tog et it to load, I have to do the following:
1. Ask my friend to exit the map
2. Close the game
3. Load a different character -> enter a map -> go back to town
4. Load the character stuck on load screen

If the above does not work, I have to load a couple different games on the PS5 and the load POE2