Ping spikes on PS5

I play the game on PS5 and have often been getting MASSIVE ping spikes while playing, the game would just freeze for 20 seconds or more, then fast-forwards to re-sync, then the same thing again 10-30 seconds later.

I'm using Wi-Fi because I have no LAN ports nearby to plug the console into. Before anyone says it will be fixed by using a LAN cable instead of Wi-Fi, I agree but it just isn't possible and I know Wi-Fi is not the issue here because I've played multiple games online prior to PoE2 on Wi-Fi and had no problems so I'm wondering is it a server issue or are there any particular settings I can change in my PS5's Wi-Fi advanced settings to make things better?

Anyone else having the same issues as well?