Network spikes in random areas at random times

When entering certain locations, my network graph and ping will constantly spike up and down making it unplayable. This only started about 3 days ago. I tried a VPN and it only seemed worse on there.

It's repeatable by entering and leaving the same maps which either lag or work perfect.

CPU: Ryzen 5 5600x
RAM: 32gb 3600
GPU: RX 6900 XT
OS: Windows 10

Pastebin with requested logs from WinMTR as requested:
https://pastebin.com/V4PHCNk2