[Poe2] The shader caching makes it unplayable

First of all, my rig specs:



Now, here's the thing. I can run Poe2 at 120fps stable on my first map. Technically even higher, as my refresh rate goes up to 165, but 120 is the general cap I put on all games just cause. I load unto a map, I see the shader graph go up for around thirty seconds causing the fps drop to around 80~, and once it's done it's back to 110-120 and I enjoy the first map.

Then I load onto another map. Shading graph goes up, fps goes down to 60 for a little while, and then back up to 80.
One more map: shader goes up, fps drops to 40, once its done then back to 60.

You get the idea, the progression keeps going lower and lower until it stabilizes at 40fps once a map has finished loading the shader cache. Just to give you an idea: I load onto a map, I wait for 30 sec to a minute at 20fps before I can finally play at around 40fps.

And don't get me started on city maps, they skip the whole process right away, I go from 120 stable on the previous map directly to the 40.

I have tried:
- Fiddling with graphic settings: everything on low, DLSS performance, windowed/fullscreen, reflex on/off etc.
- Updating graphics driver
- Updating windows
- Updating BIOS firmware(!!!)
- Setting Poe2 exe as high priority, high performance enabled, using only GPU etc.
- Deleting cache folders in AppData folder
- Setting shader cache size to 100GB (or unlimited) in Nvidia Control Panel
- Uninstalling the game and redownloading it from scratch, then verifying the files

None of it has made a difference.
The weird thing is that it ran at 60fps stable on my previous laptop, which definitely had MUCH lower specs than this one.
I honestly don't know what else to try, please help.

Important note: Once it reaches that "overloaded" state of general 40fps play, it matters literally nothing what I do with the graphic settings. I usually play with literally everything as low as it gets and DLSS on Performance. Lowering it even more, or changing the resolution to something lower makes no difference to the performance, which is something I've never seen in ANY game ever. SURELY playing it at 1 freaking pixel by 2 pixels should make the frames go up, but not POE2, nah. :)

I have this issue on Intel Arc iGPU...