Stuttering in POE 2 fix

After tons of trial and error over 2-3 days, I've discovered that disabling shader cache in nvidia control panel will remove any stuttering in POE 2. I've done this on 2 separate computers (one laptop rtx 4060 and horrible cpu, one stationary) on 2 separate accounts and it removed all stutter which is known to appear in late game waypoints.

I have tried nvidia shader cache set to 100 GB
I've also set it to 10 GB, I've tried driver defaults
All of the aforementioned options result in stutter except the only thing that works
Shader Cache Size: DISABLED

My nvidia control panel settings for POE2:
Low latency mode: Ultra
G-SYNC
Power management mode: Prefer maximum performance
Shader Cache Size: Disabled
Texture filtering - Quality: High Performance
Everything else is set to default.

I have uninstalled GPU drivers, reinstalled debloated version with NV clean installer
Disabled CPU C states in BIOS
Updated drivers
Tried vsync, on, off, triple buffering, gsync
60 fps cap with nvidia control panel or rivatuner statistics server
Reinstalled windows 11, upgraded from windows 10 to 11
Deleted cache in appdata for both POE 2 and nvidia drivers
Tried vulkan, tried lowering all graphic settings, shutting down browsers etc to free up RAM, any and all optimization that directly affects the game basically

I have not tried getting results without first only deleting nvidia cache in %localappdata% and POE 2 cache also in %localappdata% (look through local, localrow, roaming) so please tell me if you do.
Poe 2 folder deleted in appdata/roaming/
Anything in local, localrow or roaming NVIDIA cache folders that end with toc, bin, skip those that can't be deleted.

Recommended to enable 60 fps cap through RivaTuner for smooth frametimes
There is very little stutter, that is barely if even noticable when not looking for it. My monitor is 120 hz with gsync enabled, also 4k with HDR for informations sake, HDR nor resolution seems to have an impact on stutter.

Current settings:
DX12 (default)
Windowed Fullscreen
Vsync OFF, gsync ON
Dynamic Res OFF
Windowed Res 3840x2160
NVIDIA DLSS, Quality
Sharpness: 50%
HDR ON

Texture Quality: High, 16x AF
Lightning:
Shadows + GI
Shadows + GI quality low (default)
Sun Shadow Quality: High
# of lights: High
Bloom 100%
Water: High

Nvidia Reflex OFF
No fps cap foreground or background
No triple buffering
No dynamic culling
Target framerate: 60
Engine multithreading ON

My PC spec:
RTX 3090
i9 10900k
16 gb DDR4 ram 19-19-19-39 4300ish mhz

Please tell me if this works for you

EDIT
Stuttering returned after 6 hours of mixed idling and playing waystones/sekhemas
Deleting POE 2 folder in appdata/roaming fixed the stuttering, I stil have shader cache disabled in nvidia control panel

Whatever the problem is, it's clearly related to loading cached data