Capped around 60fps with vulkan and dx12, framedrops from 130 to 15 with dx11.

Recently got this problem where I get around 49-63fps on my 144Hz monitor while playing on vulkan or dx12, but if I go on dx11 I get 144fps but immidietly drop down to sub 30fps when I do a map.

I have a:
GTX 3060ti
32gb ram
Intel core i9-9900

I have checked my pc, bios, core temp, drivers, hz rate, gsync, vsync and nvidia control panel if it's fps capped. Only having this issue in Path of Exile.