Your CPU is simply old. Your Xeon CPU is basically equivalent of Intel Core 3000 series (ivy bridge).
While being server-grade, it is of course bit more powerful, on other side, it's target never was game related workloads so it might suffer there.
Additionally, old generation of Intel CPUs were suffering from CPU vulnerabilities (Spectre/Meltdown/L1TF/Retbleed...). Mitigations of these slowed some CPUs to crawl (depending on workload, the performance decrease was over 70%) - at least on Linux, but I imagine the same impact was on Windows as well. On linux these mitigations can be easily disabled, returning the CPUs to their former glory, but being the Linux guy, I am not sure if this can be done on Windows or how easy it is. The improvements you describe (better performance with lower cores and disabled HT) matches the post-mitigations symptoms.
Which brings us to OS. Windows 10 and 11 does not officially support your CPU. While it might work, some features and capabilities might be missing. The Windows server edition on other hand don't support running games (so while they might work, game optimizations present on desktop windows might be missing on server editions).
Moving forward, if you are somewhat familiar with Linux (and only if you are familiar), I would recommend your luck there as even modern Linux still supports your CPU and you can control the mitigations and PoE runs very well via Proton.
Can confirm that performance on mine old ivy bridge CPU is not affected by the mitigations in this particular case, sadly.
Yeah, ive tryed to disable as many mitigations a i can so far, and got no difference.
