Restoring minimised window - big lag with spectres in hideout

I've noticed this when comparing a necromancer vs non-summoner build. I'll have PoE in windowed mode, minimised for 30+ minutes. When restoring PoE to resume playing (or trade), the game freezes up and takes a while to recover due to all the minion effects that flood the screen in your hideout.

When you don't have any minions, the window restores instantly with no lag.