As minion player its really hard to keep up with wisps

so i would like if they could not despawn while they wait for me to kill the WALL they put in my path . they want from me to kill the corrupted monster they choose so why are they ditching me ?
im not using any meta build i have self made lich minion/specter only build (with some curses as support) and while i do good dmg its not a fast build. it takes me at least 5 seconds or more if theres rare/magic monster in the way . thats enought for the wisp to start disapearing . im at t1-2 . how do i keep up as minion player ?