Portal to town from Manor goes away

I am in the first act. I pick up everything cause I need mats. When I port to town the port so far has happened twice it goes away and I have to start from floor one. It seems the timer for the manor is total on all three floors not from when you port out. Because I collect and sell I am finding it difficult to complete this.