Mirage portal not allowing exit

I have had this happen twice so far. Sometimes the Mirage portal spawns near a doorway/archway. While I can get into the portal fine, this causes the portal to not work as an exit while inside of the mirage. Exiting the game gets me back into the normal world.