Ports in Co-op won't let you enter

This has happened to me and my friend many times, it happens no matter who is running the maps where if one player is in and someone exits, the portals wont let the guy go through that is outside until everyone leaves inside the map, then once everyone is outside people can enter again

This causes wasted portals, especially if 3 players are inside a boss map and 1 person left to re-charge their flasks, causing an additional 2 portals to be wasted to let the 1 person enter again