Xbox party problem

When trying to play with my S.O. we join game through xbox because I can't on the game as it says they are offline. Has for a while. Anyway we can join the game and be together where we are at the time but the moment we go to travel, whether it be through my portal or his or the waypoints it crashes the whole game out. Closes it out completely. Please fix he has been playing all day with another friend no problem then as soon as I get on to play with him it crashes.