Game crashes when other players join instance sometimes.

My game crashes frequently in towns, or when trading, both going to other hideouts and other players coming to my hideout. I get a vk:Queue:Submit:devicelost error message. Sometimes when I load the game I can complete the trade, other times I crash over and over every time I am in the instance with the player. Twice I have been able to have the other player remove their MTX and after that they can load in fine and I can trade. This only happens when using vulkan renderer, dx11 does not crash but is unplayable due to lag in maps. Vulkan never crashes in maps, only when around other players. I am using Linux and playing through steam.