Mercenary not joining me

I have a mercenary named Brakor who is level 84 and geared out from my main playthrough. I started a new character and at level 64 it let me invite him. I entered Lunaris temple and he does not join me. If I click the mercenary button it shows his inventory and stats pages but he is not in the dungeon with me. I clicked the Order: to me button and that didn't work either.