Can't access Aggorat for Sacrificial Heart/Dagger

After finishing the act 3 time-travel section, there is no way (at least not obvious) to get back to Aggorat. This means one can completely miss the 2 passive skill points from that area. It is also possible to end up with the heart and be stuck with it without being able to use it.