Scry -> Unscry -> Rescry to different map not moving all divination cards

I've had Ivory Temple scryed to Canyon for a while and I've been getting 1-2 sephirots per map.
Recently I scoured the scry, then scryed Ivory Temple to Crimson Township and have now gotten 0 sephirots running the same type of map for hours, while still getting close to the same amount of "the conduit" and "light and truth".

I think the specific interaction of scrying map A to map B, then unscrying that connection (in my case i used the scouring orb on map A, if that matters), then rescrying map A to map C causes map C to not get all the divination cards.