0.2.0g Incident Report

Downtime

Today we experienced around 5 hours of realm downtime for Path of Exile 2. This was caused by several overlapping factors and we will be making changes in the future to attempt to mitigate these issues.

Incident

After deploying 0.2.0g we had reports of people's Skill Gems being deleted on login. We immediately shut the realm down to prevent further damage to people's skills and attempted to roll back the database.

Unfortunately, due to a change in configuration between the databases for Path of Exile 1 and Path of Exile 2, we discovered that the rollback procedure was going to take longer than 24 hours to complete.

In order to reduce the time, we decided to take our most recent database snapshot and roll it forwards to the correct time. This process still takes a lot of time, but was still going to be significantly shorter.

Unfortunately, the process failed on our Ladder database for reasons that we have not fully investigated yet. We made multiple attempts at restoration of this database which ended up taking a significant amount of time, but eventually decided to use the old snapshot as is.

This means that the ladder will be 3.5 hours out of date, but because characters will regain their position on the ladder as soon as they gain experience, we decided that this was better than continuing to have downtime.

Causes

The initial reason that Skill Gems were being deleted is due to an accidental change where a Type ID that is stored in the database for a particular support gem was modified. When any skill with this support gem socketed into it was loaded, the item failed to load and the gem was deleted.

There are multiple ways in which this kind of problem is normally caught, but they all failed in this case.

Normally IDs of this type are marked as being unable to be changed, but this particular ID was not marked in such a way. This will be fixed going forwards.

The next is that normally, before deploying any significant patch, we run an "Item Destruction Test" which loads every item from the database to make sure that they are still valid. Unfortunately since these take a long time to run we usually only run these for patches with new content, and not for lettered patches which are generally small. We will be updating our policy on which patches need to have this test run.

Our rollback process also effectively failed due to taking an incredibly long amount of time. We will be changing the database configuration back to make rollbacks faster.

In addition, we would like to improve our deployment process to have a ready-to-go copy of the database snapshotted from after the realm is shut down, but before deploying the next version. This change will take some time to implement but should mean that we can rollback with more confidence in the future.

Conclusion

Obviously this incident is unacceptable, and we would like to apologise for all the inconvenience caused. This isn't an acceptable level of service. We will be making improvements in response to this incident and hopefully will not suffer such a long downtime again.

We will be deploying 0.2.0g tomorrow (Friday NZT/Thursday PDT) with a fix to the initial problem.