Client-server Action Synchronisation

While I'm well late to the party here (and hence doubt few, if any, will even read what I write) my experiences with desync, both here and in other games, has led me to a few hypotheses as to WHY we have such a big issue here.

No, no game is immune to latency or desynchronization, (commonly known as "desync") but, I will say, having played a WIDE variety of online games, from RTSes, FPSes, MOBAs, ARPGs and other genres, I believe I've experienced more "rubber banding" events in PoE than I have in all other games I've played COMBINED. A conservative estimate is that I've probably played a few hundred online games, and totalled readily 10,000+ hours between them, spread over about the past 15 years.

Now, for the unitiated (who couldn't digest Chris's longer explanation) desync occurs when the server decides that the client's own idea of where the player is doesn't mesh up with what the server estimates is "correct." In other words, it believes, for some reason or other, the client program has it wrong, either by error or intent. (read: hacking)

In other games, the #1 issue which caused rubber-banding to be implemented is the worry of what was known as "speedhacking," where a player used something that modified the client, to essentially boost their speed, simply by having it tell the server where they were, and be consistent enough that the server would believe it. Rubberbanding is a "correction mechanism" for when the server decides that the player clearly shouldn't have been able to do what they just did.

Now, from the way Chriss wrote things, (and taking him as an authoritative voice on GGG's position) he appears to take a 100% hardline approach to preventing any possible form of speedhacking or other hacking. In PoE's case, it appears that for the most part, the server is designed to not trust the client at all; it is REPEATEDLY telling the client where they should be at all times... And if communications between the client and server get slightly disrupted, the positions the client and server are thinking of won't match because they missed a step... And hence a desync occurs.

There are a few solutions around this sort of thing... For one, I do think it speaks to a deeper issue as well: many are familiar with how readily PoE will "experience an unexpected disconnection." Long story short, the connection part of the game's code is far from robust, leading to smaller "disconnections" that lead up to causing desync events. As we've seen, this has often come just from one's Internet route (i.e, all the steps between a player's ISP and GGG's ISP, which make up the actual "Internet" part of the Internet) just so happens to have a link that is less than 100% stable. While yes, such connection issues can be solved without modifying the game on a case-by-case basis, the "nodes" of the Internet will never be all 100% stable: the entire design of the thing was to accept that as fact. If the game's code were improved to give an overall higher fault tolerance for this sort of thing and other connection hiccups, that would likely dramatically cut down on desync.

Another route, which appears to be found in comparing PoE to OTHER online games... Would be to potentially have the server be less hardline on preventing any sort of hacking at all. While, of course, relying entirely on the client to tell the truth isn't secure at all, a degree of lenience is possible that would still catch effectively all attempts at hacking... But stop also constantly catching up non-hackers. In other words, if the server allowed for a SLIGHT bit of a "fudge factor" when checking synchronization, it'd also cut down on desync heavily. And to be honest, it would simply be a matter of the game's code recognizing that it's an INTERNET game, not a LAN game; again, as I said above, players will NOT always have perfect connections without any variance in latency or packet loss.

"

ACGIFT wrote:

While I'm well late [...]

Another route, which appears to be found in comparing PoE to OTHER online games... Would be to potentially have the server be less hardline on preventing any sort of hacking at all. While, of course, relying entirely on the client to tell the truth isn't secure at all, a degree of lenience is possible that would still catch effectively all attempts at hacking... But stop also constantly catching up non-hackers. In other words, if the server allowed for a SLIGHT bit of a "fudge factor" when checking synchronization, it'd also cut down on desync heavily. And to be honest, it would simply be a matter of the game's code recognizing that it's an INTERNET game, not a LAN game; again, as I said above, players will NOT always have perfect connections without any variance in latency or packet loss.

Agreed! Maybe it's time to recognize that by "allowing" some minimal possibility of (speed)hacking a clean and fair (end)game experience is opened up to the vast majority of players that don't even desire to play foul. Wouldn't it be wonderful if the outcome of a fight is decided by the skills of the player instead of the performance of ISPs?

Rubber banding isn't the result of speed hacking though (It can be, if hacking, but isn't if not hacking), that's the result of the client updating to reflect what the server says is correct. In POE's case GGG have tried very hard to minimize the frequency of this because it is "confusing" to the player even though every other online game (aside from RTS games) has the client update to reflect what the server says is happening quite often and thus triggering a rubber band situation quite often.

RTS games work around it by essentially operating in a lock-step process, the client literally can't portray any actions to the end user unless the server authorizes the action first, there is no potential for a desync but there is a different downside to this process. This process isn't used in a lot of games because it results in gameplay being unresponsive to user input (You issue a command and after the length of your latency has passed by, your unit will finally respond to that command!).

So yeah, GGG basically took the same implementation as everyone else and greatly dialed back the frequency of the client resynchronizing with the server to minimize rubber banding. Since resynchronization happens much less often, it is extremely possible to go out of sync for extended periods of time (Most other games will exhibit frequent minor rubber banding due to frequent resynchronization and you probably never even notice it).

There is also video evidence of the client "logic" never matching the server logic. When the client synchronizes with the server it can immediately go out of sync again. For example: People have shown enemies coming towards them and then the enemy warps back to its original location when the client resynchronizing with the server, after resynchronization has completed the same enemy will immediately head towards the player again. It's a loop of the client never properly understanding the behavior of the enemy that the server is manipulating and consistantly portraying incorrect actions by the enemy.

"

Nicholas_Steel wrote:

Rubber banding isn't the result of speed hacking though (It can be, if hacking, but isn't if not hacking), that's the result of the client updating to reflect what the server says is correct. In POE's case GGG have tried very hard to minimize the frequency of this because it is "confusing" to the player even though every other online game (aside from RTS games) has the client update to reflect what the server says is happening quite often and thus triggering a rubber band situation quite often.

Actually, the entire reason rubber-banding exists is to prevent speedhacking, since the alternate approach is for the server to implicitly trust the client... Which a hacked client can abuse by simply changing its relative position updates unreasonably fast... Which is what speedhacking is.

"

Nicholas_Steel wrote:

RTS games work around it by essentially operating in a lock-step process, the client literally can't portray any actions to the end user unless the server authorizes the action first,

This is not QUITE true... My experience with RTS games has shown cases where, in fact, there is an allowance for some degree of latency (which can be adjusted) but the span it allows to go before full confirmation is usually far, far shorter.

"

Nicholas_Steel wrote:

So yeah, GGG basically took the same implementation as everyone else and greatly dialed back the frequency of the client resynchronizing with the server to minimize rubber banding. Since resynchronization happens much less often, it is extremely possible to go out of sync for extended periods of time (Most other games will exhibit frequent minor rubber banding due to frequent resynchronization and you probably never even notice it).

Actually, the reason other games appear to have less rubber-banding is not because they resynchronize more frequently... If anything, most other RPGs actually allow the client to get even MORE out-of-sync before snapping things back than PoE does.

Rather, it's a mix of reasons, some of which I outlined above, that result in the game being far more susceptible to this sort of instance occurring at all. You mentioned one such at the end, where the game's susceptible to never actually getting a proper resync, so that one desync event immediately triggers another, because the first resync directly caused a SECOND loss of sync.

Another I noticed is that, apparently, the server makes some shortcuts when it comes to trying to process how movement is handled... Or in one way or another, does NOT run the same calculations as the client does. One example (that I wish I'd been recording at the time!) was where I managed to desync and then be repositioned on the opposite side of a solid ledge. Apparently, whatever the server did, it somehow thought I'd crossed a barrier that, normally, would've taken me an extra 15-20 seconds to walk around. That tells me that the server's path-checking calculations were WAY off. And if it was off on one thing, it will be off on others.

Make no mistake, I'm not saying that GGG is being terrible or anything. But rather, I *AM* saying that PoE's synchronization issues are bad, and NOT something unavoidable from a design standpoint. While it's not possible to cut it down to absolutely ZERO issues, that's no excuse to not minimize them. If they could recognize publicly that it's a problem without excuses, and instead of trying to give explanations or shift blame to everyone else's ISP, simply focus on trying to solve most of it and make that their focus... I think around 99% of the complaints would go away even BEFORE the fixes arrived.

That, I think, is what is frustrating most people about this; to give an example, GGG's approach to the problem feels almost as bad as what Microsoft's original approach was to the infamous "Red Rings of Death" issue with the Xbox 360. Sure, end-user behavior had an impact on the error's occurrence, but that doesn't mean the end-user is the one at fault and the developer blameless, as we saw was the case with the RRoD.

"

ACGIFT wrote:

"

Nicholas_Steel wrote:

Rubber banding isn't the result of speed hacking though (It can be, if hacking, but isn't if not hacking), that's the result of the client updating to reflect what the server says is correct. In POE's case GGG have tried very hard to minimize the frequency of this because it is "confusing" to the player even though every other online game (aside from RTS games) has the client update to reflect what the server says is happening quite often and thus triggering a rubber band situation quite often.

Actually, the entire reason rubber-banding exists is to prevent speedhacking, since the alternate approach is for the server to implicitly trust the client... Which a hacked client can abuse by simply changing its relative position updates unreasonably fast... Which is what speedhacking is.

Rubber banding is a side effect of fixing desynchronization, cheating via speed hacks (and various other cheats) in games where such things won't work will greatly increase the chances of the server invoking the mechanisms that ensure the client is in sync with the server. It will trigger a resynchronization event that will warp/rubber-band the player back to where they should be if they hadn't speed cheated in the first place, not because they cheated but because they became out of sync.

There are also various ways to detect when the client is out of sync with the server and have the server force a client resync with the server, this is (as far as I know) generally accomplished by having the client tell the server what is happening. Although the server won't use this information to determine what is happening on the servers end, it can be used to determine if what the client is saying is happening matches what is happening on the server. If the client deviates too much from what the server says is happening than the server can resync the client to match the server by having the server transmit a special command to the client that invokes the clients resync functionality. Resynchronizing from a desync will typically also incur rubber banding.

For example the client can tell the server what enemy you're clicking on to attack and whether or not you're close enough to attack it etc., the server will compare this information with what is happening on the server and if it doesn't match or deviates too much from the server, a resync command will be sent by the server. I believe one of the more recent things GGG did was to have the server determine if the player was too far away from where they should be while the player is stunned, this was needed because a stun doesn't happen on the client without being told about it via the server (The client can't determine by it self how much damage an enemy deals and/or what effects the attack has on the player) so it's entirely possible for the client to not realize the player has been stunned and allow the player to continue walking around during the stun period, potentially walking a great distance away from where the player was stunned which is hugely problematic.

"

ACGIFT wrote:

"

Nicholas_Steel wrote:

RTS games work around it by essentially operating in a lock-step process, the client literally can't portray any actions to the end user unless the server authorizes the action first,

This is not QUITE true... My experience with RTS games has shown cases where, in fact, there is an allowance for some degree of latency (which can be adjusted) but the span it allows to go before full confirmation is usually far, far shorter.

Maybe, my experience with RTS multiplayer isn't that great (mostly CnC and the original Starcraft over a LAN). In Red Alert 2 the game slowed down to the slowest gameplay speed and was reasonably responsive to input from the player (I never noticed any rubber banding or desync), the slowness I believe was because one of the computers at the time couldn't handle the game very well. In Starcraft the gameplay speed was Fast but there was delay between you issuing a command and a unit responding to the command. You could adjust how long this delay was as well, setting a longer delayed alleviated desyncs caused by network interruptions/performance issues and allowed for a smooth experience at the expense of input responsiveness.

"

ACGIFT wrote:

"

Nicholas_Steel wrote:

So yeah, GGG basically took the same implementation as everyone else and greatly dialed back the frequency of the client resynchronizing with the server to minimize rubber banding. Since resynchronization happens much less often, it is extremely possible to go out of sync for extended periods of time (Most other games will exhibit frequent minor rubber banding due to frequent resynchronization and you probably never even notice it).

Actually, the reason other games appear to have less rubber-banding is not because they resynchronize more frequently... If anything, most other RPGs actually allow the client to get even MORE out-of-sync before snapping things back than PoE does.

Rather, it's a mix of reasons, some of which I outlined above, that result in the game being far more susceptible to this sort of instance occurring at all. You mentioned one such at the end, where the game's susceptible to never actually getting a proper resync, so that one desync event immediately triggers another, because the first resync directly caused a SECOND loss of sync.

Another I noticed is that, apparently, the server makes some shortcuts when it comes to trying to process how movement is handled... Or in one way or another, does NOT run the same calculations as the client does. One example (that I wish I'd been recording at the time!) was where I managed to desync and then be repositioned on the opposite side of a solid ledge. Apparently, whatever the server did, it somehow thought I'd crossed a barrier that, normally, would've taken me an extra 15-20 seconds to walk around. That tells me that the server's path-checking calculations were WAY off. And if it was off on one thing, it will be off on others.

Yes, I think the hardest thing for a lot of people to understand when explaining to them the problems of desynchronizations is that it very likely and very quickly can lead to a cascade of desynchronizations/failed logic decisions by the game/AI on the client.

One small desync involving player position (for example) can very quickly lead to multiple issues with how the client positions enemies and your interactions with them will be based on these incorrect enemy positions. This can result in the player seeing what looks like a gap in the group of enemies when in reality there is no gap. The player reacts to what they see and tries to dodge past the enemies through this perceived gap, the client shows they've succeeded and are now running away from the enemies (you made it through the "gap" on the client) while on the server you're running against a wall of enemies/surrounded by them at this point in time. Eventually you move far enough away from your real position that the server forces the client to resync and by this time you're dead because you were busy running away for several seconds instead of defending yourself against a threat you weren't ever aware of (This is typically when players see themselves warp next to enemies upon death and causes people to be vexxed over how they were there in the first place and not where they thought they were).

There have already been numerous proposals of ways GGG can minimize the problems, however so far all proposals seem to be 100% ignored. Possibly ignored because most will incur a minor increase in resource usage and increase the running costs of the servers. Various people have claimed the increase in resource usage is extremely small and unlikely to incur much of an increase in running costs.

I think one reason desync turns into such a major issue is the horrid pathing algorithm.

Somebody posted a great video of a monster continually trying to round a corner to attack the player and continually being sync'd back to its starting position, without the player moving. So obvioulsy, starting position of monster and (starting) position of player differ between server/client. This slight difference makes the monster on client side able to find a path to the player while on server side it keeps getting stuck at the wall.

That is unbelievably shoddy. Pathing is one of the first application of algorithms you learn when programming games and it is really not so hard. Getting round a single corner with no other obstacles is 100% solvable by even very simple and well-documented algorithms.

The same applies to the pathing of the player when he clicks to move somewhere. A small obstacle at the wrong place in the line of sight can stop this pathing algorithm. So if client and server disagree as to whether the small bump is exactly in the line of sight, then only one side finds a path, the other gets stuck. I believe this is the reason why I sometimes end up in rooms that I have definitely never clicked into. In this case, the client gets stuck in the path and I click again to get around the obstacle while on the server my original path was possible and executed and the second click ends up in the room I was approaching.

This could be fixed with a better pathing algorithm, without any need for better sync code or better internet connections. It would prevent a lot of desync instances.
But honestly, I think GGG has just given up the issue and simply can't be bothered to fix such bugs. That is a big shame, because POE is still the best game out there and wouldn't need to lose so many players if they paid a bit more attention to the quality of their code.

"

Treffnix wrote:

I think one reason desync turns into such a major issue is the horrid pathing algorithm.

Somebody posted a great video of a monster continually trying to round a corner to attack the player and continually being sync'd back to its starting position, without the player moving. So obvioulsy, starting position of monster and (starting) position of player differ between server/client. This slight difference makes the monster on client side able to find a path to the player while on server side it keeps getting stuck at the wall.

Yeah I mentioned that just before in I think my first huge post on page 45, a video was re-posted three pages back too demonstrating it very clearly: https://www.youtube.com/watch?v=j8wTE71aXFI

I notice people keep going on about how D3 doesn't suffer from the same level of desync, but a large part of that is down to map layout.

D3 uses primarily open spaces with little to no rock, trees debris etc laying around for players, mobs to get caught on. this masks alot of the desync issues by the fact that pathing becomes a lot simpler. POE however has alot of very narrow passage's walls, trees, rocks and other debris lying around, these all interfere with pathing, and all the little bumbs and knocks that occur while data is being sent back and forth all mount up to eventually cause some of the large desyncs people see.

Now GGG could solve a huge number of the desyncs by simply opening up the maps and removing most of the ground clutter, but you would lose a large part of the aesthetics of the game in the process. Image caverns of wrath looking like the scavengers den from D3, there is no comparison in atmosphere.

SO its a case of woking to improve the current system and allow us the amazingly complex maps with currently have, or move to much more simplistic maps like D3 and mask the desync.

"

thantos2024 wrote:

Now GGG could solve a huge number of the desyncs by simply opening up the maps and removing most of the ground clutter

Would this make bots easier to implement?
Curious, is this the (one of the) primary reasons why we have cluttered zones/maps?

Does GGG intended these obstacles to be used for defensive positions?