Client-server Action Synchronisation

"

demonskye wrote:

What drives me sort of nuts about this is that it's EASILY fixed, and it essentially relates to "trust" of your client.

First off, let me begin with my qualifications. I'm a network engineer working for the world's largest CDN. On any given day about 25-39% of all the internet traffic IN THE WORLD passes through our servers. I've had to troubleshoot errors from tiny errors at layer 2 of the OSI model all the way up to layer 7. I've had about 10 years experience in the networking-tech field.

As for my qualifications in POE. I have 3 (primary) characters, A level 80 Chin-Sol puncture ranger, a level 86-flicker crit dagger shadow, and a level 83 Scion, which I'm not currently using due to changes in aura nodes and EK just not having the clear speed of my other builds.

As is obvious, my PRIMARY build would be my shadow. I do the most dps (>100k with flicker and 4 power charges + melee splash, or >190k with added fire damage to a single target), clear the fastest, and generally have the most survivability.

I've died a lot on my shadow. And basically every time it goes something like this: Flicker into a group of level 73-75 mobs, watch their health drop to 20-30%, desync, stop hitting anything, end up stunned, die in the middle of a pack of 60000 mobs that should have died in 1-2 more hits. I know I'm not hitting the mobs when I don't hear the sound connecting on my headphones. When that happens I smash my /oos macro and hope that it reacts before I'm dead.

So, from a network engineering perspective, what's happening here.

1. Packets are being sent to the POE servers. They're being routed here, there, and everywhere by the internet. That's just the way things work. Generally packets make it (about 99.95% of the time) and sometimes they need to be retransmitted. Let's assume that desync is NOT caused by packets dropping, as GGG has commented that it's a sync issue, not a QOS issue.

2. Packets are received by the server a calculation about that packet is performed in a matter of milliseconds, and a response is dispatched

3. The response makes it to your computer, and it verifies that you are somewhere that you can be / that your attack connected / whatever.

So, what's the problem with this?

1. It places NO confidence in the client. Essentially POE is checking every single movement that the client is making, rather than allowing the client to determine whether or not you have taken a hit, the server decides. On a good day, when I'm playing in Boston, MA and my friends are playing in Singapore, I have a ping of about 350-400ms as it's likely me connecting to the SEA servers. That's totally acceptable. What's NOT acceptable is me using flicker strike once and missing with every hit of it, and then teleporting back a half a screen. What happened?

2. My computer sent packets to the GGG servers that service the Pacific Islands and SE Asia, and in the 350-400ms round trip they took, my client chose 3 random targets with flicker. Meanwhile the server chose 3 different targets to attack (flicker + multi). The net decision is: Well, we can't decide what happened, so you didn't use the attack and let's move you back to the last point we agreed. This happens CONSTANTLY. Using movement skills (WB, Flicker, and LS for example) causes me to desync ALMOST immediately.

3. This approach also doesn't factor in outside sources of network congestion, latency, or issues regarding lag spikes (local, or at any of the hops between me and your server). If you do a simple ping-of-death on a website like google and ping it for a period of 24-48 hours every 3-5 seconds, about 98% of your pings should come back within 10-50ms, 1.5% should come back between 50-150ms, and .4% will be well above that at about 150-700ms, and the outliers will probably be about 700ms - 3s. The point I'm getting at is that even with normal connections you can expect STRONG latency spikes in otherwise normal traffic on the internet. It's the nature of the beast.

4. This approach allows for us to "dodge projectiles" and melee attacks that we see coming, but let's face facts and compare this diablo 3. I can dodge spears or fireballs thrown at me just fine... why? Because there's no server-side check to make sure of if I dodged it, and if I didn't let's rewind to the last thing that worked. I'm either standing in fire, or I'm not. It's decided on my client and passed to the server. The server is not telling me if I'm standing in fire. That's why POE cannot fix this. It was a design decision, and it was strongly likely based in anti-cheating. Putting too much information client-side allows the user to cover up what's really happening and allows for a chance to change that information before it reaches the server.

I think that the end of point 4 needs to be reiterated. I would NEVER cheat in POE. The only time I want to cheat is when I die to desync... which wouldn't be caused if the client had more trust placed in it.

There is NO discussion on this. That IS what's happening. You can claim that it's a million other things, but until you realize that the clients need to be trusted, but checked in ways that don't cause user interference, that this game WILL suffer. It will lose you users. It will cost you money. And it will hurt your bottom lines.

This has been an issue for years now, and has sat near the top of your development manifesto for the better part of a year.

The reality is that testing in a 0-latency office is not going to produce desync. Hire someone in Boston to run Wireshark and packet capture EXACTLY what's going on. Find triggers, make bug reports, and stop dodging the issue. This is the BIGGEST problem holding this game back from being the GOTY. And you guys can't / won't throw the resources necessary at it to handle it. Which is saddening.

"

Skizo wrote:

That sort of makes sense, although I am not sure I totally understand the rational behind sending info to my client that is not "verified".

Maybe give the server the ability to verify stuff like this and tell the client if everything works out or not? I am sure that would help a lot with movement skill desync (hello Cyclone).

It might as well, seeing that the game server has absolute authority, so it makes very little sense for the client to do its own thing and then have the server overrule it, when the server figures out that the action produced a different result than on the client.

"

demonskye wrote:

What drives me sort of nuts about this is that it's EASILY fixed, and it essentially relates to "trust" of your client.

First off, let me begin with my qualifications. I'm a network engineer working for the world's largest CDN. On any given day about 25-39% of all the internet traffic IN THE WORLD passes through our servers. I've had to troubleshoot errors from tiny errors at layer 2 of the OSI model all the way up to layer 7. I've had about 10 years experience in the networking-tech field.

[SNIP]

"

sanuitstli wrote:

So MUCH, but SO SO MUCH **THIS***!!!!!!

I **HATE** Cheaters, but if they can get away with it, so be it. What's the amoral Shadow personality all about ?

However, *TRUST* is the issue, and ZERO TRUST is a pathology. Like so many gross personality defects, that "work" to survive into Adolesence and Adulthood, they don't work when taking responsibility for Mature roles in life. Server-side calculation and Rubber-banding worked in GGG's Infancy, and Adolescence, but GGG is now growing up and a more mature solution is in order.

It should be possible without a mind-numbing amount of cogitation to come up with a means of AUDITING the initial conditions, and snapshot intervals, verifying the progress and outcome of client computed action. Once a cheat happens, it created divergences in the action that are detectable. A sampling of mechanical events can be audited, to lessen the burden on the server. Intervention is necessary only when a significant divergence occurs.

Once a "cheating" violation is detected, the account can be frozen, and disconnected.

The problem is that GGG is **ALWAYS** controlling the honesty of action. However, a cheater is going to cheat with some frequency, and only has to be caught "Red-Handed" ONCE!!

Everyone is being taxed with constant suspicion and prejudice by the present architecture.


Short break to say "GGG, YOU ARE AWESOME!!!", I'm not bagging on you!

[snip]


Ronald Reagan once repeated "TRUST BUT VERIFY".

Make it so ?

"

rrtson wrote:

You're insinuating that players are too dumb to distinguish between desync and lag?

"

TikoXi wrote:

"

rrtson wrote:

You're insinuating that players are too dumb to distinguish between desync and lag?

Yes.
Yes they are.
[snip]

"

It's relatively frustrating to see someone say "the desync is really bad today" when they mean "Some ISP between me and the server is lagging badly today".