Client-server Action Synchronisation

"

Mark_GGG wrote:

snip

"

Mark_GGG wrote:

I've seen that twice before, once with chests and once with a unique terrain piece. I believe both were fixed up. If you can find some terrain which causes this sort of repeated pathing mismatch then a screenshot with /debug info can help us replicate it locally and work out what needs fixing with that bit of terrain.

"

Mark_GGG wrote:

That is caused by the client(s) and server getting out of sync - albeit with some sort of pathological case that repeats.
He clicked to initiate a move action. This into was passed to the server, which passed it to your client. On his client, and on your client, the movement action was able to move, where on the server something stopped it as soon as (or shortly after) he started moving. The fact it repeats shows there was a specific problem with the terrain and/or entities like chests that was not actually pathable on the server, yet both your clients managed to find a path through on them, before being resynced back because the server noticed he'd gotten too far from his server position.
I've seen that twice before, once with chests and once with a unique terrain piece. I believe both were fixed up. If you can find some terrain which causes this sort of repeated pathing mismatch then a screenshot with /debug info can help us replicate it locally and work out what needs fixing with that bit of terrain.

"

demonskye wrote:

What drives me sort of nuts about this is that it's EASILY fixed, and it essentially relates to "trust" of your client.

First off, let me begin with my qualifications. I'm a network engineer working for the world's largest CDN. On any given day about 25-39% of all the internet traffic IN THE WORLD passes through our servers. I've had to troubleshoot errors from tiny errors at layer 2 of the OSI model all the way up to layer 7. I've had about 10 years experience in the networking-tech field.

As for my qualifications in POE. I have 3 (primary) characters, A level 80 Chin-Sol puncture ranger, a level 86-flicker crit dagger shadow, and a level 83 Scion, which I'm not currently using due to changes in aura nodes and EK just not having the clear speed of my other builds.

As is obvious, my PRIMARY build would be my shadow. I do the most dps (>100k with flicker and 4 power charges + melee splash, or >190k with added fire damage to a single target), clear the fastest, and generally have the most survivability.

I've died a lot on my shadow. And basically every time it goes something like this: Flicker into a group of level 73-75 mobs, watch their health drop to 20-30%, desync, stop hitting anything, end up stunned, die in the middle of a pack of 60000 mobs that should have died in 1-2 more hits. I know I'm not hitting the mobs when I don't hear the sound connecting on my headphones. When that happens I smash my /oos macro and hope that it reacts before I'm dead.

So, from a network engineering perspective, what's happening here.

1. Packets are being sent to the POE servers. They're being routed here, there, and everywhere by the internet. That's just the way things work. Generally packets make it (about 99.95% of the time) and sometimes they need to be retransmitted. Let's assume that desync is NOT caused by packets dropping, as GGG has commented that it's a sync issue, not a QOS issue.

2. Packets are received by the server a calculation about that packet is performed in a matter of milliseconds, and a response is dispatched

3. The response makes it to your computer, and it verifies that you are somewhere that you can be / that your attack connected / whatever.

So, what's the problem with this?

1. It places NO confidence in the client. Essentially POE is checking every single movement that the client is making, rather than allowing the client to determine whether or not you have taken a hit, the server decides. On a good day, when I'm playing in Boston, MA and my friends are playing in Singapore, I have a ping of about 350-400ms as it's likely me connecting to the SEA servers. That's totally acceptable. What's NOT acceptable is me using flicker strike once and missing with every hit of it, and then teleporting back a half a screen. What happened?

2. My computer sent packets to the GGG servers that service the Pacific Islands and SE Asia, and in the 350-400ms round trip they took, my client chose 3 random targets with flicker. Meanwhile the server chose 3 different targets to attack (flicker + multi). The net decision is: Well, we can't decide what happened, so you didn't use the attack and let's move you back to the last point we agreed. This happens CONSTANTLY. Using movement skills (WB, Flicker, and LS for example) causes me to desync ALMOST immediately.

3. This approach also doesn't factor in outside sources of network congestion, latency, or issues regarding lag spikes (local, or at any of the hops between me and your server). If you do a simple ping-of-death on a website like google and ping it for a period of 24-48 hours every 3-5 seconds, about 98% of your pings should come back within 10-50ms, 1.5% should come back between 50-150ms, and .4% will be well above that at about 150-700ms, and the outliers will probably be about 700ms - 3s. The point I'm getting at is that even with normal connections you can expect STRONG latency spikes in otherwise normal traffic on the internet. It's the nature of the beast.

4. This approach allows for us to "dodge projectiles" and melee attacks that we see coming, but let's face facts and compare this diablo 3. I can dodge spears or fireballs thrown at me just fine... why? Because there's no server-side check to make sure of if I dodged it, and if I didn't let's rewind to the last thing that worked. I'm either standing in fire, or I'm not. It's decided on my client and passed to the server. The server is not telling me if I'm standing in fire. That's why POE cannot fix this. It was a design decision, and it was strongly likely based in anti-cheating. Putting too much information client-side allows the user to cover up what's really happening and allows for a chance to change that information before it reaches the server.

I think that the end of point 4 needs to be reiterated. I would NEVER cheat in POE. The only time I want to cheat is when I die to desync... which wouldn't be caused if the client had more trust placed in it.

There is NO discussion on this. That IS what's happening. You can claim that it's a million other things, but until you realize that the clients need to be trusted, but checked in ways that don't cause user interference, that this game WILL suffer. It will lose you users. It will cost you money. And it will hurt your bottom lines.

This has been an issue for years now, and has sat near the top of your development manifesto for the better part of a year.

The reality is that testing in a 0-latency office is not going to produce desync. Hire someone in Boston to run Wireshark and packet capture EXACTLY what's going on. Find triggers, make bug reports, and stop dodging the issue. This is the BIGGEST problem holding this game back from being the GOTY. And you guys can't / won't throw the resources necessary at it to handle it. Which is saddening.