Client-server Action Synchronisation
|
I think I should make another post detailing my thoughts.
Your game is fundamentally broken on an extremely basic level. Every other game, with online capabilities, in existence has objectively better net code. Do you know why? I'll break it down for you: Other games sync the client and server every time they communicate (50 millisecond ping means they sync every 50 milliseconds) on a UDP connection. Path Of Exile syncs the client and server instances every ten-or-so seconds (from what I know) on a TCP connection. Do you see the problem here? There's no consistency with your game. It's a constant guessing game for the player. They don't even know if that monster or even their character is where it should be at any given time. They can't trust their own damn screen. How do you expect them to trust you is you don't even trust them? Other games sync more than just entity positions every X-or-so seconds. They sync position, orientation, animation, and most importantly, velocity every X milliseconds for all mobile assets. That way, the player can trust their own screen as much as they trust their own internet connection. Of course, you could trust the player even more and let them play your game without a 150% uninterrupted internet connection, but that will never happen, will it? Syncing vectorial velocity is incredibly important, along with XYZ position. Velocity gives you an error margin with the prediction algorithm, which should only predict the next half-second, but be over-ridden the very next client-server synchronization. Does that make sense? I hope it makes sense. Yes, I know this change will put a huge strain on your servers, much more than now. You can compensate by purchasing more of them; distribute the load and give the player more options. It's a worthwhile investment, in my opinion. Now I have that horrible feeling telling me I just wasted my time laying all of this out. |
|
|
It is sad to admit, but for the first time, the luster of PoE is starting to fade :(
I need some sort of update on the situation surrounding desync. My friend & I both died to desync on separate occasions during the one month hardcore league. It's extremely disappointing when you play for 5+ consecutive hours, only to die because movement skills destroy your synchronization. PoE has been my dream game so far, but I'm getting awfully tired of the desync issues. I shouldn't have to smash my /oos macro every minute, or every time I'm fighting an invader boss. I want to continue to love PoE and donate my money. However at this point, I'm getting doubtful that these issues will be addressed. I would gladly donate $50 and wait a year to play again if it meant the code being rewritten and more server space purchased. This game is outstanding in so many ways, but if the desync isn't resolved, the game is doomed. I would be so disappointed if that were the case. Please...any sort of update? This needs to be addressed again. I'm hoping the expansion can help this issue in some small way. Sometimes, I feel like GGG should spend a little less time developing their flashy armor microtransactions, and instead, hire another programmer. Last edited by TRIPL3try on Jul 16, 2014, 6:13:16 PM
| |
|
Whatever the technical reasons for desync and how it works,
either rewrite the code from scratch and start over and make it right or make it so it works within the game limits as it is today. Its better now for sure than it was last time I played. so I note improvements. Content is fine but such a technical issue really needs to be put on the priority 1 list. I dont explain to a client why their life is like shit and what I cant do about it. I make sure its not like that so it works for them. The level of desync is better now so keep the work up. |
|
|
I've been checking this forum thread for the last few weeks to see any dev response, I've supported this game from alpha, but I just cant stand playing this through the desync anymore. I'm not here to flame, I'm here to tell you that the only thing to make me play the game again is if you FIX THE DESYNCS, its seriously the ONLY thing stopping me from playing. If I want to play a hardcore character in a "hardcore" game, I want to be able to TRUST MY OWN SCREEN, and not have to guess or "play safe" or "play around" stuff just because a desync COULD happen.
Please GGG... fix this so I can enjoy your game again... Above content may contain traces of nonsense.
Reality is simply an unrealistic version of online gaming. | |
" What is there to fix? I can play the game just fine. Hardly any desync anyways, so what. |
|
|
Yes if Dirk thinks there isn't a problem, then no one in the entire game has a problem.
MY SHOP THREAD 485102
|
|
" hard to write /oos ?? its impossible to make game with 0 desync, guess there is no online game with 0 desyncs ;) ... what do u expect?? and i see improvement in this way... i myself desync less, and my friend will say same thing oh and again /oos u should be tkankful we have this little command IGN: UbogiMurarz
| |
|
So I've just finished reading all 31 pages of this thread, and this is going to be a long post because I'm going to comment on all of the things which, as I see it, are the most relevant points regarding desync.
" The assertion here is that the client cannot be authoritative on what happens in game because this would require trusting the client. Trusting the client would allow manipulation of the client by users in order to cheat. I applaud the commitment to preventing cheaters. However, I believe that they are mistaken in their assertion that allowing the client to be authoritative on game events requires trusting the client. Authority and trust are not the same thing, and neither does one imply the other. I think this point has been made adequately by several posters earlier in the thread. " demonskye has really started to get to the heart of what I'm going to suggest here. But I believe that it would probably be possible to do a few steps better than what demonskye has suggested here. To illustrate, HellGauss responded to demonskye's suggestion by saying: " I am not a security/encryption expert, but I believe that it would be possible to prevent this kind of reverse engineering by having portions of the client encrypted while the application is not running, and locked by the application while it is running. The path of exile software must have modules which are responsible for calculating things like player movement, skill use, hits, damage and so on. Perhaps the client could do these calculations, but the code that does it, the part of the client that does these calculations while the game is running, are all strongly encrypted to prevent access to the files for reverse engineering. They only get decrypted and become able to be used when the server provides a decryption key for the necessary calculatory code. Perhaps another method of achieving more or less the same thing might be by having the server actively verifying that the relevant portions of code on in the client have not been modified or tampered with at the start of each gaming session. Perhaps this might add server overhead to some extent, but I think some of my subsequent suggestions probably save as much overhead as this adds. The range of possible solutions here must be pretty extensive, and it strikes me as implausible that there could not be partial client authority solution that would be workable and prevent 99.9% of cheating, as well as detecting the remaining 0.1%. The range of people who would be able to break that sort of system and cheat successfully probably wouldn't be any larger than the range of people who can break the current system and cheat. It seems that I'm not the only one who has come to some version of this view, since Goose312 said: " Nicholas_Steel had a reasonable question about this suggestion: " The answer to this is that the server supplies the clients with the variable information to be used in any given session which will determine the outcome of the client's calculations. Certain variables will be missing in the client at the start of any given session and must be gotten from the server. The client can have an internal certificate which is encrypted, and is used to verify that it actually is the client and not some other program that is running the calculations. Further, the server can assess the results of the client calculated outcomes against the variables it supplied the client with at the start of the session, given that the server knows the client's methods of calculations. Anything that deviates too far, or in some cases (think attack speed, for instance) at all, could indicate tampering. Startkabels also responded to demonskye's suggestion saying: " Although this portion of my post is the part that I am the least certain of, I still think that there is a way to make this sort of, as demonskye called it, partial trust solution work seamlessly (as far as the player is concerned) even in multiplayer. I'd be interested in reading constructive input on how this could work from other community members. It's a matter of having some key decisions made by the server and then handing off instructions to the various clients to control game entities in that fashion. Typically a monster picks one particular target for its skills. In a multiplayer game, at any given time, the monsters in a map will all either be idle, or attacking some target or another. As long as, having received information about player position, the server tells the clients in a multiplayer game which monsters are attacking which targets, the rest of the monster behaviour can be left to the clients. So the server tells the client of player 1 that monsters x, y and z are attacking the player, but other monsters are attacking other players. Player 1's client then takes control of monsters x, y, and z, and directs them to attack player 1. Other monsters attacking other players are left to the control of their respective clients. Because this all happens client side, player 1's interactions with those monsters doesn't have any desync or latency issues. The client streams information about the portion of combat that it is in control of directly to other clients which are a part of its instance, and to the server. In some ways this should improve network performance because it distributes the bandwidth required to keep the game running across all of the clients rather than forcing the server to send and receive all of the snapshot/syncing data. Thus more frequent updates can be had for clients in multiplayer situations even as the load on the GGG server is decreased, since the GGG server does not need to directly update each client about what all of the monsters are doing - the clients do this themselves for each other. This way of doing things allows for a potentially very high rate of game state updates because the resources and responsibility for game state updates are distributed across several computers. Clients don't need to send that much information either, because they are only sending information about the monsters they are currently in control of. Given the information it receives the server can at any stage shift control of a monster from one client to another when the monster 'decides' to attack a different target. The server could also check client determined monster behaviour to ensure that it was falling within acceptable ranges. This would help to preserve the security and integrity of the gameplay. For each client, the movement, attacks, and damage of monsters that are not directly attacking the player on that client appears as something like a stream from the other clients in the instance. Damage to players from monster AoE skills under the control of other clients gets calculated for each player by his local client on receipt of the streaming information from other clients. So each player has the experience of being hit only by damage that appears in their game client as hitting them. Under this system, sometimes some desync might be observable when a player sees an AoE skill of a monster being controlled by his client not influence the health of one of the other players in the game, even though that player appeared to be within the AoE on the client which is currently controlling the monsters. That's because there is latency between clients. Player damage gets calculated locally on each player's client on the basis of whether the player was locally within the AoE at the local time the information about the location of the AoE relative to the local player was received. That is a good way to do it, because players mostly don't really care about other player's desync, they only care about their own. This suggestion makes desync something that only apparently happens to other players, never to the the player themselves. There could still be security against player cheating because the server supplies the client with the statistics which will be used in the variables for calculating player damage, movement, accuracy, stun chance, and so on, just as the server supplies information about monsters and their attributes to the clients. In terms of determining drop allocation and who gets the killing blow on monsters, this could still be calculated server side. Basically, clients are continuously updating the server and other clients with information about how much damage they are putting down on the monsters as it happens. The server knows the health value of each of the monsters and when it has received a sufficient quantity of damage for a particular monster that monster is deemed killed by the most recent player that put damage on it. Differing latency between players could potentially influence who gets the kill on a monster here, but that would seem to me to be an issue of negligible importance. The server then applies the MF modifiers of the player whose damage is deemed to have killed the monster to send out the drop information to the clients. In this model clients do not determine the drops, though they do determine, with their local copies of monster health provided by the server, and the sum total of damage dealt to each monster as provided by all of the clients to each other, when each local version of each monster will die. In other words, each client uses the same process to determine when monster death occurs as the server does. All going according to plan, and assuming no major latency spikes on any one particular client, the time at which monster death occurs should be more or less the same across all clients and the server since the sum total of information over time at each client and the server should be approximately equal. Since the server determines who 'officially' killed the monster at the time it receives damage on that monster which reduces the monster's life to zero from any one of the clients, drops get calculated by the server and sent to the client(s) on that basis. This preserves at least some of the security of the PoE economy against hackers. If a client loses connection to the server then the server can assign that client's monsters to another client. Perhaps a sophisticated version of this suggestion could have the server check with other clients to see whether they are still connected to the client that has disconnected from the server. Then the server could instruct those other clients to update the client which the server can no longer communicate with. At a local client that has been totally disconnected, things would stop moving and damage calculations would cease if streaming information from other clients and the server ceased. In some sense the idea here is that the server is telling all of the clients how to make their calculations and what the data to use is, it is simply outsourcing the processing power, much like seti@home does. Because of this the server can easily check the results of the client calculations to ensure that no tampering/hacking has happened. The nice side effect of doing this is that player's experience of desync gets totally marginalised. Many of these ideas have been suggested by previous posters in the thread, I hope that my elucidation of it adds some new ideas. To this point I have been describing how my preferred PoE networking model would operate. I now turn to some of the arguments and discussions of other networking methods that have been discussed. These are outlined in the OP as such: " I will refer to these as I have labelled them in the quote. I have argued that the correct way to resolve desync is to follow some partial distributed version of Option 1. " So, the first thing to point out is that lag and desync are different, but not unrelated. The greater the latency difference between client and server the more time there is between the issuing of a command on the client and its receipt by the server for other server side actions to occur. Thus, desync potential increases to some extent as latency increases. Having said this, the rate of snapshot information sent from the server to the client and client to the server also has a major effect. It seems to me that a big part of PoE's problem is that even in low latency situations the rate of snapshot information is far too low to maintain reasonable synchronicity. That is part of why I favour my solution to the problem, because the latency issues are eliminated and the frequency of snapshots is greatly increased by having multiple clientservers constatly updating each other with the set of monsters that each has control over. Having made that point, what I want to say about DexDeus' comment is that the experience of a lag spike which gets you killed is, for most players, probably less common than the experience of desync which gets you killed. If I understand desync correctly then it would also be true that an unexpected lag spike could cause subsequent desync which gets you killed anyway, but the primary culprit is the low rate of snapshots by the server (sometimes a monster will run up and stand next to me, but not attack me and not do any damage, and when I attack it I do no damage, though the attacks still cost me mana, but more on that later. This can persist for a good 5 to 10 seconds sometimes before the monster teleports back to where the server authoritatively says it is). Eliminating desync by having actions occur only when the command is verified by the server would make lag spikes much more noticeable as the cause of player deaths. However, lag spikes happen less often than general desync which is caused by prediction entropy and a low rate of server provided snapshots. The quantity of prediction entropy could be thought of as a function of the rate at which the server sends current state snapshots to the client. More frequent snapshots means less prediction is required. Unfortunately, when latency is high the less prediction there is the more the latency is felt. This will become relevant later on in this post. QuakeLive is one game in which the experience of desync basically never happens. Quake updates its client with snapshots from the server that contain the location of all obejcts on a map 30 times per second. This effectively eliminates desync completely. It does also result in lag spikes in Quake being lethal in high pressure situations, since players effectively just stop moving and taking actions. But that just goes with the territory of PVP gameplay. Serious players know that the only way to really test PVP skills on a level playing field is on LAN, though internet based competitions do take place and reliability is hugely better than it was even a decade ago. The idea of a level playing field for PVP in PoE is a bit silly anyway given that most of the time players have differently powered builds, gear, and levels anyway. So hardcore PVP deaths due to latency spikes can't be a reason not to use that system. Players are smart enough to know that "I lagged out" is a reasonable excuse for losing a PVP match. But perhaps the relevant point here is that because of the high quality of the hosting in QuakeLive servers I cannot say that I see lag spikes often at all. Typically not more than once or twice a month, and even then they don't last long, maybe a few seconds here and there before returning to normal. I have noticed that PoE has much more regular lag spikes than Quake does. That's probably more a reflection of the quality of the hosting and/or server stability than anything else though. Thus, GGG could take steps to bring the rate of lag spikes on the PoE servers more in line with what the experience of Quake is like. This could be done by getting a higher number of more localised instance servers with closer geographic proximity to users. More local instance servers would also have the double advantages of resulting in generally lower ping times for users and less powerful hardware requirements since the user load per server would be smaller. Thus, cost might well even out - though that is something that would have to be carefully evaluated, to ensure that there were no adverse effects on the GGG business model (but here, one might wonder how much the desync issue as it is has been damaging the GGG business model anyway). This point largely echoes demonskye's comments: " The most important point in these last few paragraphs is that given good quality of service, player deaths and, thus, player experience, given a latency based confirmation of action system (Option 2) would generally be a vast improvement over the current system. Having said that, I think there are probably better solutions relating to my rejection of GGG's lack of distinction between authoritative clients and trusted clients. Perhaps one of the best elucidations of the sort of idea I'm trying to get at was the first post on page 18 by StarTraveller_DK. (not quoted here for brevity's sake (Haha! OMG this post...)) In the manifesto GGG are concerned with making sure HPWs (http://www.abbreviations.com/term/136794) can still play PoE. " I feel that this is a counter-productive policy. The majority of seriously hardcore gamers will move heaven and earth to ensure they have a fast internet connection for their gaming. Given that GGG want to target the hardcore gaming market, it makes no sense to spoil the experience for those of us with good connections just to cater to a few HPWs who are unwilling to take the necessary steps to secure themselves a good connection. Far from alienating players, making the game more latency dependent, providing good, stable, geographically distributed servers with excellent hosting would empower the most enthusiastic players to ensure that their gaming experience in PoE was a high quality experience by making sure they have a good ISP and connection. With the current prediction/low sync rate system improving your internet connection just doesn't really help - you're going to get desync either way. Morover in view of demonskye's earlier comments, this line just seems wrongheaded: " Well, I live in New Zealand, Wellington to be precise. But GGG is actually a New Zealand company. QuakeLive, run by id Software in Mesquite, Texas, has servers in Auckland. Assuming that GGG are right that being far away makes it feel like playing drunk, why can GGG not put a PoE server in Auckland like id does with the QL servers? Do you really feel that your own company's location is so far away that you can't run a server here? That doesn't even really make sense, it just sounds like a good reason to put a server here. That aside, if I'm not mistaken then the PoE servers I'd be playing on are located in Australia. F1 in game typically shows me with a fairly stable (though much less stable than my QL ping times to Aussie servers) ping time of 50 milliseconds. This seems entirely reasonable. It's very similar to what I get playing on an Australian server in QuakeLive. In QuakeLive with it's high rate of snapshots, on 50ms of latency, it does not feel like I'm playing drunk at all, and Quake is hugely latency sensitive. 50ms is pretty acceptable for FPS games. As 074 repeatedly commented in previous pages more than this isn't particularly favourable, but I find things playable all the way on up to 80 or 90ms. We're still talking less than a tenth of a second of latency there, and it's noticeable but not a deal breaker. So I'm afraid I have to disagree that New Zealand should qualify as "far away". More than this, as someone who plays League of Legends, I have to echo the comments of others in that your assessment of the MOBA genre seems totally erroneous. Now, on to a few issues with desync as it currently is that I think could be fixed up reasonably simply. I want to point out that the way desync currently happens the player is often punished for it (duh, even when it doesn't kill the player). One example of this is mana usage. If a monster which is out of sync approaches my player character, and I attack it, the attack will use my mana, even though it could not possibly connect to the monster, and I would not have made the attack at all if it had not appeared to me that there was a monster next to me. This particular form of player penalisation is especially brutal in endgame maps which have the life, energy shield, and mana recover 50% slower and/or players have no life and mana regeneration modifiers on them. This seems to penalise the player for the fact that desync has occurred, even though the player is not in control of the desync. I think that if I make an attack against a monster which isn't really there on the basis of falsely supplied information, then, complimentarily, the attack really ought not to have happened at all since there wasn't really a target. Thus, I think that players should not be penalised in this case - I want my mana back. This is much like what Raghin said: " If the game's software can detect that a monster has desynced, it can detect the period of desync too. If it can detect the period of desync it can detect attacks made against desynced monsters (this point should be glaringly obvious since the game can obviously detect that an attack was made - my mana decreases, and that no damage was done because of desync). If it can detect that I can quantify the mana used in such attacks and refund it. Lastly, a comment on the effectiveness of banning in F2P games. Ocylix raised the issue of how business model relates to banning cheaters by saying: " There are other ways of dealing with this than just banning the account. If cheating is detected on the client system it is possible to ban the mac address of the computer, ip bans are also possible. A disable code could be sent to the game client by the server to prevent other installations on that machine. I know that bans for cheating in QuakeLive, which is also free to play, are notoriously difficult to escape. If you've actually read this post this far, I'm very impressed. Thanks GGG for paying attention to my feedback and I really hope you sort this, the only real flaw with an otherwise perfect game, out as soon as possible. Last edited by HoneyBadGerMan on Jul 19, 2014, 11:29:23 AM
|
|
" The thing with encryption though, is that it's the parts that aren't encrypted that are the issue. Unless the game uses SSL or similar encryption for the transfer of data to/from the server (Greatly increasing server work load/slowing the process down heavily) the hackers will directly attack the data stream and completely bypass any local encryption, possibly with a forged certificate (Dunno how hard that is to pull off). I also don't think encryption can work when the values being returned to the server can vary. Also the part that I quoted above would improve responsiveness in multiplayer with other human players even when the client still does not have any authority/trust over anything. It would largely resolve the issue of more players = more "impossibly difficult to predict" scenarios. Lastly, I kinda stopped reading about 2 3rds of the way through :/ Your post just kept going, and going, and going, and going, are you sure you're not running on Energizer power? Your post does a pretty good job at summarizing a bunch of stuff from this thread. Edit: The New Zealand comment from Chris was likely made before the Aussie server existed. Computer specifications: Windows 10 Pro x64 | AMD Ryzen 5800X3D | ASUS Crosshair VIII Hero (WiFi) Motherboard | 16GB 3600MHz RAM | MSI Geforce 1070Ti Gamer | Corsair AX 760watt PSU | Samsung 860 Pro 512GB SSD & WD Black FZEX HDD Last edited by Nicholas_Steel on Jul 19, 2014, 4:33:06 AM
|
|
" http://www.twitch.tv/jmyers123/c/4289878 This is what needs to be fixed, sure you can play the game just fine, but its not fine if you want to play hardcore. Then you need to be able to trust your screen. As proven by that rubberband death, you cant trust your screen Above content may contain traces of nonsense.
Reality is simply an unrealistic version of online gaming. |
