GGG, you seriously need to do something about your international ISPs

"

sidtherat wrote:

Spoiler

1) local instance instability. poe uses instances (thin poe 'clients' that run chromeless, by dozens per standard DC node). these instances are provisioned based on statistical data of prior load and usage (as instances are short lived i doubt they do live up/downscaling, so it is pre-planned and instances are pre-spun). this is done to maximize the number of instances per node. there are cases - like VERY laggy builds being popular in given league - where this statistical approach results in instances being disturbed by so called 'noisy neighbour' instances. simplified example: there are 10 instances per node. 'normal' players are using around 9% of node cpu cycles with 10% margin used for peaks (instance load is VERY non-linear). everything is fine. but if one of these players uses some kind of ward-loop idiocy with 500 casts per second and faces expedition with mobs invulnerable to the damage type it will use 30 or more % of node power STEALING it from other players. this is very noticeable for players affected. real scenario most likely operates at 50-100 instances per node
in theory it can be provisioned for - making the buffer larger or allocation logic smarter - but it costs. hosting costs are SIGNIFICANT and im sure GGG is cutting corners wherever possible. there is a reason league starts are what they are (ignoring bugs for a moment)

re-entering map MIGHT fix it, if the original instance is closed and you get loaded into newly spun on a different node. but if the problem is not severe, orchestrator might keep assigning you to the same instance/node. it is also possible that youll get 'couldnt join any instances' if the instance is already dropped/crashed and its state wasnt synced before so the backup couldnt be created

"

sidtherat wrote:

Spoiler

2) routing. various issues are in play here, im familiar with just some of them - but there is no secret in traffic being cut unevenly and some players having better access to busy routes than others (big players in general either use their own hardlines or pay to even bigger players for exclusivity). POE is a minor player, so small that i bet it gets the worst possible prio everywhere possible. i doubt GGG has any leverage here. there are known issues at connection points between various big networks, there are maintenance breaks of accidents (one forest fire somewhere remote could result in entire DC being downgraded due to to damaged power supply backup - these issues ripple and multiply like crazy

it is possible to request certain routings, use internal ISP landlines if possible etc - but i bet each choice has a hefty price tag attached that only the big peckers can accept.

in short - small players cannot really do much and POE despite what many here think is marginal player with pretty much no negotiation leverage. i think that the most realistic path - that GGG wont take - is to hard-limit certain idiotic mechanic in their game (like self-harm loops, cast when stunned etc) - these overwhelm their engine and make any form of load prediction guesswork. and that can only be fixed by some serious OVERprovisioning. given that GGG doesnt want to pay for it we get what we get each league 'the diarrhoea' builds are popular

"

Zrevnur wrote:

"

sidtherat wrote:

Spoiler

1) local instance instability. poe uses instances (thin poe 'clients' that run chromeless, by dozens per standard DC node). these instances are provisioned based on statistical data of prior load and usage (as instances are short lived i doubt they do live up/downscaling, so it is pre-planned and instances are pre-spun). this is done to maximize the number of instances per node. there are cases - like VERY laggy builds being popular in given league - where this statistical approach results in instances being disturbed by so called 'noisy neighbour' instances. simplified example: there are 10 instances per node. 'normal' players are using around 9% of node cpu cycles with 10% margin used for peaks (instance load is VERY non-linear). everything is fine. but if one of these players uses some kind of ward-loop idiocy with 500 casts per second and faces expedition with mobs invulnerable to the damage type it will use 30 or more % of node power STEALING it from other players. this is very noticeable for players affected. real scenario most likely operates at 50-100 instances per node
in theory it can be provisioned for - making the buffer larger or allocation logic smarter - but it costs. hosting costs are SIGNIFICANT and im sure GGG is cutting corners wherever possible. there is a reason league starts are what they are (ignoring bugs for a moment)

re-entering map MIGHT fix it, if the original instance is closed and you get loaded into newly spun on a different node. but if the problem is not severe, orchestrator might keep assigning you to the same instance/node. it is also possible that youll get 'couldnt join any instances' if the instance is already dropped/crashed and its state wasnt synced before so the backup couldnt be created

If you are responding to me: This doesnt fit the symptoms of most issues this league. Like if there were issues then usually all or most EU server locations were affected. Your description does however fit other cases of issues, personally I didnt have many other issues this league though.

"

sidtherat wrote:

Spoiler

2) routing. various issues are in play here, im familiar with just some of them - but there is no secret in traffic being cut unevenly and some players having better access to busy routes than others (big players in general either use their own hardlines or pay to even bigger players for exclusivity). POE is a minor player, so small that i bet it gets the worst possible prio everywhere possible. i doubt GGG has any leverage here. there are known issues at connection points between various big networks, there are maintenance breaks of accidents (one forest fire somewhere remote could result in entire DC being downgraded due to to damaged power supply backup - these issues ripple and multiply like crazy

it is possible to request certain routings, use internal ISP landlines if possible etc - but i bet each choice has a hefty price tag attached that only the big peckers can accept.

in short - small players cannot really do much and POE despite what many here think is marginal player with pretty much no negotiation leverage. i think that the most realistic path - that GGG wont take - is to hard-limit certain idiotic mechanic in their game (like self-harm loops, cast when stunned etc) - these overwhelm their engine and make any form of load prediction guesswork. and that can only be fixed by some serious OVERprovisioning. given that GGG doesnt want to pay for it we get what we get each league 'the diarrhoea' builds are popular

But GGG isnt the old GGG anymore. They are part of Tencent and Riot supposedly has better solutions already. So from my POV there isnt really any clout/money (vs the world) issue. They just would need to cooperate with Riot for a joint solution.

"

Timbo Zero wrote:

I am fairly sure I already covered the first part, to the point I gave up trying to explain further.

"

Timbo Zero wrote:

On the second point, Tencent having a share in GGG in no way reflects how much leverage (not convinced on this part of Sids reply but my expertise is 20 years old when I retired from the field) they are prepared to place for a given company they are invested in. That would, if it happened or could happen, depends entirely on returns on their investment (aka dividends from shares)

"

Timbo Zero wrote:

but this to my knowledge is speculation and not proven by financial statements (which could be found online, if you cared enough)

"

Zrevnur wrote:

What I dont agree with in such threads are "they cant do anything" kind of comments: They (Tencent) probably have the power to do something about this.

"

Timbo Zero wrote:

Your arguments have all been ‘because I know they know’ which is heavily flawed logic.

"

Timbo Zero wrote:

I have explained as best I can as to why, you are unable or unwilling to accept.

"

Timbo Zero wrote:

I am moving on.

"

Timbo Zero wrote:

Good luck

"

sidtherat wrote:

unless that proves to create a serious dent in +income, they wont do anything about it.
<snip>
but i believe things will not change till something drastic improves

"

sidtherat wrote:

distributed system architect here

there are two potential issues at play here, both costly to fix (not hard, costly)

1) local instance instability. poe uses instances (thin poe 'clients' that run chromeless, by dozens per standard DC node). these instances are provisioned based on statistical data of prior load and usage (as instances are short lived i doubt they do live up/downscaling, so it is pre-planned and instances are pre-spun). this is done to maximize the number of instances per node. there are cases - like VERY laggy builds being popular in given league - where this statistical approach results in instances being disturbed by so called 'noisy neighbour' instances. simplified example: there are 10 instances per node. 'normal' players are using around 9% of node cpu cycles with 10% margin used for peaks (instance load is VERY non-linear). everything is fine. but if one of these players uses some kind of ward-loop idiocy with 500 casts per second and faces expedition with mobs invulnerable to the damage type it will use 30 or more % of node power STEALING it from other players. this is very noticeable for players affected. real scenario most likely operates at 50-100 instances per node
in theory it can be provisioned for - making the buffer larger or allocation logic smarter - but it costs. hosting costs are SIGNIFICANT and im sure GGG is cutting corners wherever possible. there is a reason league starts are what they are (ignoring bugs for a moment)

re-entering map MIGHT fix it, if the original instance is closed and you get loaded into newly spun on a different node. but if the problem is not severe, orchestrator might keep assigning you to the same instance/node. it is also possible that youll get 'couldnt join any instances' if the instance is already dropped/crashed and its state wasnt synced before so the backup couldnt be created

2) routing. various issues are in play here, im familiar with just some of them - but there is no secret in traffic being cut unevenly and some players having better access to busy routes than others (big players in general either use their own hardlines or pay to even bigger players for exclusivity). POE is a minor player, so small that i bet it gets the worst possible prio everywhere possible. i doubt GGG has any leverage here. there are known issues at connection points between various big networks, there are maintenance breaks of accidents (one forest fire somewhere remote could result in entire DC being downgraded due to to damaged power supply backup - these issues ripple and multiply like crazy

it is possible to request certain routings, use internal ISP landlines if possible etc - but i bet each choice has a hefty price tag attached that only the big peckers can accept.

in short - small players cannot really do much and POE despite what many here think is marginal player with pretty much no negotiation leverage. i think that the most realistic path - that GGG wont take - is to hard-limit certain idiotic mechanic in their game (like self-harm loops, cast when stunned etc) - these overwhelm their engine and make any form of load prediction guesswork. and that can only be fixed by some serious OVERprovisioning. given that GGG doesnt want to pay for it we get what we get each league 'the diarrhoea' builds are popular