Season 3 Statistics and Demographics Thread

The objective of this thread is to analyze race data to mathematically determine:

• the extent of class imbalance in specific race types as well as races in general
• statistical measurements of true player skill with a high amount of fairness and relevance
• ranking Season 3 racers along a wide variety of such metrics (not a single list)

Unlike previous undertakings on my part to evaluate player skill, this time it is not intended to be a solo effort on my part; I want to take feedback from the racing community on metrics to measure skill. However, in order to make this fair, all bias must be checked at the door, including mine. Which means: I want opinions which are substantiated with and/or involve the manipulation of data that is both measurable (numeric and easily analyzed statistically) and verifiable (generally speaking this means available as CSV export in the corresponding Events forum thread).

Expect numerous updates to this opening post over time.

Since I'm going off of CSV files here, and GGG truncates CSVs after 2000 entries, races with more than 2000 characters entered will only have the top 2000 finishers entered for statistical analysis. Additionally, very low-level characters (such as duelists which players use simply to get Cleave for their templars) will be trimmed from the data.

DESCENT RACES

(as of D060)

• Descent Class-Adjusted XP per Race Ladder (as of D060).
  "Per race" means that races you do not participate in do not count against you; instead, an average is taken of the results of the races you do participate in.
• Descent Class-Adjusted Race Results (as of D060).
  Newest events sorted to the top.

COMPETITIVE SOLO RACES - Deaths count as zero XP here.

• Competitive Solo Class-Adjusted Race Results (as of C038)
  Each event is listed in a separate tab.

S03D007 numbers up.

Descent Metagame Shifts:

• Population-wise Duelist and Ranger numbers stayed pretty steady, while there was an exodus away from Templar and into Witch and Shadow. Marauder numbers declined slightly.
• Performance-wise it seems those who were trying out Shadow for the first time didn't like what they found, since Shadow performance tanked. Marauders and Rangers both found how to make things work better, with average Ranger performance actually outshining average Duelist performance this time around. Duelist performance dropped slightly, while Witch and Templar performance stayed near previous levels.

"

CharanJaydemyr wrote:

The fact that the non-Descent populations are much more evenly distributed is a clear indication to me that it's not just the excellent Duelist starting nodes, but the skills and gear.

If it weren't for the very first race having a tremendous duelist bias, I might be inclined to agree. For some reason a huge number of players assumed duelist would be the way too go with no concrete evidence to make that decision, and since they were essentially correct, there aren't many players leaving to try greener pastures; if people really do flock to the OP, we'll be seeing more rangers in the next one.
-------------------------------------------------------------------------------------------
S03C012 results are going to be delayed as I ponder automated (Excel formula-based) methods for weeding out single-point farmers. The problem is that several players have taken to leveling a single account to level 5 (worth 1 reward point), then logging out and starting a new character, getting that to level 5, etc. Since these folks are playing several shorter races rather than the full duration, they slant the average results towards the low end of the spectrum; previously, I was filtering out all but their highest-scoring character, but I intend to filter them out entirely. I'm not saying it's a despicable practice or anything, but it does run contrary to the assumptions behind my statistics.

Very nice work, Scrotie!

It seems like there are two interesting things these charts are attempting to capture: class preferences and class balance. The tools to capture this are population ratios and exp ratios.

I propose a tweak on the way the class balance is captured. The issue is that the top 2000 gives average exp results that are fairly low relative to where the strong racers compete, and the top racers is where class balance really matters. Having a lot of low-level finishes doesn't mean the class is weak; just popular. When the class distribution at the top and bottom is different, the average skews toward the larger pool. Normalizing experience (Rank 1 = 100% exp, everyone else a ratio of Rank 1's exp), the class-adjusted averages so far have been around 15-30% of the total, so the averages are skewing toward the bottom. The presently implemented class-adjusted results have not been divorced from popularity.

I suggest comparing average exp of top 100 entries for each class. That would give a better idea of how the classes compete among strong racers, thus giving a better picture of class balance. So the change is in how you filter data- instead of 2000 data points, prune to 600 data points. You still have population distributions to evaluate class popularity in a separate table. Class-adjusted exp is partitioned from that to be more focused on class balance with less influence from popularity. Being more popular does not give the class a greater share of the 600 data points so averages should have less random noise based on the noise of how popular a class is among low level players who are not yet competitive. Instead it is the best 100 of each class competing against the best 100 of each other class.

"

PolarisOrbit wrote:

I propose a tweak on the way the class balance is captured. The issue is that the top 2000 gives average exp results that are fairly low relative to where the strong racers compete, and the top racers is where class balance really matters.

Is it?

Is it good game design if two classes are well-balanced at the very highest echelons of play, but when put in the hands of the average player, the first consistently overpowers the second?

For the typical viewer of this thread who is deciding which class to pick, assuming that they are a top racer is actually a horrible assumption. It's much safer to make as little assumption on their skill level as possible, which is why that's precisely what I've done. I intend this thread to have the broadest utility to the broadest audience; to me, that means including both the low-end results and the high-end results, excluding neither, and thus being neither elitist nor populist.

Of course, that's all relative, and elitists (top racers and the fans who watch them) are inclined to see moderate views as populist, and populists (those who find racing too difficult and think Reward Points should come easy) are inclined to see moderate views as elitist. And to that I say: go ahead. Make your own metrics; you can post them here or in a thread of your own. I'm not going for a niche audience this time, but if you are, that's Kool and the Gang, brother.

However, I did discover a different and far more fundamental biasing results towards the low end of the spectrum: so far I've been doing Descent class balance by character, which is a flawed assumption. When a single account has more than one character, the amount of race time each character is allocated is less than an account with only a single character in the standings. I've developed a new formula architecture to count each character fractionally based off the account's total XP, and also record multiclass selections; for example, if one account has both a ranger with 140000 XP and a duelist with 60000 XP in the same descent race, assuming at least one is dead, they will now be counted as 0.7 rangers and 0.3 duelists, where previously they had counted as one of each. It's not mathematically perfect (time played isn't available on the CSV), but it's more accurate, removing some low-end skew from descent races past the first (which, if I remember correctly, had unique accounts for each character).

I also came up with a fix to the single-point-farming problem.

I took a foruming break from redoing all the previous race results with the new architecture, so a massive update (including revisions to previous data) will be up within a day or so; I'm probably going to sleep before completion.

edit: Descent updated to current. I'll get on the comp solos later.

"

ScrotieMcB wrote:

"

PolarisOrbit wrote:

I propose a tweak on the way the class balance is captured. The issue is that the top 2000 gives average exp results that are fairly low relative to where the strong racers compete, and the top racers is where class balance really matters.

Is it?

Is it good game design if two classes are well-balanced at the very highest echelons of play, but when put in the hands of the average player, the first consistently overpowers the second?

I guess you could say that this is the difference between which class is best and which class is easiest?

Added D019, and updated the Average CAXP Ladder; due to the large number of single-race high scores, I have eliminated all accounts with only a single placing from the ladder (you need to play in at least 2 races to get on the list).

"

ScrotieMcB wrote:

Is it good game design if two classes are well-balanced at the very highest echelons of play, but when put in the hands of the average player, the first consistently overpowers the second?

For the typical viewer of this thread who is deciding which class to pick, assuming that they are a top racer is actually a horrible assumption. It's much safer to make as little assumption on their skill level as possible, which is why that's precisely what I've done. I intend this thread to have the broadest utility to the broadest audience; to me, that means including both the low-end results and the high-end results, excluding neither, and thus being neither elitist nor populist.

Well that's two separate things, and you are capturing it with one number.
-Class Balance (according to top tier strategies)
-Class Learnability (according to average tier strategies)

Having top-tier players in the pool pollutes the numbers on Learnability, and having low-tier players in the pool pollutes the numbers on Balance.

I appreciate that the formula is very elegant and mathematically sound, but ask yourself what that formula really tell us, and what are the questions we would like to answer?

Anyway it's your spreadsheet do as you will. If I felt really strongly about it then I would make my own competing sheet. Obviously I'm not going to do that =D

"

PolarisOrbit wrote:

Well that's two separate things, and you are capturing it with one number.
-Class Balance (according to top tier strategies)
-Class Learnability (according to average tier strategies)

I don't believe that even top players stop learning, and I don't believe that there's no such thing as cookie-cutter racing (top tier strategy in the hands of the less skilled); the latter is especially true in the age of streaming.