Comprehensive judge bias review

[Veveco]

37 minutes ago, Sombreuil said:

I suspect several of your critics are similarly challenged and lack the capacity for anything apart from the unhelpful knee jerk reaction we are all familiar with.

 The argument of "I don't understand (or don't want to) therefore you must be wrong" is sadly a very common reaction 

[mercedes]

2 hours ago, Veveco said:

 The argument of "I don't understand (or don't want to) therefore you must be wrong" is sadly a very common reaction 

don't get me wrong but of course if,like me,you don't have any form of knowledge of how stastics work it is hard to understand.

Those graph for example well very helpful to my ignorant brain...I'm reading everything bit by bit because I need time to go back to some part but it's clear to me it is a very honest work.

@shanshani and @Henni147 i'm sending you both a big hug and hopefully you can block these people both on twitter and youtube,they don't deserve your time at all.

While the web is a great place to comunicate and talk about common intrests I feel some people feel so confident behind their screen and fake alias that they can insult and harass right left and center ,we all know they wouldn't even dare face to face tho.

 

 

[Veveco]

Warning: long post and a lot of stats...

 

On 10/20/2019 at 3:05 PM, mercedes said:

don't get me wrong but of course if,like me,you don't have any form of knowledge of how stastics work it is hard to understand.

Those graph for example well very helpful to my ignorant brain...I'm reading everything bit by bit because I need time to go back to some part but it's clear to me it is a very honest work.

 

Your reaction is perfectly reasonable and adequate. Allow me to re-phrase my thoughts to be sure that I am not misunderstood (and expand on it as my answer here goes far beyond your specific comment): I don't blame in any way people who are not trained in stats and might find the work difficult to understand. I also don't blame people who are wondering if the author *might* be biased because a reasonable dose of skepticism is a healthy reaction in general. It'd be fine for anyone to ask for clarifications for instance. If they did so, however, they would understand that it is purely a mathematical model that doesn't take into account the author's opinion whatsoever, so "FanYU biAS" is 100% irrelevant given how the scores were computed.

 

In addition, I'd like to say that the burden of proof is always on the one proclaiming something, not the other way around. Shanshani has provided ample evidence to back up the claim of nationalistic bias in judging. Anyone wanting to disprove this notion needs to bring their own work, otherwise it's just baseless words. So, playing devil's advocate, there are 2 ways by which Shanshani's work could theoretically be biased and that people could test:

1) the data encoded are erroneous (by genuine mistake or voluntarily) in a way that would favor a certain outcome

2) the model itself is biased toward a certain type of outcome

First option is easy to check. Anyone can download the score sheets of any competition and do the calculation themselves (Excel can do most of the work for you so you don't even need any training in stats to do this) and double-check that the data is accurate.

 

Because I'm a bit of a nutcase, I took it upon myself to test the 2nd option, a bias in the model itself, to prove a point. Since the conclusion of the work claims that there is a nationalistic bias, changing the nationalities of either judges or skaters should completely change the results. Note that Shanshani already provided some data on this in a way by offering to switch a judge nationality for another "culturally similar" and see how it impacted the scores. I went one step further, and made a complete "shuffling" of skaters nationalities (more explanations under spoilers for detail).

 

Spoiler

The idea of shuffling is to randomly assign the nationalities of all skaters within a given competition (since z-scores are always calculated for one given competition). Here is a completely made-up example to illustrate the idea:

 

 

As you can see, we are just re-assigning nationalities randomly. (To be clear: I didn't choose anything myself, I did this based on randomly assigned numbers, courtesy of the "RAND" function in Excel). Names and scores don't change. Nationalities remain the same but are redistributed randomly (so it is possible that some skaters keep their true nationality by chance). As a result, the average z-score of home/other will change, either up or down based on random luck.

 

The goal of this is to test how the model reacts in a case where scores are completely independent of nationality. In other words, by assigning nationalities randomly to skaters, we artificially create a data set lacking any nationalistic bias, and thus we can check whether the model itself is biased or not. If the model finds a bias in judging (which we know isn't there because we made it so), there is an issue with the model.

 

So, what results did I get? 

The short answer is: exactly what one would expect from an unbiased model run on a random data set.

 

- The numbers of "statistically biased" judges dropped drastically:

• p < 0.05: 92 judges in the original data -> 7 judges in the re-shuffled data (out of 177 judges)
• p < 0.01: 74 judges in the original data -> 2 judges in the re-shuffled data (out of 177 judges)

The stat-lovers will notice quickly that 5% of 177 is ~9 judges and 1% of 177 is ~2 judges so we are right in the range that we'd expect by chance. This is a first indication that the model is in fact unbiased.

 

Spoiler

Note also that these numbers are lower than what Shanshani obtained in her example by switching for culturally similar countries: 18 & 9 judges, respectively. While this is not conclusive evidence by itself, it does suggests that nationalistic bias >>> cultural bias > random variation. Much more work is needed to really prove this (including a variety of "cultural combinations"), but it is an interesting result.

 

-Plotting the graphs to test for global & federation-level bias, you can see how different the results are from the original graphs (under spoilers):

Spoiler

 

From the first graph, you can see that assigning random nationalities to skaters completely abolishes the "bias" seen in the original data. Both curves are centered on zero now. Stat-lovers will also notice that the distribution for "home" scores is wider than the "other" scores, which absolutely expected given the relation between sample size and standard deviation of the samples (in brief, each judge has less "home scores" than "other scores" so there is more uncertainty/variation for the first measure than the second).

 

The 2nd graph (boxplots) shows the results for individual federations with our re-shuffled data. (I kept the same scale as the original plot for ease of comparison). I don't think I need to explain much how different this result is from the original. While there is some random variation between federations, it is very limited (no country has a median above 0.5) and the overall median (for all countries) is almost exactly at 0 (I bet no one even noticed there are 2 dashed lines - one on zero, one for the actual median observed for all countries!).

 

The 3rd graph (frequency of bias) is also very different from the original plot. From the "world" column, for instance, we can see that overall no bias (~50% of judges) > low bias > medium bias > high bias. As expected from the distribution (see graph 1). There is some variation between countries (as expected), particularly in federations with less judges, on the right side of the graph (as expected; note that countries are ranked by order of # judges, from CAN with 24 to FRA with 10 judges).

 

So, in conclusion, the stats & graphs shows that the model behaves absolutely as we could expect for a random data set and does not introduce any bias in itself. I did not change the identity of skaters, nor their score, showing that it doesn't matter here who judges are scoring on an individual level (in fact, studying this would require a different type of analysis). Only the nationality is important.

 

And the best part? No one has to trust my word on it. You can play the "change the nationality" game yourself. Even doing it for a few judges is enough to understand that nationality is what drives the results and that your individual preference for a given skater doesn't even factor in.

 

And on that note, I'll stop boring people with stats and go on with my life.

[mercedes]

@Veveco you were not misunderstood,not by me at least! I only meant it’s hard to understand but that doesn’t imply in any way whatsoever that your work is fake or biased.

And it’s hard for me because I don’t know statistics (also it’s in English so double the fun 😂) certainly not because you and @shanshani didn’t explain it properly!

again thank you for the work 

[hanyuismyBoo]

I'm new here but all these data science IS BREATHTAKING! glad to know that we have heroes like @Veveco and @shanshani to do all these tedious calculations. Thank you for this!!! now I can answer the trolls with FACTS  anyways I need to go back to my undergrad thesis 

[shanshani]

update coming soon~

[SitTwizzle]

Well, this may not be the right thread, because, even if the article is about nationalistic bias in figure skating judging (pre-2018 by the way) this is not my point today.

https://www.buzzfeednews.com/article/johntemplon/the-edge

My point is in ISU's answer :

https://www.documentcloud.org/documents/4367530-ISU-Statement.html

“Judges who make mistakes and/or are over marking skaters receive a warning and can be penalized by the ISU.”

Where can we send proofs of misjudging, for instance at last GPF? (Or the (in-)famous fake call on LGC at Worlds 2017, which is a bit old.)

And this was rather collective misjudging...