Comprehensive judge bias review

[WinForPooh]

1 minute ago, shanshani said:

yeah @veveco did us a great service by showing in graphical form just how bad it is

 

Absolutely, thank you @Veveco for putting in that time and effort, too! Seriously what you @shanshani are doing is what ISU should be doing if they had any interest in making this a fair and just sport. They should be paying you excellent money for all this analysis, they could use this to give the sport some credibility it really needs rn. I don't even care if the results of it show that I've been too narrowminded in condemning the lot of them as biased old dinosaurs (my apologies to dinosaurs), I really, desperately need this data and analysis to see what's going on without being influenced by my own (admittedly grim) perceptions.  

 

Like, I have so many questions: unless there's an 'everybody go home figure skating won' performance like Yuzu GPF '15 or maybe Wakabond at Worlds, do American and European judges score the subjective components low for Asian skaters? Do non-Soviet European judges score Russian skaters lower than Asian judges? How much of the scores are actually decided by outdated cultural perceptions and politics that have nothing to do with figure skating? 

 

I'm not saying I expect you to answer those questions  but I haven't seen anybody even begin to compile data that could be used to answer such questions at some point, this could be the starting point of something really big. 

[Veveco]

Keep in mind that it is a somewhat small sample size for some federations (10 judges minimum) but we can see some patterns emerge.

 

I'm glad that Australia is at least showing that avoiding systematic bias is possible (yes, standards are really so low that I'm happy with even one good student...) but I'm far less impressed by my fellow European federations (other than Finland which was on the reasonable side at least). I don't know what's worse: Russia's consistent-but-somewhat-moderate bias or France/Italy/Germany's variable-but-with-nearly-50%-of-high bias ?

 

There are so many more questions I want to ask too. Comparisons between big and small competitions, comparisons between disciplines, comparisons between scoring medal contenders vs lower ranked skaters.... But for those we need to go back to the individual scores (not the judges' averages that @shanshani took time to compile) so it'd be even  more work (so not for today but I'll keep it in mind for the future)

[shanshani]

Now in webpage format: https://fsjudging.wordpress.com/2019/10/17/judging-bias-and-figure-skating-part-one-nationalistic-bias/

 

If you want to share on twitter, probably best to share this link instead of a forum post.

 

Also someone should totally screencap the wall of shame and tweet it at the ISU.

 

Here's the tweet with the link, if people who are more on twitter than me want to retweet:

 

 

https://twitter.com/shanshaniii1/status/1184801032863272960?s=2

[shanshani]

@yuzuangel if you could actually retweet it, I would be very grateful. or get someone who has a big twitter following to do so. I'm useless at twitter haha

[yuzuangel]

2 minutes ago, shanshani said:

@yuzuangel if you could actually retweet it, I would be very grateful. or get someone who has a big twitter following to do so. I'm useless at twitter haha

should I just retweet it or retweet it with a comment? If so what comment?

[shanshani]

Just now, yuzuangel said:

should I just retweet it or retweet it with a comment? If so what comment?

"We now have solid evidence that the judges are rotten" would be the click-baity thing to do haha

 

"Statistical analysis shows what we knew all along: the judges do not judge fairly" is slightly more restrained

[mercedes]

8 hours ago, shanshani said:

yeah @veveco did us a great service by showing in graphical form just how bad it is

Thank you @shanshani and @Veveco,the graph part really makes easier to understand it fot me.

[rockstaryuzu]

3 hours ago, shanshani said:

"We now have solid evidence that the judges are rotten" would be the click-baity thing to do haha

 

"Statistical analysis shows what we knew all along: the judges do not judge fairly" is slightly more restrained

Or we could go with "Statistical Analysis of Judges Performance: Bias or No Bias", which is even more neutral-sounding. It'd be bad if all this good work got dismissed as being biased, itself. 

[shanshani]

Omg an actual judge (one of the ones that don't show evidence of bias) is following my blog

[Paskud]

1 hour ago, shanshani said:

Omg an actual judge (one of the ones that don't show evidence of bias) is following my blog

Congrats! I hope your work will get even more recognition, because it's really amazing!

[Veveco]

I just read some of the comments on the web version and....  Some trolls don't even bother reading the methodology before crying for "fanyu bias", do they? (and as such, display obvious bias themselves, ironically enough). The analysis is done for all skaters and all major competitions, and it is completely blind to the identity of the skater being judged (other than the nationality) so whoever the author likes doesn't even enter ANYWHERE in the mathematical model. It's not about the *right* score, it's about statistical trends of scoring compared to the average score. You'd get the exact same result if the analysis was done by a Nathan or Shoma fan  This actually makes me really mad.

You handle them with more grace than I ever could @shanshani . You have provided all the methodology, as well as the data so anybody can verify your work. (Likewise, I'm perfectly willing to share my R codes for the graphs should anybody ask for them). So unless they do the work and find a flaw somewhere (which they haven't), they have literally no ground to call for bias in the methodology here. I'd be a different story if somebody came with a different statistical model to have an actual debate.

To be fair, I've also seen a lot of positive reaction online (and some being very offended at troll comments), but it still makes me mad on your behalf. So I just wanted to thank you again for your hard work and your patience 

[shanshani]

24 minutes ago, Veveco said:

I just read some of the comments on the web version and....  Some trolls don't even bother reading the methodology before crying for "fanyu bias", do they? (and as such, display obvious bias themselves, ironically enough). The analysis is done for all skaters and all major competitions, and it is completely blind to the identity of the skater being judged (other than the nationality) so whoever the author likes doesn't even enter ANYWHERE in the mathematical model. It's not about the *right* score, it's about statistical trends of scoring compared to the average score. You'd get the exact same result if the analysis was done by a Nathan or Shoma fan  This actually makes me really mad.

You handle them with more grace than I ever could @shanshani . You have provided all the methodology, as well as the data so anybody can verify your work. (Likewise, I'm perfectly willing to share my R codes for the graphs should anybody ask for them). So unless they do the work and find a flaw somewhere (which they haven't), they have literally no ground to call for bias in the methodology here. I'd be a different story if somebody came with a different statistical model to have an actual debate.

To be fair, I've also seen a lot of positive reaction online (and some being very offended at troll comments), but it still makes me mad on your behalf. So I just wanted to thank you again for your hard work and your patience 

Thank you! It's actually really stressful dealing with some of the comments because of how it blew up. I basically skipped watching Skate America because I had to take a short break from the FS community and figure skating in general And I basically stopped looking at GS after a few hours because I decided it was beyond my emotional bandwidth to deal with it. Support is really helpful.  

[Veveco]

4 minutes ago, shanshani said:

Thank you! It's actually really stressful dealing with some of the comments because of how it blew up. I basically skipped watching Skate America because I had to take a short break from the FS community and figure skating in general And I basically stopped looking at GS after a few hours because I decided it was beyond my emotional bandwidth to deal with it. Support is really helpful.  

[Sombreuil]

Just caught up with this - I can’t wrap my head around the methodology because I have problems with maths but it’s very impressive.  I suspect several of your critics are similarly challenged and lack the capacity for anything apart from the unhelpful knee jerk reaction we are all familiar with.  It was the same with the analysis of H&L that brought me here - plenty of unsubstantiated somewhat hysterical criticism but no solid analysis of their own or of their favourite skaters performances to challenge what was being said.  I hope you are able to keep enjoying FS while updating your figures as the season progresses.  TBH catching up with the few good performances at SkAm will probably be better for your blood pressure than watching the whole thing, especially with US commentary  

[Henni147]

10 minutes ago, shanshani said:

Thank you! It's actually really stressful dealing with some of the comments because of how it blew up. I basically skipped watching Skate America because I had to take a short break from the FS community and figure skating in general And I basically stopped looking at GS after a few hours because I decided it was beyond my emotional bandwidth to deal with it. Support is really helpful.  

 

My biggest mistake this weekend was to read some comments on ISU's official YT Channel. It caused me physical pain 

 

I can 100% understand that you skipped SA. I really need a break, too. However, please don't let yourself down by some ugly/annoying/stupid comments. Your work is amazing and very valuable for the integrity of this sport. 

 

#Footnote: Some sick haters attacked my videos on YT and wanted me to delete them. At the beginning I tried it with friendly, diplomatic responds, but I quickly realized that they don't deserve a single minute of my time and attention. I rather invest it in new quality content and try to please all those people who appreciate my effort. It's really not worth it to argue with vicious idiots, especially if they do not even take the time to read your work.