Judging bias in the 2018-2019 season--statisticsss yay

[wingman]

I agree with you that one needs to gather all the data into one place. I could write codes to scrape, but if skatingscores already did this, I'd like to ask them for permission and credit them for that work. (Or to ask the ISU directly).

 

However, I would refrain from actively looking for things like "block biases" though. We should always look at the data in the most objective way as possible, free of assumptions and biases, and then just accept whatever the data tell us, regardless of whether that contradict or affirm our beliefs (of course, after making sure that we did not make silly errors in the computations or the modelling step). Otherwise, our brain is very primed to selecting and presenting convenient information that affirms our biases, and we just end up seeing things we already assumed, which I think is worse than not analyzing the data at all.

 

Just by glancing at the nice tables you made, I honestly don't find any conclusive evidence of under or overscoring. eg: if we were to zoom in at Jun's score at GPF: the judges are quite divided on him: JPN + FIN + KOR judges give him huge + scores, while three others are huge -, and two are in between. Should we conclude that one group is biased +, the other is biased -, and the "middle ground" ones are fair? No. We can conclude that there's no clear "majority" view on his skates. Doesn't mean that the + or the - are "wrong" camps.

 

While I totally agree that humans have biases in just about everything (subconscious or conscious), if you think about how these biases manifest, it's pretty hard to uncover, unless if it's fairly blatant. (which in my experience as a statistician, it's not at all obvious from this figure skating dataset). 

We (the statisticans) also have our own biases. So that's why we need lots of data + an objective analysis method.

 

 

[shanshani]

1 hour ago, wingman said:

I agree with you that one needs to gather all the data into one place. I could write codes to scrape, but if skatingscores already did this, I'd like to ask them for permission and credit them for that work. (Or to ask the ISU directly).

 

However, I would refrain from actively looking for things like "block biases" though. We should always look at the data in the most objective way as possible, free of assumptions and biases, and then just accept whatever the data tell us, regardless of whether that contradict or affirm our beliefs (of course, after making sure that we did not make silly errors in the computations or the modelling step). Otherwise, our brain is very primed to selecting and presenting convenient information that affirms our biases, and we just end up seeing things we already assumed, which I think is worse than not analyzing the data at all.

 

Just by glancing at the nice tables you made, I honestly don't find any conclusive evidence of under or overscoring. eg: if we were to zoom in at Jun's score at GPF: the judges are quite divided on him: JPN + FIN + KOR judges give him huge + scores, while three others are huge -, and two are in between. Should we conclude that one group is biased +, the other is biased -, and the "middle ground" ones are fair? No. We can conclude that there's no clear "majority" view on his skates. Doesn't mean that the + or the - are "wrong" camps.

 

While I totally agree that humans have biases in just about everything (subconscious or conscious), if you think about how these biases manifest, it's pretty hard to uncover, unless if it's fairly blatant. (which in my experience as a statistician, it's not at all obvious from this figure skating dataset). 

We (the statisticans) also have our own biases. So that's why we need lots of data + an objective analysis method.

 

 

I'm not planning on going around looking at if any random group of nationalities over or underscores skaters collectively, since obviously that would run into issues with false positives. But I think looking at Soviet bloc judging is actually worthwhile--it's a belief that a lot of fans have that is more likely to affect final standings than a single biased judge. It's possible the data will show that former Soviet countries don't actually have correlated scores, in which case that would be useful to find out since it means a popular fan conspiracy theory is wrong. But, on the other hand, I have noticed that there does seem to be some interesting alignment between, RUS and GEO/UZB/UKR/KAZ scores in particular competitions, and it would be useful to know if that's systemic or if it's confirmation bias.

 

[shanshani]

The judge database now includes all the grand prix data! yay!

 

I'm now in the process of adding Senior B data. Unfortunately, it will go very slowly and proceed in a somewhat arbitrary order (mostly based on which skaters are my fav and which Senior Bs participated in). An old copy of the database with only the Grand Prix data may be found here.