shanshani

eh...that's not really true. Japanese judges are better than your average federation's judges, but many of them are still pro-JPN biased. You can read about it here:

Now in webpage format: https://fsjudging.wordpress.com/2019/10/17/judging-bias-and-figure-skating-part-one-nationalistic-bias/ If you want to share on twitter, probably best to share this link instead of a forum post. Also someone should totally screencap the wall of shame and tweet it at the ISU. Here's the tweet with the link, if people who are more on twitter than me want to retweet: https://twitter.com/shanshaniii1/status/1184801032863272960?s=2

yeah @veveco did us a great service by showing in graphical form just how bad it is

Updated the OP to include some lovely graphs courtesy of @Veveco. @WinForPooh these graphs may be of special interest to you, as you seem to be interested in federation-level bias

JPN judges play the game less than the other big feds, but there are still a substantial number of biased Japanese judges. Whether they would have been assigned is another question though. I actually didn't find any evidence that Mexican judges are biased in favor of US skaters in my judge bias project. I haven't posted the document where I switched around the nationalities of judges yet, but I finished about 75% of it, including both Mexican judges, and they scored US skaters and non-US skaters about the same on average. Regardless, I really think the ISU should have a policy where they either have the judges of all top competitors on the judging panel or none. The fact that one medal contender would get a friendly judge but not others will rightly make people doubt the objectivity of the panel, especially when so many judges are demonstrably biased.

That would be awesome! Although let me do some edits first. I'm also going to put it up on wordpress/a similar platform so it might be easier to tweet that? Also you should definitely tweet the wall of shame part at the ISU. lol

better to correlate with ZDifference, because p is necessary correlated with the number of competitions as it measures the strength of the evidence. more competitions = more evidence and sure, I can take a look, at least for a few specific feds.

Oops I forgot to upload the updated version of the database. The old one doesn't include Finlandia. Should be fixed in the next 5 minutes Thanks for the questions! 1. It's not entirely meaningless, but it's also subject to a lot of noise. There's clearly something to be said for the fact that 80%+ of the Russian judges are flagged, but overall the % of judges from a federation that are flagged is as much a function of how much data are available on the judges, specifically regarding how they score their own skaters. It's easier to detect bias the more data there is (with really small sample sizes ZDifference has to be very large for the judge to get flagged), so it's relatively easy to catch Russian judges, who have many opportunities to score many skaters across many disciplines, but harder to catch small fed judges, who have far fewer opportunities to score their own skaters. Russia in particular appears to have a relatively small stable of judges who go to a lot of competitions each, making it easier to catch them even in comparison to, say, US and Canadian judges who tend to go to fewer competitions per judge. Nonetheless, I do believe Rusfed is genuinely worse than most other federations, but the % of flagged judges exaggerates the difference for reasons that don't have much to do with the inherent bias (although I would say it's less that Russia's level of bias is overstated and more that other feds' level of bias is understated). 2. I could definitely take a look at that, although presently the sheet isn't built to make that easy. The easiest way to do it is just to average by judge (and maybe I'll add that to the sheet--shouldn't take too long), but it's probably better to aggregate the underlying data, which would take a bit of extra time to do. It's a good idea though, and I could add it as a supplemental at some point. It would also be helpful for looking at federations where the amount of data on each individual judge is small, but aggregating it produces a decent amount of data. 3. Hm, well there's no reason in principle this should be impossible, but I would have to think a while about how to do it in a way that doesn't take forever. Calculating the degree a skater is likely to benefit from bias judging sounds like an interesting idea. It might be possible, but I would have to think quite hard about how to do it. One complication is that there might be other kinds of effects (eg. are judges more likely to be more biased in favor of the higher ranking skaters from their country?) which would make it difficult to generalize. You would also ideally take into account the trimming procedure used to calculate the actual score, and how the score would interact with placements (but I think this would be too difficult to account for).

It would definitely be quicker to list the judges who despite having a substantial judging record look pretty unbiased (p > 0.3). But less satisfying . I don't think literally every single judge is biased though. Well at least you aren't Russia, that section is a wall of deep red. The Russian judges who weren't flagged (all two of them) were almost certainly because of low sample size. One of them is reallyyy borderline as well, like p=0.0504 or something.

It's possible but would require a lot of fiddling. My suspicion is that you won't find anything (except for Russia related countries because bloc scoring)

It should be possible, but alas I did not structure the excel file in a way such that it would be easy. My impression is that ice dance is really more biased. One slight hiccup is that because the score differences are normalized by competitor (and therefore by discipline), the extent of the bias is hidden a bit as information about the raw scores is lost. But even then, ID judges are the worst.

Well, I only looked at one kind of judging bias, so I can only confidently say whether a judge is “good” or “bad” in that particular area. It's possible that a judge is biased in one way but not biased in another (for instance, in one of the competition threads, someone mentioned that Adrienn Schadenbauer seemed to stick more closely to the guidelines than the typical judge, though we can see that they are a pretty biased judge in terms of national bias). Although I do think being biased in one way should raise questions about a judge's commitment to objectivity in general. Unfortunately, it's a lot harder to come up with ways to test other kinds of bias--reputational bias, for instance, seems like it would require someone to make judgments about how a skater ought to be scored and compare the judges actual score to that. I don't necessarily think there's anything wrong with that, but it will naturally be easier to attack than a subjective-judgment-free method like the one that I used here.

It's probably a sign of mild insanity that I did all of this, but I'M FINALLY (sort of) DONE (with part one), HERE IS THE JUDGE CALL OUT POST YOU'VE ALL BEEN WAITING FOR it has statistics. a lot of statistics. and names. anyway, I hope you guys read it and let me know if any part is unclear or requiring further explanation, so I can clean up the writing/formatting/whatever before I try to get people outside this website to read it too I guess it's only mildly Yuzu related though forgive me for hijacking this thread edit: as someone recommended to me, I should probably host it as an article on a separate website as well. does anyone have any recommendations?

Reserved space

ehh, screw it, let's publish. comprehensive review of judging bias here would welcome comments before I try to disseminate this more widely

Thank you @yuzuangel@rockstaryuzu@Old Cat Lady@Veveco@Pepipanda@river@Dori6886 for your help!

Part 3: Judges against Rival Skaters Forthcoming, check after the Grand Prix

Part 2: Bloc Judging Forthcoming

Limitations/Other considerations Despite the fact that so many judges show evidence of nationalistic bias by the methods used here, I actually think that they are somewhat limited in their ability to catch nationalistic bias. (Which ought to indicate how bad the problem is.) First, a judge can quite easily avoid detection by only being biased “when it counts” ie. in only a selected number of competitions, when there are medals or spots at stake and a tweak in judging can make the difference. Because this bias will be washed out in the average with all of the other competitions where the judge was not being biased, this type of bias, if detectable at all using these methods, will only be so after a judge has built up an extremely substantial judging record. Another related type of nationalistic bias this metric is not good at catching is when judges selectively underscore only direct competitors, but score everyone else normally. Because all non-home-country skaters are averaged together, this type of bias has little impact on the overall ZDifference, and consequently it is very difficult to detect it using the method here. I hope to address this in a future segment which will examine whether judges from federations with top competitors underscore the direct competitors of their skaters. Stay tuned for that. Thirdly, if there is bloc judging going on, or any other score-trading or collusion scheme to increase a skater’s score, it will function to weaken the evidence for biased judging by reducing a biased judge’s apparent difference with the other judges on the panel when scoring their own skaters. On the flip side, if there is a score-trading or collusion scheme to lower a competitor skater’s score, that may wrongly introduce apparent bias on the part of the judge from the home country of that skater. (Nonetheless, I don’t think this creates a major concern about wrongly flagging judges, because unless there is some grand conspiracy against all of the skaters from a certain federation, one instance of apparently biased judging will be washed out when averaged with the rest of the scores that judge has given.) However, we can use this same dataset to take at least a partial look at bloc judging, and we will do so in a forthcoming post, so stay tuned for that too. Finally, it is also possible for judges to “game the system” by overscoring non-direct competitor skaters, thereby inflating the “Z-other” portion of the calculation. I don’t think this is a major concern now, but if somehow this were adopted as the primary means of track judges’ bias, then it would be a concern in the future. (Although they would chiefly do this by overscoring lower ranking competitors, which might actually be a good thing, since it would combat reputation bias). Conclusions There is, as always, more to say than I have said, but as I don’t want to spend literally forever on this or produce something so long no one wants to read it, I will end it here and leave it to others to raise questions if there is a gap you would like me to fill. The overall conclusion is pretty clear: figure skating judging has a massive problem with nationalistic bias, and many judges are extremely blatant in their favoritism for their own countries’ skaters. This also raises questions about judges’ commitment to objectivity in general. Though we have only looked at a very specific type of bias, and arguably not even the most significant one (just the easiest one to tackle using statistical methods), one might suspect that lack of objectivity in one respect bleeds in to lack of objectivity in others. What about other forms of bias that are also often discussed, like reputation bias and big fed bias? If judges are so demonstrably biased in one way, it seems reasonable to suspect that they are also biased in others.

Discussion First, the most obvious and basic question: what does bias mean? Here, I’ve been using bias to mean a demonstrable, mathematical difference between how a judge scores their own skaters and other skaters. By claiming a judge is “biased,” I don’t mean to impute anything in particular about their psychology, nor am I making any claim about the origin of the bias. It may be conscious, or it may be unconscious. It may be a deliberate attempt to manipulate the results, or simply the same kind of lack of objectivity fans often display concerning the skating of their favorite skaters. Personally, I am not overly concerned about the causes of the bias, only that it exists--after all, let us remember that these judges, through their scoring, determine young athletes’ futures. I don’t know about you, but I don’t want the future of these young, incredibly hard-working people to be determined by a group of people who are unable to be objective, whether that lack of objectivity is the result of corrupt intent or simply clouded judgment. However, let me address some specific explanations for bias which I do not believe are true or at least have problems, as well as other attempts to defend the judges from criticism. 1. The bias is just due to cultural preferences. People tend to look more favorably upon programs they are culturally familiar with and score them higher, and obviously a judge and a skater from the same country are more able to culturally understand each other. First, skaters from the same country often have very different skating styles and skate very different types of programs, so it strains credulity to believe that there is something quintessentially “Russian” or “Canadian” or whatever about all skaters who skate under the same flag. Sasha Trusova’s programs look completely different from Alina Zagitova’s which are completely different from Alena Kostornaia’s, and they’re even coached and choreographed by the same people! Second, if this were true, we would expect to see judges from culturally similar countries scoring each others’ skaters higher. For example, Canada and the US are two extremely culturally similar countries, so Canadian judges should overscore US skaters and vice versa. Fortunately, the sheet is built such that it’s easy to test this proposition (just change the country code inside each judge’s individual page to see how they score specific other countries’ skaters) and in fact, we do not see this. The vast majority of Canadian judges score US skaters like a unbiased (or even anti-biased, ie. biased in the other direction) judge would, and so, too, the other way around. (You can confirm this yourself by downloading this sheet here, which has each judge's nationality switch to that of a large, at least somewhat culturally similar or at least geographically close figure skating country. Or you can switch around judges' nationalities yourself on the original sheet, and it will recalculate all the stats for each judge (you have to do this inside each judge's individual sheet, however, not on the summary sheet). There are a few exceptions to this (former USSR countries’ judges tend to score Russian judges higher, although the level of bias is not quite as severe as it is for their own skaters. Also, I believe South Korean judges score North Korean skaters higher, but whether those two countries have similar cultures seems quite debatable. We will look at this in more detail in a future post), but I believe there is a better explanation for the exceptions. In general, culturally similar countries do *not* score each others’ skaters higher. 2. It’s just human nature to be biased. We should realize that judges are humans too and not robots. Judges are not all the same. They do not all show evidence of bias. For instance, Glenn Fortin (CAN), Katharina Heusinger (GER), Andreas Waldeck (GER), Ayumi Kozuka (JPN), Shizuko Ugaki (JPN), Linda Leaver (USA) all have reasonably substantial judging records that do not evince any substantial evidence of bias. This clearly indicates that it is possible for judges to be unbiased, at least when it comes to nationality-related bias. Evidently, not all human have this particular human-nature related flaw. Even among the judges who are biased, there are considerable variations in the degree of bias shown. The worst offenders, for example Salome Chigogidze (GEO), Nicholas Russell (GBR), and Elena Fomina (RUS), have ZDifferences in the range of 1.5-2, whereas the lowest statistically significant differences are in the range of 0.5. This shows that it is certainly possible to reduce the level of bias of the judges overall by getting rid of the worst offenders and replacing them with less biased judges, even if some low level of bias (say, less than 0.5) is difficult to get rid of and may not be practically significant enough to be worth dealing with. 3. Your metric looks at judges’ bias by comparing them to the mean of the judging panel. Doesn’t that assume that the mean of the judging panel is right? But sometimes it’s the outlier judge that is right, and the other judges that are wrong. It may be true that the outlier judge is indeed “right”--I avoided making any assessment of what a skater “should objectively” have scored because those types of assessments lead to unproductive fan wars and involve a level of personal judgment that I did not want to introduce to this study. However, let me note that outlier scores only “count against” judges if they align with expected patterns of nationalistic bias. If a Japanese judge scores a Filipino skater way above average because only that judge was being objective and the other judges all underscored him due to some other form of bias (reputation, small country, etc.), then that will actually count very mildly in favor of the Japanese judge. Only if a Japanese judge scores a Japanese skater way above average does it count “against” that judge. But in that case, there are still at least 3 other data points to consider (I only start calculating p if a judge has scored her own skaters at least 4 times), and if the judge only shows a pattern of “correcting” the scores for her own skaters, one has to begin wondering whether they are truly being objective. Again, judge is not labeled “biased” for having scores that deviate from the mean, a judge is labeled “biased” if there is a difference between how they score their own skaters and other skaters. A judge who scores both groups 2 standard deviations above the mean would not trigger the flagging formula, despite having scores that are way out of whack with the other judges. It is only the difference between the judges own scores for her skaters and other skaters that matters. The only situation in which I think this may be a substantial concern is for judges from a small federation who have only judged one or two unique home country skaters. In that case, it’s possible that a personal (rather than nationalistic) preference, or a genuine belief that a particular skater (but less a large group of skaters, as that would affect Z-other as well, and thereby decrease ZDifference) is underscored, which just so happens to coincide with a national flag, gets “wrongly” flagged as nationalistic bias. This being the case, we may wish to be a little bit more lenient on small-fed judges. However, this defense hardly applies to the judges of large and powerful federations like Russia and the United States, who will judge many different skaters from their own country through their judging career, and whose skaters cannot credibly claim to be underscored because of their nationality.

Methodology The basic idea behind determining judge bias was to take each judge, compare their scores for each skater to the scores the rest of the panel gave, quantify how far off the panel judge was, and then put all the numbers together in one place so I could compare how a given judge scored their own skaters in comparison to the rest of the panel, versus how they scored other skaters. I then ran a standard statistical test on the difference in order to figure out the strength of the evidence the judging record provides of bias. In order to do all this, I started by recording all the scores given by all the judges in every senior-level international competition (excluding competitions like Japan Open and Shanghai Trophy) since the scoring system changed in 2018. This gave me a bunch of spreadsheets which you can find here. (Fortunately, I had some help at this stage, thanks people at planethanyu!) If you open these spreadsheets, you’ll notice they look something like this: Odhran Allen Doug Williams Maria Fortescue Veronique Verrue Andreas Waldeck Lorna Schroder Miwako Ando Mean IRL USA ISL FRA GER CAN JPN Yuzuru Hanyu JPN 164.01 157.34 172.44 161.55 166.7 167.4 170.74 165.74 (This example from the 2018 ACI men’s free skate is completely random, obviously.) As you can see, it lists judges, judge nationality codes, skaters, the skater’s nationality code, and the scores given by each judge, as well as the average of all the judges’ scores. From this data, I first determined how much higher or lower each judge scored each skater compared to the rest of the panel by subtracting the mean of all the scores from the individual judge’s score. This produced what I called the score deviation. Using our friend Yuzu as an example here, this produces: Odhran Allen Doug Williams Maria Fortescue Veronique Verrue Andreas Waldeck Lorna Schroder Miwako Ando IRL USA ISL FRA GER CAN JPN Yuzuru Hanyu JPN -1.73 -8.4 6.7 -4.19 0.96 1.66 5 So Odhran Allen scored Yuzu 1.73 points below the other judges, Doug Williams 8.4 points below, etc. This is what’s shown in the second block on the competition sheets. Now, unfortunately, we can’t just leave it at that, because we want to compare how judges score skaters across competitions, segments, and disciplines, in order to get the largest, most robust data sets. -8.4, while already a lot even in the men’s free, would be absolutely massive in the short, and also bigger in ladies or pairs than in men’s. Therefore, in order to make these score deviations more comparable, I had to standardize them into z-scores. Note that this is an extremely common method of standardizing data. Here’s how it’s calculated: first, I have to determine something called the standard deviation of the judges’ scores. Standard deviation is another one of those common statistical measures, and it quantifies how spread out a set of numbers is from the average of those numbers. So if judges are all over the place on someone’s scores, then the standard deviation will be relatively high, whereas if judges are all more or less in agreement, the standard deviation will be low. In this case, the standard deviation of Yuzu’s scores was a fairly typical (in men’s) 4.85. To calculate the z-scores, we just need to divide each of our score deviations by the standard deviation of Yuzu’s scores. One way to think about the z-score, then, is that it tells you how many standard deviations a judge scored a skater above or below average. Here is what happens to Yuzu’s score deviations once they’re converted into z-scores. Odhran Allen Doug Williams Maria Fortescue Veronique Verrue Andreas Waldeck Lorna Schroder Miwako Ando IRL USA ISL FRA GER CAN JPN Yuzuru Hanyu JPN -0.36 -1.73 1.38 -0.86 0.2 0.34 1.03 Z-scores typically range from -2 to 2, though occasionally you’ll see numbers outside that if a judge *really* disagrees with the other judges (this occur at roughly a 5% frequency). Underscoring (ie. scoring below the other judges) turns into a negative z-score, while overscoring (ie. scoring above the other judges) turns into a positive z-score. Using z-scores actually build in a measure of leniency for the judges. If there’s a lot of disagreement within the panel about a skate, then the z-score will be less extreme than the raw score difference, so a big difference with the average will “count” less. On the other hand, it does mean that if there’s a lot of agreement among panelists, someone who is a lone outlier may have a more extreme z-score in comparison to the raw score difference. But overall, the z-score makes it a bit easier for biased judges to hide. Oh well, it’s not like they hide very well in the first place. Once these z-scores are computed for each judge, competition, and segment, all of the z-scores associated with a given judge are collected together into one sheet, the individual judge’s sheet. You can find these in the big judges database. If you click on any judge’s name in the sheet labeled “Judges”, you’ll be taken to the individual sheet for the judge you clicked, where you can see the collected z-scores for all the competitions they’ve judged. On the left you’ll see a bunch of summary statistics. I’ll explain those in a second. On the right you’ll see z-scores. As you can see, they are labeled by skater and nationality, and at the top there’s a code which tells you which competition and segment is being shown in a given set of columns. This is composed of [Year][Competition Code] [Segment Code]. Using a formula, all of this data is split into two groups--z-scores for home country skaters and z-scores for other skaters. The z-scores for home country skaters are averaged, producing Z-home on the left. Same thing for the z-scores for other skaters, producing Z-other. The difference between the two, which is what we’re ultimately interested in, is then calculated as ZDifference. (You’ll also see these metrics in the overall judge summary). You can think of the ZDifference as representing the degree of bias that a judge has shown to home country skaters. In terms of raw score as a rule of thumb, a ZDifference of 1 represents about 7-8 points in men’s, 6ish in ladies and pairs, and 6-7 in ice dance over the course of a competition. In other words, a judge who has a ZDifference of 1 will give, on average, 7-8 bonus points in men’s, etc., versus what they would typically give a skater who is not from their home country. Of course, the degree of bias shown is not the only thing that matters when it comes to assessing a judge’s level of bias. If a judge shows a ZDifference of 1 but has only judged their home country skater a couple of times, it’s possible that that ZDifference is simply due to random chance or other factors. On the other hand, if the ZDifference persists across many competitions, we can be much more confident that the judge is biased. This is where the metric p comes in. p is another standard statistical measurement, and in the context of our data it represents the chance out of 1 that an unbiased judge, ie. one that scores home country skaters no differently than other skaters, could arrive at a record that evidences equal or greater bias than the actual judge’s record purely by accident. So in other words, the lower p is, the higher the chance that there is some kind of systematic difference between how a judge scores home country skaters and other country skaters. By convention, a p value below 0.05 is considered statistically significant, and that is the standard I will be using to flag judges, though in many cases we’ll see that p will be far below that threshold. For instance, in the case of Russian judge Olga Kozhemyakina, p=0.000000000000003 (note that that’s a 0.0000000000003% chance an unbiased judge would produce a record equal to or worse than hers). It’s better not to see statistical significance as a binary thing, however. Instead, you should become more and more suspicious of a judge as p drops. Notice that many judges whose records were not flagged nonetheless have fairly low p values--I suspect that many of these judges’ records will start getting flagged as more scores start coming in. By considering ZDifference and p jointly, we can make a full assessment of a judge’s judging record. ZDifference tells you the severity of the historical bias, whereas p tells you the probability it came about by chance. I used a standard statistical test for the difference between two means, the Welch's t-test, in order to calculate p. I used the one tailed version of the test, because we’re only looking for bias in one direction. Welch’s rather than Student’s was used because I noticed that judges with extensive judging records tended to have different variances for home country skater scores versus other scores. (If you didn’t understand this paragraph, that’s okay--unfortunately, it requires a lot more effort to explain how calculating p works in depth, so I will have to pass on doing that. If you would like to learn, I would recommend you take an introductory statistics class.)

Webpage version As the Senior Grand Prix season is about to begin, I thought now would be an appropriate time to take a systematic look at the judging records of our judges. Despite the fact that figure skating judges play a significant role in determining the career trajectory of young athletes, it is well known that they face little accountability. Only the most egregious cases of biased scoring are ever punished, in spite of the fact that biased scoring is rampant and well-documented. Nor does the International Skating Union (ISU) appear to keep any kind of running record of judges’ performances, so judges avoid all scrutiny as long as they keep their bias to a moderate amount at every competition rather than an extreme amount at a single competition. In order to improve accountability, then, I’ve decided to compile and analyze the data myself. Over the course of this season I plan on looking at a few different issues: 1. Nationalistic bias 2. “Bloc” judging, judges being systematically biased in favor of skaters from other particular nationalities, typified, for instance, by judges from former USSR countries scoring Russian skaters more favorably. 3. Underscoring of top competitors by rival countries’ judges. For instance, do American judges underscore Yuzuru Hanyu? Japanese judges and Nathan Chen? Russian judges and Rika Kihira? Etc. My hope is that in addition to holding judges accountable, we may also be able to confirm or dispel some suspicions that fans have long held but don’t necessarily have evidence for. In the first post, we will look at nationalistic bias. We will discuss which judges show statistically significant evidence of nationalistic bias, how we’re able to determine this, as well as address some concerns and limitations. Part One: Nationalistic bias First, as some of you may already know, I have compiled a database of the judging records of every senior international level judge who has judged a competition from the beginning of the 2018-2019 season onwards. Yes, every competition and every judge (the only exception being invitationals like Shanghai Trophy or Japanese Open). It may be found here. Note that the file is too big to display on google docs, so you will have to download it. First, let’s take a look at the top line conclusions. The database currently contains 312 judges, 177 of which I examined for nationalistic bias (the other judges didn’t have an extensive enough judging record), 92 of which showed statistically significant evidence of nationalistic bias. Of those 74 showed strong evidence. On the whole, judges scored their home skaters quite differently than they scored other skaters. For instance, if we plot the average z-scores (this will be explained in detail later) of judges' scores for their home skaters versus other skaters, we get the following density plot: As you can see, this shows two very different patterns of judging! Who are these judges? Well, here they are, divided by federation. Names are included if they meet the standard benchmark for statistical significance (p<0.05), bolded if they meet a stricter benchmark (p<0.01), and bolded and italicized if they meet an even stricter benchmark (p<0.001). In parentheses next to the federation is the number of judges of that federation *tested* (not the total number of judges recorded, some of whom may not be tested due to insufficient data). Federations that do not have any judges that show statistically significant evidence of biased judging are not listed. Wall of Shame Austria (3) Adrienn Schadenbauer Canada (23) Andre-Marc Allain, Cynthia Benson, Leanne Caron, Reaghan Fawcett, Karen Howard, Leslie Keen, Patty Klein, Nicole Leblanc-Richard, Erica Topolski China (5) Dan Fang, Shi Wei, Fan Yang Czech Republic (7) Frantisek Baudys, Jana Baudysova Spain (2) David Munoz Finland (7) Merja Kosonen, Virpi Kunnas-Helminen, Leo Lenkola France (8) Ronald Beau, Jezabel Dabouis, Elisabeth Louesdon, Philippe Meriguet, David Molina, Florence Vuylsteker Great Britain (5) Christopher Buchanan, Stephen Fernandez, Sarah Hanrahan, Nicholas Russell Georgia (1) Salome Chigogidze Germany (13) Christian Baumann, Ulla Faig, Uta Limpert, Claudia Stahnke, Elke Treitz, Ekaterina Zabolotnaya Hungary (2) Attila Soos, Gyula Szombathelyi Israel (2) Anna Kantor, Albert Zaydman Italy (11) Matteo Bonfa, Rossella Ceccattini, Raffaella Locatelli, Isabella Micheli, Tiziana Miorini, Miriam Palange, Walter Toigo Japan (16) Miwako Ando, Tomie Fukudome, Ritsuko Horiuchi, Akiko Kobayashi, Takeo Kuno, Kaoru Takino, Sakae Yamamoto, Nobuhiko Yoshioka Kazakhstan (2) Yuriy Guskov, Nadezhda Paretskaia South Korea (2) Sung-hee Koh, Jung Sue Lee Latvia (1) Agita Abele Lithuania (1) Laimute Krauziene Mexico (1) Sasha Martinez Poland (3) Malgorzata Grajcar, Malgorzata Sobkow Russia (14) Maira Abasova, Julia Andreeva, Sviatoslav Babenko, Igor Dolgushin, Elena Fomina, Maria Gribonosova-Grebneva, Natalia Kitaeva, Olga Kozhemyakina, Lolita Labunskaiya, Igor Obraztsov, Tatiana Sharkina, Alla Shekovtsova Switzerland (3) Bettina Meier Sweden (4) Inger Andersson, Kristina Houwing Ukraine (2) Yury Balkov, Anastassiya Makarova USA (21) Samuel Auxier, Richard Dalley, Janis Engel, Kathleen Harmon, Taffy Holliday, Laurie Johnson, Hal Marron, Jennifer Mast, John Millier, Sharon Rogers, Kevin Rosenstein Let me note that there are many judges who are just on the right side of the borderline of being flagged, who I suspect will be flagged as more data comes in and evidence mounts. There are also many judges who will almost certainly be flagged as soon as they’ve judged enough to meet my minimum threshold for testing. On the other hand, note that due to the sheer number of judges tested, there may be a few judges (chiefly in the non-bolded, maybe one or two bolded, extremely unlikely for bold+italicized) who wind up on this list by pure luck (but the odds that any particular judge on this list is actually unbiased and just unlucky is pretty low). Generally speaking, the odds that I have missed a judge who deserves to be flagged are significantly higher than the odds that I have flagged someone who doesn’t deserve to be flagged. We can also compare federations. Here we will examine federations with at least ten judges, looking at the degree (the ZDifference) by which their judges have historically been biased. The following bar chart shows what percentage of each federation's judges have records that fall into 4 different categories of bias: no bias (anything equal to or lower than 0), low bias (0-0.5), medium bias (0.5-1) and high bias (greater than 1). Note that this includes judges who did not meet the minimum threshold of data quantity to be flagged on an individual basis. As you can see, some federations are all over the place like Canada, with some judges falling into each of the bias categories, whereas for others (chiefly Russia), every judge has a record that favors home country skaters. However, though Russia is the most consistently biased of the federations examined, other federations have a larger percentage of judges who have records displaying a high degree of bias. Of these larger federations, only Australia appears to have a good claim to be unbiased on any kind of systematic basis. This can be seen more clearly by examining the following box plot, which shows the distribution of ZDifferences of judges from each of these federations in a slightly different manner than the previous graph. (If you've never had to read a box plot before, the box is where the middle 50% of judges from each federation lie, and the line in the middle of the box is the median--or middle--judge from that federation. The lines extend out to the full range of each federation's judges, excluding the judges represented by the dots, who are outliers within their federation.) Of course, these numbers will change as the season progresses and more competitions are added to the data set. I will try to keep this post updated as time passes. (Everything is up to date as of 17 Oct 2019). In order to understand how I came to these determinations (and how to read the data), let’s take a look at methodology. As this discussion will involve a lot of statistical concepts that are probably unfamiliar to many people reading this, I’ll try to explain and offer links to statistical concepts that will be invoked here. Rest assured I did not make these metrics up--these are standard statistical tools used to analyze all kinds of data.

ugh...so much data entry that I'll have to do to incorporate this competition into my data set. but on the other hand, lots of data. hmmm I'm almost done writing up my judge bias review. The last thing I have to do is switch around judges' nationalities in order to show that bias isn't due to "cultural familiarity/preference" because judges from culturally similar countries don't overscore each others' skaters. Which, for the vast majority of judges is true. But then you see one or two judges (and a couple more than that from former USSR countries *cough* *cough*) who hit statistical significance and then some, and you begin to wonder about those people. We always suspected bloc judging from former USSR countries, and it looks like the data bear that out for at least some judges, but while most anglophone judges don't overscore other anglophone countries' skaters, we also have Richard Dalley, USA ice dance judge, sitting at p = 0.002 for how he scores Canadian IDers versus others. Suspicious.

Sasha's PCS being held down is fair, but Alina's scores are inflated as hell. Sasha should not have to land 4 quads to beat her, Alina is not that much better than her in everything else. (Note that in a full competition Alina would have an advantage from the short, so with only a 6 pt gap in the free it's not clear Sasha would even win.) Nor should Rika be unable to beat her even if she skates clean. Rika and Alina's PCS should be the same at best and Rika should win handily on TES.