Mock judging panel

[robin]

A thread to organize and discusss a mock panel to rescore past competitions. Everyone is welcome to join

So we already have the following volunteers

@Raomina: app

@Xen : TP and judge

@hoodie axel: judge

@yuzupon: spreadsheets

@yuzuangel: judge

@Fay: TP

@Neenah: side-judge

@Nuitsuki: side-judge

@Murieleirum: side-judge

me (with reservation): side-judge

 

TP results of first round, CoC 2017 ladies short program;

https://planethanyu.com/topic/625-mock-judging-panel/?do=findComment&comment=198833

 

[robin]

Quoting the discussion from the general skating chat here

Spoiler

14 hours ago, robin said:

 

 

I'm bringing this here. Would anyone be interested in mock judging?

 

3 hours ago, hoodie axel said:

BTW, how would the rescoring work? I have tried to score the SPs of a few favourite skaters in the way BoP scores programs and have felt satisfied with the results. I've also tried to score SPs in the way the judges do and have felt my IQ slowly dropping.

 

3 hours ago, robin said:

 

Well this is an open call so if you would want to be part of it we‘re at one 

 

 

Maybe we would have a TP for edge and under calls but we would probably just use the spin and StSq levels already set by the official TP (unless there is someone confident in doing that job) and have at least 6 (maybe even 4 is enough?) judges who would judge all the elements and PCS and enter them in a google spreadsheet? It should be possible to prepare one everyone could work on simultaneously that calculates the results. 

 

2 hours ago, hoodie axel said:

I think this is a good idea, but going back to Sochi (and again I don't know what you want to rescore here), I think TP for Spin and StSq levels is a good idea, because I think Yuna's StSq levels were called incorrectly. At least in the LP.

 

But what standard of judging? IN and PE are at least a little subjective. Going to Patrick Chan's SP in CoC 2016, would we have someone give him anything more than a 6 for IN and PE? OTOH, a skater having a meltdown doesn't necessarily have to be performing too badly, like Boyang in his SA 2016 SP, who would be getting a 7 for PE there from me. Would the judges necessarily evaluate everything properly without standards for PCS? For instance, I don't think Seimei's CO in 2018 was a 10 (close to 9), but is there a standard there?

 

2 hours ago, Xen said:

Well I can possibly join both, time permitting.

Why do I have an odd feeling I'll end up on TP for all the mock panels? o_O I can't do spin levels, but if I try hard, maaaaybe I can do stsq levels. Ah, what am I getting myself into?!

 

2 hours ago, robin said:

 

You‘re asking the really difficult questions here and there’s a reason why I am not volunteering myself as a judge 

Probably this should be discussed among the mock judges once we have set a panel? But we definitely don’t have to adapt common practices by judges, like scoring down PCS just because the program isn’t clean. 

 

 

There we go, we found our brave TP  

 

actually I‘d love to join, too, but i don’t feel comfortable judging anything but jumps GOE lol. this would end up being a super stressful experience for me so no thanks 

 

2 hours ago, hoodie axel said:

There shouldn't be a rule for scoring down, but I just want the objectivity to be there. I don't think Patrick Chan's strength was his IN in a few SPs, but I can't say that other judges wouldn't mark highly when judging a clean performance.

 

2 hours ago, robin said:

 

Since there isn’t a commission breathing into your necks in case you score wildly different from other judges, I don’t think you have to worry about how others score... if that’s what you mean? Just go by your own opinion. The discussion around why everyone scored the way they did would be more interesting if there’s differences, anyway 

 

2 hours ago, hoodie axel said:

Hahaha yeah, true, true. It will be very interesting to see how everyone scores. Maybe I'll involve myself, too, just to give unpopular opinions. The discussion would indeed be very nice to see.

 

2 hours ago, Xen said:

I'm a nice TP, definitely not The Shin Amano. =) But I will demand lots of ice cream, salsa and chips after each judging rounds. And maybe a plush toy (a pooh!) to squeeze while I dish out the ruling. =P

Actually compared to you guys, I'd be the most generous judge EVAR on PCS. "You have an 8, and you here have a 9, and oh, 8.25's for everyone else!" Seriously, I feel a bit bad if I go under 7.5 for most senior men's or ladies, since that's borderline not passing your final exams. 7.75 is sort of my baseline,  under that is when existential crisis start.

 

For judging- I'd consider skipping Sochi all together. Men's might be the only discipline, with pairs perhaps, that we have some degree of agreement on. Ladies is also a hotbed of drama and controversy scoring.

 

2 hours ago, hoodie axel said:

You'd give Nathan's Olympic outings anything more than a 5 for PE? You're indeed generous.

 

2 hours ago, Xen said:

7.5....time to cue artistic existential crisis since it went under my bar of 7.75. I don't think he's that bad yet, but needs re-evaluations. Hey, I'm just a judge trying to keep my job, okay? And far far away from investigation. >_>

 

2 hours ago, robin said:

 

LOL we‘ll make sure to send you a care package 

 

What‘s most important is being consistent so as long as you score everyone high that should be fine hahah

but make sure to leave enough room upwards so that he difference between the really good ones (not naming anyone) and some others can be reflected 

 

I was actually thinking of not going into the previous quad at all because the rules were so different 

 

2 hours ago, yuzuangel said:

This sounds like fun and I'd love to participate, but I'm not sure I can guarantee to not be extremely biased...

 

2 hours ago, hoodie axel said:

This would be great. I might do exactly one panel. I'd love to really score Medvedva and Zagitova's Olympic outings, if you know what I mean. 

 

2 hours ago, Xen said:

I think the major changes were step sequence levels, in which case it shouldn't be too hard for me. If slow-mo is allowed, I can probably ID steps (will need help with Choctaws though ugh). What would amuse me is if results were published-you need some thick skin for some of the onslaught that may come from some corners of the FS community. It would also be amusing though, if the results got thumbed up by say, Max and Angelo.

 

1 hour ago, yuzupon said:

I can maybe contribute to figure out the spreadsheet formula if no one else is doing it.

I can't judge, cz I'd be scoring everyone below 7 in PCS, and GOEs from me would range from -3 to 1. I tend to underscore everything and everyone. Yes, even Zu, despite being outraged when he is being underscored by judges  (Cz I always feel like his score should probably not go that high by itself. But relatively to how the others are scored, goddammit! Zu is underscored).

I can't do TP either cz while jumps, and maaaaaybe spins are somewhat recognisable for me, this is how I differentiate Stsq and Chsq of skaters that still some semblance of steps in their Chsq: "Is there Loop? If yes: Stsq".

 

1 hour ago, robin said:

 

We‘re not an official panel and should use all the technology we can use (shade intended) so slowmos should be fine  

 

It should also be fine to replay elements for GOE scoring?

 

1 hour ago, hoodie axel said:

Pretty much.

 

For reference, I scored H&L 2017W at 214. Shoma got 190 for Loco and Boyang got 193 for La Strada. Didn't mark SPs, but I think Shoma barely edges out for Silver. Maybe.

 

If we're doing slow-mo... Pre-rotation checks, please!

 

1 hour ago, Xen said:

 

Depends...I actually don't mind fancams to a degree, since it usually lets me see the actual edge depth a bit better. Necessary in case someone's counter goes off a flat edge. Well if you guys want to be really mean, I could take a stab at TP calls with just fancams.

Maybe we should take all this talk to that thread for the actual mock panels? The issue with PR is that judges and TPs usually don't call Pre-rotations. And no slow-mo is permitted. So for Pre-rotations, there are the ones where the skater definitely pre-rotated over 180 degrees. But what about PR between say 120-180 degrees? Would we call that a weak take off/jump entry? And if so, do you actually want a TP to mark those differences out? Or leave the ones under 180 for judges?

 

Lastly, a part of me is a bit nervous about a mock panel coming from a hanyu fan forum. Yes, a lot of us can be very neutral when we have to score, but I can imagine the drama that it can cause from some other fan groups. It's not like Max and Angelo only got praise and love for their commentary and analysis-even they have been called biased. o_O

 

38 minutes ago, hoodie axel said:

Measure pre-rotation so we can see if the jump as at least number of turns in the name - 0.75 turns. So check if a triple has 2.25 rotations. Between 2-2.25 UR. Below 2 DG. Check for full blade takeoffs, make a notation, and tell them what it is. They're actually easy to spot, though, if the angle's alright, so it should strictly be a GOE deduction, no need to invent further rules.

 

A jump with 180 degree PR will be called rotated if it lands a quarter short (we don't need a concept of doubt), but will not get high GOE because it's only minimally rotated.

 

The "judges" will be watching anyway, so they'll see what to do.

 

 

37 minutes ago, yuzupon said:

Oh, wait. Are we publishing the mock panel outside of the forum? I thought it's just for internal fun?

 

About Ambessi and Dolfini. I give up on the people who call these two biased based on that analysis. I am not sure how you can look at that analysis with clear slow-mo examples to back what they were talking about, and then your only response is to cry "Biased!!!". It comes from people who clearly cannot look at the argument at face value and instead attack based on reputation/past argument/personal preference. But you're right. Based on how these two got treated, any mock panel coming form Yuzuru Hanyu fan forum would be met with fire and fury. I would advise against publishing it outside of the forum.

 

27 minutes ago, Xen said:

I think there are people who aren't Hanyu fans who are aware of this place. =(

There are plenty of Hanyu fans who have antis stalk their twitter too. It's pretty sad that there are no two ways about it-only that hopefully people who find this useful, outnumber the antis.

Anyways, I'll be okay with being TP if there's a need.

 

26 minutes ago, robin said:

I will make a thread in a minute! A new one since this is a new start

 

Also, I think what‘s important is that we don’t take each other and ourselves too seriously. We could do something like, publish as google sheets but put a password on it that can be shared via PMs? To avoid any drama and also to make it an overall less stressful experience for everyone involved 

 

Actually, if we make it a lighthearted thing I would join, too (but take my scores out of the overall scores, maybe. I‘d score on a hunch and wouldn’t want to be made responsible for the scores I give out lmao. Maybe that way there would be more members who‘d want to join for the learning experience)

 

eta: there‘s also a mechanism where links won’t be displayed to non-members on the whole forum. You have to be logged in to view links so that’s one filter already 

 

 

[robin]

1 hour ago, Xen said:

 

Depends...I actually don't mind fancams to a degree, since it usually lets me see the actual edge depth a bit better. Necessary in case someone's counter goes off a flat edge. Well if you guys want to be really mean, I could take a stab at TP calls with just fancams.

Maybe we should take all this talk to that thread for the actual mock panels? The issue with PR is that judges and TPs usually don't call Pre-rotations. And no slow-mo is permitted. So for Pre-rotations, there are the ones where the skater definitely pre-rotated over 180 degrees. But what about PR between say 120-180 degrees? Would we call that a weak take off/jump entry? And if so, do you actually want a TP to mark those differences out? Or leave the ones under 180 for judges?

 

I think we can tweak how we judge a bit to what's more ideal. How about adding pre- and underrotation and judging as you would judge underrotations? Maybe add anything that goes over the acceptable PR of a jump to the UR and lower the BV if necessary. I think pre-rotations can go both into the GOE as poor take-off and into lowering the BV

 

Logistically, maybe the TP does their work first, uses fancams if they want to, and the rest can be judged with the results of the TP already present

[Xen]

6 minutes ago, robin said:

 

I think we can tweak how we judge a bit to what's more ideal. How about adding pre- and underrotation and judging as you would judge underrotations? Maybe add anything that goes over the acceptable PR of a jump to the UR and lower the BV if necessary. I think pre-rotations can go both into the GOE as poor take-off and into lowering the BV

 

Logistically, maybe the TP does their work first, uses fancams if they want to, and the rest can be judged with the results of the TP already present

Hmm. Okay, that I can work with. I think marking out the ones over 180 degrees PR is fine. As for anything less than 180, it depends on the jump-most men's tend to only go 90-120 PR, while plenty of female skaters tend to hit about 120+ degrees PR.

 

I would also like to advise that we add at least another TP. Is there anyone we could add? Say, Fay, Yata, Meoima, Lady Lou? I probably only have a few hours during the weekend to look at stuff, so it's probably better that you guys give me ahead of time a list of skaters and competitions you want first. Probably initially just zoom on the jumps UR/edge/PR issues since those are the bigger score-differentiators.

 

This reminds me of that Planet Hanyu commentating panel idea. XD

 

[robin]

6 minutes ago, Xen said:

Hmm. Okay, that I can work with. I think marking out the ones over 180 degrees PR is fine. As for anything less than 180, it depends on the jump-most men's tend to only go 90-120 PR, while plenty of female skaters tend to hit about 120+ degrees PR.

 

I would also like to advise that we add at least another TP. Is there anyone we could add? Say, Fay, Yata, Meoima, Lady Lou? I probably only have a few hours during the weekend to look at stuff, so it's probably better that you guys give me ahead of time a list of skaters and competitions you want first. Probably initially just zoom on the jumps UR/edge/PR issues since those are the bigger score-differentiators.

 

 

Sure, we can see if there’s someone else ready to join you in the TP. If it’s too much work you really don’t have to overlook levels, too. 

 

eta: @Fay agreed to help with the TP  

 

[Xen]

10 minutes ago, robin said:

 

Sure, we can see if there’s someone else ready to join you in the TP. If it’s too much work you really don’t have to overlook levels, too. 

 

eta: @Fay agreed to help with the TP  

 

Oh @Fay!

[Fay]

Yeah, I want you to have someone to scold when you grumble about technical things being ignored! 😃

[hoodie axel]

30 minutes ago, Xen said:

Hmm. Okay, that I can work with. I think marking out the ones over 180 degrees PR is fine. As for anything less than 180, it depends on the jump-most men's tend to only go 90-120 PR, while plenty of female skaters tend to hit about 120+ degrees PR.

 

I would also like to advise that we add at least another TP. Is there anyone we could add? Say, Fay, Yata, Meoima, Lady Lou? I probably only have a few hours during the weekend to look at stuff, so it's probably better that you guys give me ahead of time a list of skaters and competitions you want first. Probably initially just zoom on the jumps UR/edge/PR issues since those are the bigger score-differentiators.

 

This reminds me of that Planet Hanyu commentating panel idea. XD

 

Could you elaborate? You're going to mark PR differently depending on whether it's a man or a woman?

 

Also, if it's a competition from this quad, I can really only think of one event I liked as a whole -- WC men's 2017. But I've already scored the FS and that won't be fun.

 

I guess Oly Ladies' FS would be a nice train-wreck, just to see how many of Satoko's jumps were overlooked, or just how low I can go with the top 2's PCS.

[Xen]

Just now, hoodie axel said:

Could you elaborate? You're going to mark PR differently depending on whether it's a man or a woman?

Okay, my issue is that what is considered "weak" take off-would there be a difference between male skaters and female skaters. And what we think as a baseline PR for "strong take offs." What standards do we keep?

 

For many male skaters, I would personally say anything above 90 degrees PR on a lutz is weak (in fact, Yuzu, Nate, Boyang and Mikhail barely PR, at least it is within 45 degrees). While on the ladies, many ladies do PR significantly (even Yuna has slight PR, around 90, which is very good for a female skater already). And for many edge jumps, I can barely stand anything above 120 degrees-male or female.

 

Lastly, what about corridor judging? Do we follow that or allow points to really differntiate between say TR and SS?

[Fay]

7 minutes ago, Xen said:

Okay, my issue is that what is considered "weak" take off-would there be a difference between male skaters and female skaters. And what we think as a baseline PR for "strong take offs." What standards do we keep?

 

For many male skaters, I would personally say anything above 90 degrees PR on a lutz is weak (in fact, Yuzu, Nate, Boyang and Mikhail barely PR, at least it is within 45 degrees). While on the ladies, many ladies do PR significantly (even Yuna has slight PR, around 90, which is very good for a female skater already). And for many edge jumps, I can barely stand anything above 120 degrees-male or female.

 

Lastly, what about corridor judging? Do we follow that or allow points to really differntiate between say TR and SS?

Nathan prerotates close to 90 degrees, Misha and Yuzu prerotate least, I’d say. 

[hoodie axel]

Measure pre-rotation so we can see if the jump as at least number of turns in the name - 0.75 turns. So check if a triple has 2.25 rotations. Between 2-2.25 UR. Below 2 DG. Check for full blade takeoffs, make a notation, and tell them what it is. They're actually easy to spot, though, if the angle's alright, so it should strictly be a GOE deduction, no need to invent further rules. Uniform for all skaters.

 

If you like, just tell us the degree of PR, and where they landed, and we'll make the judgement call.

 

A jump with 180 degree PR will be called rotated if it lands a quarter short (we don't need a concept of doubt), but will not get high GOE because it's only minimally rotated.

 

The "judges" will be watching anyway, so they'll see what to do.

 

No corridor judging.

[robin]

11 minutes ago, Xen said:

Okay, my issue is that what is considered "weak" take off-would there be a difference between male skaters and female skaters. And what we think as a baseline PR for "strong take offs." What standards do we keep?

 

For many male skaters, I would personally say anything above 90 degrees PR on a lutz is weak (in fact, Yuzu, Nate, Boyang and Mikhail barely PR, at least it is within 45 degrees). While on the ladies, many ladies do PR significantly (even Yuna has slight PR, around 90, which is very good for a female skater already). And for many edge jumps, I can barely stand anything above 120 degrees-male or female.

 

One reason why I would lean towards being a liiiittle lenient with PRs is that since TPs haven’t actually been calling them skaters haven‘t had motivation to work as much on them as they work on URs. We can put the threshold for ladies a little higher but then again men and women are also judges the same in URs so I am not sure...

 

and yes, pls no corridor judging

[robin]

5 minutes ago, hoodie axel said:

Measure pre-rotation so we can see if the jump as at least number of turns in the name - 0.75 turns. So check if a triple has 2.25 rotations. Between 2-2.25 UR. Below 2 DG. Check for full blade takeoffs, make a notation, and tell them what it is. They're actually easy to spot, though, if the angle's alright, so it should strictly be a GOE deduction, no need to invent further rules. Uniform for all skaters.

 

Hm the thing is there is a slight degree of PR that’s acceptable. Should we not take that into account at all?

 

But ofc just strictly counting the rotations is the most straightforward and would also make things easier and simpler for the TP 

 

I think it’s fine either way as long as we‘re being consistent. 

[Neenah]

I want to try this but as a side judge since I don't have much experience. I find the idea interesting and I always liked the scoring system so I want to have fun with it. Maybe I will  join the panel for later competitions if we continue and I gain more confidence   

[hoodie axel]

5 minutes ago, robin said:

 

Hm the thing is there is a slight degree of PR that’s acceptable. Should we not take that into account at all?

 

But ofc just strictly counting the rotations is the most straightforward and would also make things easier and simpler for the TP 

 

I think it’s fine either way as long as we‘re being consistent. 

This takes it into account!

 

PR=180 degree. UR=90 degree. So 1/2 turn+1/4 turn = 3/4 turn allowed. So total number of turns-allowed turn = (number of turns-1) + 1/4 turn minimum.

 

I think counting rotations makes more sense. Suppose a skater does a 3Lutz with 0 PR. If he lands say half a turn short, he will have done 2.5 turns in the air. This is more rotated than someone who PRs 180 degree and lands 90 UR, and shouldn't deserve a UR call, and in fact deserved high GOE for good technique. +3 if all three turns are done in the air!