Mock judging panel

[Xen]

Um, I think I'll stay firmly as TP. =) My over generous scores as a judge would net a lot of tomatos to the face.

Thinking back about how I tracked URs for ladies...I might not be that far from Shin Amano on URs now.

[Raomina]

Just dropping in to say that I've written a web app specifically for 'live' judging past competitions like these, so you guys could try it if you want? 

https://flickering-inferno-3394.firebaseapp.com

 

It hasn't really been beta tested by anyone aside from me since it's just a personal project so there would most likely be bugs that I'm not aware of but I can try my best to fix them if I can capable of doing it. 

 

The set up of the app is basically such that it automatically loads the skaters in start order in the competition and someone (like a TP caller) has to add the elements in in the 'Edit' mode. But once it is done, that person can export the event (in Options at the top on the individual event page or the all events page) and share the file with  other users who can then import them (in Options on the all events page) and use it for judging. 

 

For test purposes, I have added the elements for all the Men's SP for all the GPF according to the actual protocol so if you load the GPF event for any season, you should see the elements listed for each Men's SP ready for judging. 

 

The scoring is done by checking bullet points and the scores will be calculated automatically and displayed in two formats at the end. It should be self explanatory but I can answer any questions if needed. 

 

Hopefully this can be useful and easy to use and I'm open to any feedback such as interface as well! E.g 

I've been debating about how to score the PCS as sliders might not be the easiest.

 

 

Edit: oh, and just to add, it's made to be a mobile app so that you can judge on the phone/tablet as you watch on the screen. 

[Xen]

4 minutes ago, Raomina said:

Just dropping in to say that I've written a web app specifically for 'live' judging past competitions like these, so you guys could try it if you want? 

https://flickering-inferno-3394.firebaseapp.com

 

It hasn't really been beta tested by anyone aside from me since it's just a personal project so there would most likely be bugs that I'm not aware of but I can try my best to fix them if I can capable of doing it. 

 

The set up of the app is basically such that it automatically loads the skaters in start order in the competition and someone (like a TP caller) has to add the elements in in the 'Edit' mode. But once it is done, that person can export the event (in Options at the top on the individual event page or the all events page) and share the file with  other users who can then import them (in Options on the all events page) and use it for judging. 

 

For test purposes, I have added the elements for all the Men's SP for all the GPF according to the actual protocol so if you load the GPF event for any season, you should see the elements listed for each Men's SP ready for judging. 

 

The scoring is done by checking bullet points and the scores will be calculated automatically and displayed in two formats at the end. It should be self explanatory but I can answer any questions if needed. 

 

Hopefully this can be useful and easy to use and I'm open to any feedback such as interface as well! E.g 

I've been debating about how to score the PCS as sliders might not be the easiest.

Oh nice! I'll go poke around with it!

 

So regarding the events- which ones do we judge? I think we have agreement on COC ladies top 6, Sp and LP correct? I'm thinking also Europeans Ladies, and possibly SC 2017 (think there were complaints that Marin Honda got robbed?). Any other controversial ones I'm missing?

[robin]

@Raomina That's amazing! Especially being able to check bullets instead of calculating the GOE ourselves

 

@yuzupon I think the spreadsheet can wait for now? And you could focus on your exam without the temptation of sketching the sheet

 

5 minutes ago, Xen said:

So regarding the events- which ones do we judge? I think we have agreement on COC ladies top 6, Sp and LP correct? I'm thinking also Europeans Ladies, and possibly SC 2017 (think there were complaints that Marin Honda got robbed?). Any other controversial ones I'm missing?

 

I think we can start with the COC ladies SP (we might as well do all of them imo) just as a test and see from there?

[yuzupon]

@Raomina life saver!!!!!!!!!!!

Phew!

 

[hoodie axel]

My GOE assignment is a little different from most.

I see height, coverage, flow and match to the musical structure, see if all the rotations were completed in the air, and then see if someone had a transition into it, or delayed the jump. I couldn't care less about Tanos or Rippons unless there's literally nothing else to see.

 

So for instance, Boyang's 4Lz in the WC17 FS and 4CC18 FS would easily be a +2.5 to +3. Height, coverage, flow, matched to the musical structure, all the rotations completed in the air. I believe he's slightly getting better at delaying the rotation as he gets better SS, and therefore can go into it with more speed, and thus transfer more power into the take-off.

[Xen]

Yes, but strictly speaking we should be following bullets as they are. Otherwise, it might as well be "let's judge by our standards that we created based on our bias" rather than "let's judge how it should have turned out by ISU existing rules." Hence why I mentioned being hesitant to call PRs unless they are extremely egregious-and even then, if we follow how TPs should usually go, the benefit is to the skater, not to my eyes or biases (honestly that's all that's keeping me from giving Shin Amano a run for the money).

 

IMO, Boyang's jumps are usually a nice +2 if he doesn't have a mess up on the landing. Yuzu is usually a +2, +3. Shoma I can see some getting +1, as he does have good flow out, even if I don't like the knee torque. Most US men though, I think Jason Brown probably has the best looking jumps now (otherwise I like Andrew Torgashev in the Juniors ranks). Overall, the east asian skaters have pretty nice jumps, and Aliev has nice looking ones (minus that PR on the lutz), and Mikhail as gorgeous jumps.

[Neenah]

Checking points is not the correct way to use GOE   I know many people think that it is better to count points and give GOE accordingly but these are guidelines for a reason. The judges should have the freedom to round up or down if the element checks an odd number of bullets depending on quality. They should also be able to reward innovation and spectacular elements that would not be rewarded with just checking points. FS elements have so many small details going into their execution and skaters are always trying to distinguish themselves so it is very important that the judges can and do give the appropriate reward even if it is not stipulated in the rules. The fact that the judges sometimes abuse the system does not mean we destroy a good part of it that should (theoretically) be to the skaters and the sport's benefit.

 

This is a typical thing in any scoring rubric that aims to evaluate something subjective (we use it with student generated content such as essays and videos all the time)

[yuzupon]

27 minutes ago, Neenah said:

Checking points is not the correct way to use GOE   I know many people think that it is better to count points and give GOE accordingly but these are guidelines for a reason. The judges should have the freedom to round up or down if the element checks an odd number of bullets depending on quality. They should also be able to reward innovation and spectacular elements that would not be rewarded with just checking points. FS elements have so many small details going into their execution and skaters are always trying to distinguish themselves so it is very important that the judges can and do give the appropriate reward even if it is not stipulated in the rules. The fact that the judges sometimes abuse the system does not mean we destroy a good part of it that should (theoretically) be to the skaters and the sport's benefit.

 

This is a typical thing in any scoring rubric that aims to evaluate something subjective (we use it with student generated content such as essays and videos all the time)

 

I understand that. But checking the bullet points should help, no? We round up or down, but with a clear indication of just how many bullet points were checked. Instead of having only 1-2 positive bullet points checked, but the jump "feels" like a +2GOEs (especially when "compared to other skaters' jumps" come into play), hence +2 it is. My understanding is that the bullet point checking is there to try to minimise the subjectivity as much as possible. And, that there is a universal standard in scoring, instead of pitching skaters against skaters, which makes scoring fluctuate insanely from one competition to the other. 

Rubrics for evaluating students are also supposed to work that way, yes?

 

For the purpose of this Mock Judging, we're trying to do that by following the guidelines as much as possible, aren't we? After all, none of us here would (should?) declare we have enough expertise to know when and by how much we reward what details which skaters did.

 

What comes out as the result later, could then perhaps be a start of argumentations on whether trying to stick to the guidelines as close as possible could (would) potentially further or hinder the sport.

 

*On @Raomina's app, the GOEs deductions are done flat, with explanations. It's quite an extensive app. I am still not sure how scores from multiple people can be gathered, however. I'll explore more once my exam is done.

[hoodie axel]

TBH I'm just excited to do this

[hoodie axel]

45 minutes ago, Xen said:

Most US men though, I think Jason Brown probably has the best looking jumps now (otherwise I like Andrew Torgashev in the Juniors ranks).

What do you think about Nathan?

[Neenah]

8 minutes ago, yuzupon said:

 

I understand that. But checking the bullet points should help, no? We round up or down, but with a clear indication of just how many bullet points were checked. Instead of having only 1-2 positive bullet points checked, but the jump "feels" like a +2GOEs (especially when "compared to other skaters' jumps" come into play), hence +2 it is. My understanding is that the bullet point checking is there to try to minimise the subjectivity as much as possible. And, that there is a universal standard in scoring, instead of pitching skaters against skaters, which makes scoring fluctuate insanely from one competition to the other. 

Rubrics for evaluating students are also supposed to work that way, yes?

 

For the purpose of this Mock Judging, we're trying to do that by following the guidelines as much as possible, aren't we? After all, none of us here would (should?) declare we have enough expertise to know when and by how much we reward what details which skaters did.

 

What comes out as the result later, could then perhaps be a start of argumentations on whether trying to stick to the guidelines as close as possible could (would) potentially further or hinder the sport.

 

*On @Raomina's app, the GOEs deductions are done flat, with explanations. It's quite an extensive app. I am still not sure how scores from multiple people can be gathered, however. I'll explore more once my exam is done.

Checking points is useful for us here because we want this to be a discussion and a learning experience but for the judges doing it on the spot it really is not possible or realistic. The GOE are subjective by nature and that is not a bad thing per say but it can be misused easily. The only way to solve this problem is to give the judges time to rewatch everything they want and to have a second judge review the scores (which is what we do with students marks). The ISU is trying to that by having 9 judges who are scoring the same thing and taking the mean but the fact remains that the judges have to make quick decisions without consulting the rules every time, so they rely on their own interpretation of the bullet points. 

 

I will check the app when I have time and I think it would also be useful to us to discuss how we interpret the bullet points and weigh them against each other some time in the future

 

 

[Xen]

8 minutes ago, hoodie axel said:

What do you think about Nathan?

He has nice jump entry, and in terms of entry goes, his jump technique is fine, nothing that makes me want to grit teeth on. Height is okay, distance covered not that great-more height than distance, depends more on rotation speed and on axels at some point he made me wonder if he was spinning almost in the same position. Landings make my ankles hurt, flow on exit is non-existant. Musicality is a bit, meh (better in SP, problem in the long). He would probably usually net a 0 from me. No clear mess-ups, but nothing extraordinary to add GOEs to.

 

(His OG PCS scores shocked me, I initially thought the PCS for the long was 82, which might have made more sense-but then someone woke me up to the fact it was 87)

[Nuitsuki]

Wow @Raomina you are such a genius!!! 

The app is simply fantastic! It has notations everywhere, users don’t need to remember the whole rulebook to use it! 

 

I’ll learn from you guys for the 1st round, but i’d dearly love to sign up for judging in next round!

[robin]

32 minutes ago, Nuitsuki said:

Wow @Raomina you are such a genius!!! 

The app is simply fantastic! It has notations everywhere, users don’t need to remember the whole rulebook to use it! 

 

I’ll learn from you guys for the 1st round, but i’d dearly love to sign up for judging in next round!

 

Do you want to be a side-judge as well?