Project: Tell the ISU to rewrite its rules with data science

[wingman]

Join us for the best data project ever!

 

Dear the computer scientists, the data scientists and the FS lovers amongst us,

 

Since we've been discussing about new scoring system and its effects... I'm totally won over (and shocked to the core) by Ambesi's argument here

https://the-a-factor-fs.tumblr.com/post/165975610600/kiss-cry-episode-1#Part_4

The ISU needs to base their decisions on facts and numbers. Ambesi has been calling for a data revolution, but seems like the ISU hasn't taken notes.

According to him, the new system won't change anything, and he presented some very good points.

1- Judges correlate TES with PCS: high TES automatically equals high PCS. So changing TES will not change the problem

2- Why not change base multiplier for PCS

3- Personal inflation for some skaters; Biases from federation

The last point I find most ridiculous. It is ridiculous how experts like him (and many experts amongst us) frequently make comments like "How many US judges are there on the panel? Then we would know if Hanyu get 10 in PCS"; or "Those American girls get high scores because of their nationalities"; or "some skaters automatically get 9s in PCS and that just doesn't make sense'.

We, lovers of this sport, have put up with such blantant score manipulations for too long! We should give constructive feedback.

 

The good news is, all the scoring data, the videos, the rule book etc are available publically. So I call for a data analysis project to present the ISU with numbers, facts. Some of us have already done interesting anlaysis (eg: @yuzuangel ). Let's pool resources to do the most ambitious and timely project ever: recommend the best scoring system for the ISU.

 

Specifically, we want to use data to answer the following questions

Q1: is TES a clear predictor for PCS?

This is dead easy to do.

Q2: what predicts an individual skater's score? 

factors: competition, judge's federations, etc etc (need some discussions, since we need to take out the "true" score obtained by mechnically applying the rules from the rule book.) One way to get this gold standard would be to average many independent experts' scores (to be discussed further on how to gather such data).

One can also do something similar to @yuzuangel's analysis on her blog: eg: compare skater's score x judge's federation

Q3: more simulations like the one Ambesi mentioned in the article

(more discussion later).

 

A report that answers Q1 and Q2 is already huge. And it will make a huge impact, since it will strike a chord with many observers, reporters, fans and skaters.

 

I do know that this is an ambitious projects - but there are many of us! Plus, as far as I can tell, NO ONE is doing analiysis data analysis whatsoever. So anything that we present, as long as it is objective and a good analysis, is already interesting.

 

Specifics on this project: we will need people to do the following:

Phase 1:

* data collector / data manager: ideally in this mix we have some fan with FS knowledge, some with CS skills(to scrap data from ISU and related competitions, for example). Detailed technical data, skater data, judges etc would be best

* statisticians: I'm one, more needed

Phase 2:

* report writer: format, edits etc

* PR team: disperse results to influential reports, broadcasters and major sites

 

Right now we need people in phase 1. If you're in, please reply below (say briefly what skills you have and what role you want). Then we can organize a private board on this matter.

I hope this project will take off!

 

[yuzuangel]

This is really interesting! Would definitely be interested in drawing up a game plan/doc and working on this!

 

I have a feeling that the ISU won't listen though, but it could be powerful enough..btw Buzzfeed has this ~*groundbreaking*~ (not really) article about figure skating biases and it didn't really pick up the way they probably wanted: https://www.buzzfeed.com/johntemplon/the-edge?utm_term=.kuaZ9Rglw#.xvKJbENRW

 

But if we do end up getting good data and results, and we tweet it at people like Max, maybe it'll help with traction

[Hydroblade]

This, as a study, sounds very interesting, though i agree with Yuzuangel, i'm not sure ISU would listen.

I have loads of free time now, i'm not a computer scientist but i'm a computer engineer focused on software engineering(doing backend rn)  so if i can help with anything please let me know i like this kind of projects.

[souyouki]

I can help with the CS number crunching part... maybe some kind of a UI for visualization purposes.  I'm sort of new on FS knowledge though...

 

From what disciplines will we need to collect data? Just men's or all of them?

And from what years, and what competition types (nationals, GP, senior B)?

[Xen]

Okay, I'm not sure if we could do anything to influence ISU, and the issue is not necessarily ISU, but the federations themselves. 

I'm not really that knowledgeable about pairs or dance, but maybe can help out with singles knowledge. 

Major deadline in April, but maybe can help out occasionally and after April. 

I do think the completion of the project will probably come in too late for the GOE changes though. But maybe we can halt the BV changes, or at least balance it out. 

[Hydroblade]

12 minutes ago, Xen said:

Okay, I'm not sure if we could do anything to influence ISU, and the issue is not necessarily ISU, but the federations themselves. 

I'm not really that knowledgeable about pairs or dance, but maybe can help out with singles knowledge. 

Major deadline in April, but maybe can help out occasionally and after April. 

I do think the completion of the project will probably come in too late for the GOE changes though. But maybe we can halt the BV changes, or at least balance it out. 

maybe if fans of other skaters joined us we would have more credibility but yeah i don't think WE can do something about it

Still, i'd like to see the actual numbers so i am interested in this

[Neenah]

Sakatingscores.com already have a good (and expanding) database and they are doing some interesting statistics as well. I remember when it first started the owner (programmer) posted on GS and they said they had a code to read the pdf protocols. Maybe you can try contacting them,.The site is already getting attention from the skating community.

 

I am computer scientists but my work is more in web development and user research so I am not sure how much help I could be.

[guitarist]

Ooooohhh, this is an awesome idea. I don't expect ISU to listen, but I'd love to participate. I'm a software engineer. Mostly I do full-stack, but I also have some experience with AI and ML using Python. That might be useful. I'm not too knowledgeable on stats and data science, but I can implement stuff if given requirements. I can also help with data parsing and basic database stuff. Should be more fun than work, at any rate!

[hoodie axel]

Good luck

[wingman]

Hi guys,

Oh it's great - I'm glad that there are so many people interested. 

@halley and I have started collecting some data and doing some analysis. Halley has been really amazing. We have 

* a git folder to share code, data and analysis outputs

* a private trello board to list tasks

 

If you guys want to be involved, send me a message!

 

We will put some stuff up on a private wordpress soon. The idea is to get feedback from people with FS insights to refine our stats models (eg: should look at the relationship of variable X with Y, or should account for this and that, or get data from so and so). 

 

Once we got to the wordpress stage, we will send you guys link! 

The reason we want to keep it private and take place outside of the Planet is 

a/ to avoid trolls, and 

b/ not be a project affiliated with a particular fan group, otherwise people will automatically assume that we are biased.

 

(Sorry, life got hijacked by work and what not. I'll try to get the wordpress thing up and running this weekend). 

 

 

[wingman]

Dear everyone,

Our beta wordpress is up! If you guys are interested in contributing, please PM me or @halley and we will send you passwords to view the site.

Many thanks!