Congratulations to Maura Fitzgerald for taking first place in our in-class Kaggle competition! First a couple comments, and then the final results are below.
Were these Kaggle-competitive scores?
The top scores were in the ballpark of the winning scores in the external version of this competition. The students in the class were given slightly different data sets, so I asked Will Cukierski, from Kaggle and one of the top scorers in the external competition, whether Maura’s score would be competitive if she had participated in the external competition. My reasons for asking are I want “proof” that we have awesome data scientists in this class. Will said,
The essays in the official set included two more tricky sets (which were graded 1-40 and not just 1-5ish), and also had a complete holdout group (so you have a train set, a leaderboard set, and a final test set and you never get to see ANY of your score on the final essays that matter).
I would say Maura would easily have been in the top 20, and likely in the top 10. Also, in the official competition there is more competition, which dangles a carrot in front of your nose and lets you know sooner where the asymptote is going to be. I have no hesitations in calling her efforts competitive!
Will also explained that “roughly, 0.8 is strong. 0.75 is worthy of a decent grade. 0.7 is probably a minimum effort line. It’s much harder to go from 0,75 to 0.76 than it is to go from 0.70 to 0.71.”
Of course that means that Chris Mulligan’s efforts were competitive and possibly others in the top few.
Ranking Data Scientists Using Predictive Modeling Competitions
Given that Kaggle ranking is being considered as one of the signals to look for in a data scientist (someone who can do well in predictive modeling competitions), I wanted this to be something the students experienced. Now some of the students have demonstrated competitive scores in a predictive modeling competition. And more importantly, all students took a stab at it- some of whom had never even written code or didn’t know what machine learning was before the course began.
However we’ve discussed multiple times in class that the skills assessed in predictive modeling competitions are by no means the complete set of skills a data scientist needs. The brief version of this being the data set has been created for you (no data wrangling, data munging, data processing, data cleaning, data collecting, no map-reducing!), no question asking or setting up the problem, no data visualization — so the point being in these competitions, a lot of the work of a real data scientist has already been done for you. All that matters is a “black box” algorithm that predicts- you don’t even have to actually understand it or the meaning of what you’re doing to implement it, and you can still do well.
Of course there are people like Will, who competed and did well, and who I know is an extremely thoughtful data scientist, so I’m not saying these competitions don’t potentially have some strong signal in them.
Competitions give incentive to students to write code, build models, do feature extraction and selection, and experiment with different algorithms. I’m proud of the students for competing successfully, and I found this to be a very effective pedagogical tool.
Women in Kaggle competitions
The top list of Kaggle competitors includes no women. Reasons for this could be (1) No women are competing because there are no women “good enough” to compete or (2) No women are competing because there are women who are “good enough to compete” but they’re opting out or (3) Women are competing but not doing well.
We had men and women in the class (more men than women), and it was a requirement that they all participate (obviously). The winner was a woman and the top 10 included other women. Now I never doubted that women were capable of doing well in predictive modeling competitions! And I don’t expect it to be 50% women in external competitions, but rather proportional to the number of women with these skill sets. And even though women are often the minority in computer science, math and engineering courses, there are enough women out there with these skills that they could be appearing high in Kaggle rankings, if they competed.
If predictive modeling competitions are being held up as a “gold standard” and women aren’t competing, then this is a problem. It’s problematic if a system used to detect “great data scientists” is a system that women opt out of. I don’t think it’s sufficient to say “women can compete if they want to” because there is some systemic problem preventing them (us) from doing so.
We can still use competitions to identify people good at predictive modeling, but this is not sufficient.
Results
And Chris Mulligan’s visualization:
There’s been a bit of discussion (and code sharing) in the Kaggle forums since the close.
https://inclass.kaggle.com/c/columbia-university-introduction-to-data-science-fall-2012/forums/t/3287/congratulations-maura
[…] results gave first to Maura, who implemented some awesome ensembling. For commentary take a look at Rachel’s blog post. There’s a bit of discussion in the forum, including my write up of my […]
Re: “there is some systemic problem preventing them (us) from doing so.”
I learned in my games psychology class that this sort of game does not appeal to everyone. It is the sort of highly competitive game that would appeal to young men more than any other demographic group. You have to be careful about designing a game with a prominent leader board since it discourages many players from taking part if they are not already confident in their skills and they are not highly competitive. If you wanted more participation by a broader demographic some things you could do are: make localized leader boards (like for classes) and add more cooperative elements focusing on what the group(s) as a whole can achieve. So that’s my two cents: it’s the game mechanics that doesn’t appeal to other people as much as it does young men, not the subject matter.