Predict Oscars 2019 with Data Science
Being a filmmaker and data scientist it is my Holy Grail to predict the Oscar winner. Which nominee is your Favourite? (no more puns I promise)
Collecting data
The first step was to decide on what information I want to use to make predictions. Similar analysis in the past usually used other awards like Golden Globes or Director’s Guild Awards. Knowing the nominees and winners of those is certainly a helpful feature.
Scraping award winners
The easiest way to scrape award data would be to use IMDB. Unfortunately they prohibit such behavior. So instead I used wikipedia. It has a pretty consistent way of storing movie award information but there were some exceptions. I picked 7 awards that seemed to have the biggest impact.
I collected the nominees and winners of those awards since 1960. Each award was turned into a feature with 3 values:
Notebook: Scrape Awards From Wikipedia
Getting critics’ ratings
Wanting to go a step further I decided to also use movie critics’ ratings. Critics publish their reviews in variety of places over the internet. Thankfully Rotten Tomatoes aggregates and presents them in a nice consistent way. Each review has a binary rating where fresh means good and rotten means bad. This is actually decided by Rotten Tomatoes staff based on their reading of the review. Binary rating is simple but it’s better to have something more granular. Thankfully most of the reviews also included a more classic X/Y rating that critics published in the original article. Of course each reviewer can have a different way of rating, so they needed to be properly normalized into 0–1 scale.
I only used reviews that were published before the Oscar ceremony and therefore were not affected by the decision. Unfortunately there weren’t that many reviews posted in the internet before 2000, which forced me to only consider last 20 years when using critics’ ratings.
Code: Scrape Rotten Tomatoes reviews
Model
The major issue in this project was very small number of examples. Between 1960 and 2018 there were only 319 Oscar nominees. To evaluate my model I used leave-one-out cross-validation. Each year in this period is tested separately with training data coming from all the other years.
Metrics
In the world of movie awards there is no place for runner-ups. Only the winner counts. That’s why the most important metric was the accuracy — percentage of ceremonies in which the model correctly predicted the winner.
While simple, this metric is not telling the full story. As there are always at least five nominees, it’s still quite a good result if the winner had second-best predicted score among them. Model that is consistently close to predicting a winner is better than the one that was just lucky few times.
I came up with two metrics:
Normalized Score
For each year, scale the probabilities returned by the models so that the maximum is 1 and minimum is 0. Take the average within the winners.
Rank Score
Take the average Rank Score of the winners. Rank Score is equal to 0 for the lowest and 1 for the highest rank (others are uniformly distributed over 0–1 range)
In order to choose which one is more correlated with the accuracy, one could do some math calculation. Instead I did a randomized experiment. I generated a sample of score distributions and calculated all three metrics for each item. It turned out that Rank Score is empirically more correlated with accuracy so I decided to use it for the hyper-parameter optimization.
Classifiers
The final classifier was a combination of 5 different classifiers from scikit-learn library:
The final score was a weighted average of probability scores from those classifiers. Hyper-parameters and weights were determined in another layer of cross-validation. Classifier’s weight was equal to its Rank Score over the training set.
Code: Prediction Model
Selecting Features
One dilemma I had to solve was whether to actually use critics reviews in the final prediction. The problem with those features was that they didn’t cover the 20th century. I limited evaluation to ceremonies after 2000 and compared results of models consisting of award features, critic features and both.
It turned out that there was no significant gain from using critics reviews so I decided to only consider award features and run prediction on data from 1960–2018 period.
Results
The results were pretty good. The model was able to predict the winner correctly in 46 out of 59 ceremonies. Only in 3 cases the winner was not within the first two predictions.
Important thing to note is that since 2010 the number of nominees increased from 5 to at least 8 each year. Understandably it makes predicting more difficult. We can see that the accuracy decreased over the years but the other metrics were more consistent.
As we can see, winning other awards can be a good indicator of success during the final gala. However, there are cases when the decision is a big surprise. Braveheart was nominated just for Golden Globes and DGAs but it didn’t stop it from winning with Apollo 13 — winner of DGA, PGA and SAG. I guess everyone was convinced by Mel Gibson’s Freedom Speech (I know I was).
The award that is most correlated with winning the Oscar is given by the Directors Guild of America. What’s more surprising is that success in Cannes is not a good sign if you want to win the academy award.
And the winner is…
It’s not a big surprise as most of the experts put their money on Alfonso Cuarón’s beautiful semi-autobiography. If Roma wins it will be the first foreign film to achieve this. Green Book also stands a good chance being a winner of Golden Globes and PGAs.
Conclusions
Based on the historical results, there’s a 95% chance that the winner will be Roma or Green Book. First will make you cry and the second will make you laugh — both in great style and for very good reasons.
Some of you may complain that it spoils the show if we know everything ahead of time. But as academy is expanding it becomes harder to make the correct prediction. And to be frank, it’s probably for the better.
📝 Read this story later in Journal.
🗞 Wake up every Sunday morning to the week’s most noteworthy Tech stories, opinions, and news waiting in your inbox: Get the noteworthy newsletter >
