Bad films are bad

which is why there's goodfil.ms

Why Ratings Systems Don't Work

Today’s XKCD reminded us of a problem we’ve known about for a while:

XKCD's recent summary of online ratings

A little over a year ago, we started building a movie site with this exact problem in our sights - how could we make film ratings more useful? Bad ratings are all over the internet, and as someone who’s tried to make an improvement, I thought it was worth responding.

The problem with average ratings

Average ratings are, in their essence, an attempt to distil the entire set of ratings about an object (in our case, films), down to a single number. Which seems useful - after all, it saves you so much time deciding whether something’s any good. The problem is that that’s all it’s good for - telling you whether something’s good or not.

If you come across a film and see it has a 7.5+ average rating on IMDB, it’s a sure bet it’s going to be well made, and plenty of people must like it (whether you will, or not, is another matter). If the rating is below a 6.5, you should probably be wary. The rating scale might have 100 points on it, but all the useful ratings are crammed into the top third, and suddenly things are looking like XKCD’s example above. This is the same situation for Rotten Tomatoes ratings, too - great for the ‘is it decent/is it awful’ discussion, but without any depth.

Histograms are even worse

So, if a single average rating isn’t enough, maybe it’s valuable to show the spread of people’s opinions, to try to give a bit of ‘colour’ to the ratings. After all, if something was polarizing, there’s a chance you’ll hate it. The problem is, histograms are just. awful. Let’s look at some real world examples:

Starship Troopers
Fifth Element
Blade Runner

I don’t know how this is supposed to help you beyond telling you ‘these are all 3.5 to 4.5 star films’, which you probably already knew. After all, these movies are superficially similar (well-known science fiction films), but anyone who has seen all three would agree that they have a vastly different character, and appeal to different kinds of people. The histograms aren’t enough to capture that, they end up just adding confusion.

See patterns, not averages

Within Goodfilms, films are rated on two axes - first there’s quality, which is the 5-star system everyone’s used to; and then there’s rewatchability, how much you’d enjoy watching that film again. It means we can display ratings on a graph, instead of a line, and it’s able to convey a lot more information. It takes a little getting used to, but it can capture both the ‘good or not’ question of an average rating, as well as an understanding of the kind of film it might be. The key is that people are really good at seeing patterns, and a scatter plot caters to that. Let’s compare the examples from before:

Starship Troopers

Now, the top of the graph is ‘highly rewatchable’, and the right of the graph is ‘high quality’. So, these ratings say that there’s a lot of disagreement over whether it’s high quality of not, but generally this scores high-rewatchability. So, maybe not the most intelligent movie, but good fun.

Fifth Element

Here we have a cluster in the top right of the graph around 4/5 quality and 4/5 rewatchability. But more interestingly, almost nobody thinks this film is worse than 3/5 in either dimension. Which is great - it means you’re likely to enjoy this movie.

Blade Runner

Blade Runner clearly has the highest quality film of the three, but doesn’t have as many high-rewatchability scores as Fifth Element. This happens with a lot of classics (which Blade Runner is, without a doubt) - they’re very interesting and rewarding to watch, but you wouldn’t rush out to see it again. So, watch this if you’re in the mood for something really good.

We’ve done better

The point of this is we can do better than a 5-star ratings system, and at Goodfilms we think that two ratings, quality and rewatchability, is the right one for films. We’d love to know if you agree, and we would really love to hear about other people who are trying their own novel ratings systems for their own domains. Let’s move past the simple 5-star system, shall we?

Goodfilms

Goodfilms is a way to share the movies you watch with your friends. We rate movies on two criteria - ‘quality’ and ‘rewatchability’, so you can admit to your guilty pleasures and properly capture the feeling you get when a film leaves you exhausted. Sign up now and keep track of the films you love, and find great, challenging or silly new ones to watch.

Comments