Moneyball-ing Eurovision: Could Big Data Help the UK Win the Song Contest?

By James O Malley on at

Tonight Britain answers perhaps the most important question about Europe that it will face this year: Who should represent us in the Eurovision Song Contest?

Six finalists have been selected by the BBC to be voted on by viewers on a special programme - and the winner will then jet off to Stockholm to do battle with our European adversaries on the 14th May.

Unfortunately, Britain’s recent record is less than glittering, finishing last three times since 2000. In 2003, Jemini scored nil points, and last year Electro Velvet clocked up a total of 5 points. This is a depressing state of affairs. Britain is a country that is steeped in popular musical heritage. Why are we so bad?

In this time, Britain has tried several different approaches to picking a song that will excite the European masses: We’ve tried subjecting acts to a primary vote (that’s the method being brought back this year), we’ve tried BBC bosses simply picking someone, we’ve tried drafting musical supremo Andrew Lloyd Webber in… but whatever we do, we just don’t win.

What’s clear is that whatever Britain is doing isn’t working. We need new tactics. And I think I have the solution...


Not that again. No. We need Big Data.


For most of its history the sport of baseball proceeded much as it ever had, with coaches picking which players to buy and when to play them based on their personal experience and knowledge. It wasn’t a science, it was an art, with an entire canon of received wisdom about the best strategies to use on the field dictating their decisions.

In 2002, under the leadership of coach Billy Beane, his team (the not particularly remarkable Oakland Athletics) suddenly started performing well above meagre expectations. The team had neither the resources nor the money of the likes of the New York Yankees. Yet they were doing great.

The reason for the dramatic change in fortune was because they applied science to the process. Using “sabremetrics” - baseball statistics on player performance - they were able to spend their budget wisely and spot players that had been underpriced by the baseball establishment’s traditional measures. The biases were obvious: While a human would naturally remember a player who can hit big home-runs, they were able to find cheaper players who were just as good at getting on base, scoring just as many runs. Within a few years, all of baseball was transformed as all of the major teams adopted a more scientific approach.

This was later written about by Michael Lewis in Moneyball, and dramatised in an Oscar-winning film starring Brad Pitt and Jonah Hill of the same name.

Can Eurovision learn from Baseball?

So we have a selection system that was dominated by received wisdom and faulty human intuition, which wasn’t achieving the desired results. Sound familiar?

This is why I think it is time that we took a “Moneyball” approach to the Eurovision Song Contest. Instead of picking a song out of the air, or leaving it to faulty humans to pick, can’t we take a scientific approach? Why don’t we analyse what songs have performed well previously, augment it with broader listening data and spot hidden value: What if a big data analysis can tell us what sort of song it is that audiences are just waiting for?

Is This Possible?

If we’re going to make this happen, then there’s two major components we need: A database of existing songs, with data on their different attributes. And then we need some sort of algorithm to make sense of it, and turn it into meaningful instructions that we can then use to find the perfect Eurovision act.

So it is excellent then that the march of technology is making good progress in the so-called “machine listening” field.

Machine Listening

Dan Ellis, a Professor of Electrical Engineering at Columbia University described to me some of the progress that has been made, explaining that machines are now pretty good at figuring out genre and artist, at least for easily-distinguished genres. Apparently it works mostly by recognising the instruments used. Current machine listening is also pretty good at transcribing the beat and tempo of a piece of music, as well as the chords and the notes - though “getting the full polyphonic transcription is still hard, but we've made a lot of progress. Figuring out which instrument is playing each note is still hard”, he warns.

There are also advances in figuring out the emotion or mood of the music. But ultimately, more raw data from humans might be needed.

“A lot of this kind of machine learning work depends on the kind of data you can use for training - you need pairs of the music content, and some human-relevant labels you want to predict,” he states.

Ellis suggests that, a website that collects user listening data and contains a user curated music database might be a good source of such data, as that already has many tracks that are tagged for different moods, styles, genres, and so on.

On a technical level, Dr Bob Sturm, a lecturer in Digital Media at Queen Mary’s University, agrees with Prof Ellis about the difficulty in picking out which notes are being played by which voice/instrument on a track when multiple instruments are being played simultaneously. He calls this the “cocktail party problem”. But in his view there is also a more fundamental problem: We can’t yet make machines actually think like humans.

We can create to a certain extent “general schemas to help translate the digital sound data into representations that move closer to our conceptions of things in music: frequencies, harmonics, noises, silences, impulses, pitches, chords, instruments, voices, beats, melodies, rhythms, structures, styles, and so on”, but this isn’t enough.

“Essentially, we want to endow a computer with the same sensitivities as a human who belongs to a particular culture, and is familiar with particular musical practices," he said.

“So, low-level information is within reach, but the high-level information — essentially what we find useful for talking about music — is not.”

Machine Composition

Analysing the data is one thing, but what about the act of composing the song? Could we automate the process of actually writing the song for our Moneyball Eurovision entry?

Bob isn’t convinced. He says my idea is “fun” but “based on a few unsupported premises” - which is academic jargon for “bullshit”. He thinks it isn’t true that studying winners will tell us how to make winners - nor is it necessarily true that a hit song is caused by its sound.

“Music is inherently a human activity, steeped in a cultural environment outside the purview of any data analysis”, he says. And to be fair, he has a point. Eurovision is a contest in which Jedward have competed multiple times.

Wogan wept...

Dr Rob Toulson, a composer and the Director of The Cultures of the Digital Economy Research Institute at Anglia Ruskin University, seems to agree. Though he does think there are already creative and technical “suggestions” that will increase the chances of a song being a hit that we do know about.

“With respect to pop music, listeners are often hooked to the chorus or the lead vocal, so it makes sense to start a song with that key hook in order to first capture the listener’s attention. Repetition is well known to draw people in too – think of Kylie’s ‘Can’t Get You Outta My Head’ for example, like it or not you are hooked.“

He also points to the Australian artist Sia, whose song ‘Chandelier’ is perfectly designed to manipulate us. “the verse moves into the chorus and just as the chorus ends, instead of dropping back to the verse, it raises up into an even bigger chorus that takes the listener to a higher and ongoing emotional state.”

Sadly though, this might be too hard for computers to do.

“It’s all about tapping into human emotion that makes a song a success, string sections and guitar solos translate emotions that cannot be said in words, and those hooks make us want to listen to a good song over and over again. But it’s exceptionally hard to predict. Indeed, if something is predictable then it is more likely to not be successful – which is the entire conundrum of the problem, as humans we identify with patterns and consistency, but we want to be challenged and surprised too.”

So it appears to be that in baseball, a player can be boring and still be successful (a run is a run, even if the player’s favourite colour is beige and his favourite band is Coldplay) - but music requires almost the opposite. Which is harder to analyse.

Dr Toulson points to the work of Dr Susan Rogers, who is a psychologist at the Berkley College of Music, but who is also a former record producer who has worked with Prince and the Barenaked Ladies. “She explained this well at a music production conference recently, saying that music producers need to ‘violate the listener’s expectation’ in order to gain a connected and emotional response”, Rob said.

“People get bored very quickly and we intrinsically celebrate innovation and uniqueness. Unfortunately algorithms only work well on patterns and predictability, so in my opinion it will always take a creative and artistic human to violate those listener expectations and create successful songs that the algorithm would predict to be failures.”


So Could it Happen?

So I’ve made the arguments for a “Moneyball” solution, and it seems that though the technology for analysing music is getting better, we’re still some distance from being able to ever task a machine with writing the hit that Britain so badly needs. Even the experts, it seems, are sceptical about this approach.

Undeterred I also contacted Daz Sampson, who found fame in 2006 as our British Eurovision entry with his song 'Teenage Life'.

He ended up coming in 19th place out of 24 acts and scored a grand total of just 25 points. Surely he, more than anyone else must understand the pain of losing? I asked him if from his unique purview, whether he thought a more data driven approach could help us steer well clear of nil points. His response:

“sadly i dont do much eurovision stuff nowadays but i will say until the [BBC] bring me back then basically we are fucked.” [sic]

Not you too Daz! Surely there must be something to be optimistic about? If not my hare brained scheme, is there any hope in this year’s potential British Eurovision songs?

“as for this years british songs WTF go to any music lesson in any british primary school and you might get better songs.” [sic]

The problem, as Daz sees it, is “it's been run by dicks [-] full stop”.

So I guess what he’s saying is that even if we could come up with an algorithm that could write the perfect Eurovision song, we’d still be struggling with larger, external forces that are beyond our control. Ah well, looks like Scandinavia will be winning again.

This post was originally posted on 29th February