Eurovision’s Running Order Bias Is Real and Unavoidable Written by Ben Robertson on April 25, 2023

The Eurovision Song Contest community is well aware of terms such as “’he death slot’ or ‘the pimp slot’. These refer to being drawn second or drawn last in the running order at the Song Contest, and are seen by the community as being the worst and best running order positions respectively.

Evidence backs up this theory, with academic research showing a clear relationship between being performed later in the show with a higher chance of performing well. This is backed up by research from similar shows, such as the Idol franchise. The modern-day data for our Eurovision Semi Finals shows that, yes, second place in the show has the most negative drag, with last place the most advantageous.

However, there is little surprise that audience-based television shows create such a pattern. During the Eurovision Song Contest, one of the most notable statistical quirks is that the number of people watching is not constant. Data from the Spanish broadcaster last year shows that the number of viewers has a near-perfect correlation with running order (except, of course, a boost for people tuning in for Spain’s performance) with Stefan, the act who closed the show, being watched by over three million more viewers than We Are Domi who opened the show.

Viewing figures in Spain during the 2022 Eurovision Grand Final

The simple hypothesis here is that running order bias, the idea that the later in the show you perform, the better your result, is purely a product of this effect. If viewing figures are better towards the end of the show, then there are more prospective viewers watching.

We at ESC Insight have argued for years, since the running order at Eurovision was first decided by producers in 2013, that human influence on the running order would have an impact on the results. Jon Ola Sand, Executive Supervisor for the Eurovision Song Contest at the time, commented that there was “no statistically significant impact” on the effect of running order placing on the result.

Now is the time to challenge that statement, and we want to do so without arguing it is because millions more people are watching.

The Perfect Timing

The inspiration for completing the following data analysis came from the rule change that came with the 2023 Eurovision Song Contest. This year in Liverpool the ten qualifiers will be decided fully by televoting, rather than the 50/50 jury/televote split we have used in recent years.

That means the data set that we want to use to prove if running order exists will no longer be possible. Since 2014 the EBU has published the full rankings of all scores in the Eurovision Song Contest, the full first to last place rankings from each televote but also from each juror, in both the Semi Final and the Grand Final. Roughly 200 jurors each year (5 from each participating broadcaster) vote on the Eurovision Song Contest and from 2014 to 2022 their rankings from the Semi Final to the Grand Final are publicly available.

It is this data my analysis focuses on. The jurors are, with few exceptions, the same people from the Semi Final to the Grand Final. They don’t change like televoters do, nor should they be jumping out of the room for a toilet break or to put the kettle on. There are not more people joining the jury later in the show, but the same five people in theory are equally focused from opening act to closing act. 

The Eurovision jurors are therefore the perfect control group. If their votes are impacted by running order bias, then we have proved that this competitive improvement in songs towards the latter part of the competition isn’t just due to more voters, but intrinsic human nature.

And now is the time to study this, to use the data available from 2014 to 2022, as we will never get such a detailed data set of jurors’ voting records in both Semi Final and Grand Final ever again.

Methodology

To assess if running order has an impact on the results of jurors at the Eurovision Song Contest, our methodology needs to isolate each juror’s ranking of the songs that took part in both the Semi Final and Grand Final. Let me show how this works by using data from just one juror.

The results below are from Carrie Grant, who was head of the UK jury at the Eurovision Song Contest in 2014.

The 10 countries listed here are the ten countries that qualified for the Eurovision Grand Final from the 2014 Semi Final. That means Carrie Grant voted on these songs twice, once on Wednesday 7th May night and once more on Friday 9th May.

We first convert the rankings of these songs in the Semi Final into a 1-10 ranking, removing rankings of other entries that did not qualify. We then do the same for the Grand Final, creating a 1-10 ranking of only these ten entries.

Our next step is to use simple addition and subtraction to work out how the relative rankings of the songs changed in the 48 hours between voting. For example here in the Semi Final Carrie Grant ranked the Maltese song ‘Coming Home‘ eighth in the Semi Final, a ranking position of 7th compared to the ten songs that qualified. By the time of the Grand Final, Carrie had switched her vote for Malta much higher, to a placing of 3rd out of 25, and 3rd compared to the 10 songs from the same Semi Final (behind Austria and Finland). This resulted in Malta being given a relative ranking increase score in this system of +4.

We then compare the change of each ranking position to the relative change in running order. Each running order position is given a value between 0 (opening the show) and 1 (closing the show). In Malta’s case, they opened the Second Semi Final in 2014 but were drawn late in the second half of the Grand Final, and in this system we calculate that their relative running order position improved +84% compared to the Semi Final.

Of course Carrie Grant was not the only juror voting on the 2014 Second Semi Final and the Grand Final, and rather than just taking her rankings, we combined all of the rankings together of all the jurors and assessed how the total rankings increased or decreased for each of the entries. This total ranking was converted into a 1-10 ranking (1 being the most gain in the Grand Final, 10 being the highest relative drop) and also made a 1-10 ranking of running order change (1 being a later running order in the Grand Final compared to the Semi Final, 10 being earlier).

We conclude by then running Spearman’s Rank correlation analysis on this data set. This gives us a value between -1 and +1. A value of -1 suggests a perfectly negative correlation (suggesting earlier running orders place better) and +1 is a perfectly positive correlation (where later running order place better). For the 16 different Semi Finals from 2014 to 2022, the Spearman’s Rank correlation values were as follows:

2014: +0.41, +0.50

2015: -0.08, -0.12

2016: +0.52, -0.09

2017: +0.54, +0.56

2018: -0.66, +0.41

2019: +0.38, +0.61

2021: -0.11, -0.08

2022: +0.30, +0.22

In total 8 of these values are positive, and 6 of these are negative, so at first glance one may believe that there is little to no correlation with running order for how jury members vote. However it is important to note that 5 of those 8 positive values show a positive correlation so strong that each of them, to a 90% confidence level, suggests that a relationship exists between running order and final ranking. In contrast only one negative correlation is so strong to suggest such confidence, the 2018 First Semi Final. This show has quite a few outliers including eventual Eurovision winner ‘Toy’, which placed much better with juries in the Semi Final despite the earlier running order.

If one combines all of these Spearman’s Rank calculations together to calculate a critical t-value for this data we arrive at a value of 2.32. With the 16 results we have above, any value over 2.131 would be enough to give us a conclusion that, with a 95% confidence, a later running order did result in jurors giving songs a better placing.

Based on this I can confidently say that yes, there is statistical evidence that running order bias exists at the Eurovision Song Contest. And that evidence exists within the cohort of music professionals who don’t change and rank each and every performance.

Other Thoughts Beyond This Research

It is important at this point to explain what this research tells us. Yes, we are highly confident that running order bias is a real phenomenon, but we don’t make any claims about how large that running order bias is. Just because the correlation exists does not mean it is anywhere close to a divisive factor between the different competing songs – but it does mean in a close contest it could make all the difference. In reality the size of the difference is small.

It is worth noting at this point though a very shocking fact that emerged while processing this data. We recorded only six jurors since 2014 who kept their 1-10 (or 1-9 ranking if their own country qualified) the same from Semi Final to Grand Final. Every other juror changed at least one of their rankings relative to the other songs.

Yet even though most jurors do alter their rankings from one night to the next, these movements are on the whole small. The following list shows the top 10 increases and decreases from jurors over this time period from Semi Final to Grand Final.

Now the colours here respond to the running order, with green or red colours showing the song had a significantly better or worse running order in the Grand Final respectively. As expected, these correlate very well to which songs did better or worse from jurors. But the movements are small, running order alone can not change a no-hoper song into a contender for victory, and only four songs out of 160 saw shifts in ranking from Semi Final to Grand Final of more than 1 ranking position.

While our data analysis suggests running order is a real and human phenomenon, unavoidable to avoid even from those professionals judging the songs, I want to take some time to suggest another theory. The reason for suggesting this is that we have to remember our investigation took place between 2014 and 2022. During this time the producers were also setting the running order. This human influence on the running order may also correlate to certain songs, those that were coming into the Grand Final on a late surge of hype in Eurovision week also may have ended up getting late running order slots. And that surge of hype may also be what convinces jurors to change their rankings.

If I look at those songs that did much better in the Grand Final with jurors than running order alone would suggest, songs like ‘1944’, ‘Fuego’ and ‘Nobody Like You’, all of them exceeded expectations with their live performance at Eurovision compared to what the community would have expected earlier in the season. As much as this table correlates to running order, it also correlates nicely with which songs got the Eurovision buzz during the rehearsal period.

This would follow a known theory from political science, which one academic quotes as “bias toward maintaining existing affect even in the face of disconfirming information”. What this means in a Eurovision Song Contest context is that jurors are likely to already have gone into a Eurovision Semi Final with preconceived ideas about how they would vote, and their natural bias when challenged with new information (the performance on the Eurovision stage) is to maintain their existing rankings. However with social media coverage, with watching and re-watching the performances back many times, and, most importantly, time, those jurors are able to reflect on those better-than-expected performances by the Grand Final and reward them more accurately on the Friday night. Or so goes the theory.

This would be a great starting point for further research into this topic, so we can understand why we see this correlation with later running orders giving a better ranking for jurors. To extend this further I’d be looking to interview different jury members from previous contests to discuss their rankings and their rationale for why their opinions altered from one show to the next. Another angle of inquiry would be to interview show producers about their running orders, and to understand how their running orders were created and about how they act and react to this statistic about later running orders being more successful when they create the order for the competition.

There are eight different presentations in the EUROVISIONS conference “Science Slam” which takes place in the Capstone Theatre, Liverpool, on Tuesday May 9th. All video presentations are available to view on this link, and the public can vote for their favourite topic here. 

ESC Insight is the official media partner of the EUROVISIONS conference.