Now we’re into 2013 and Malmo is getting ever closer, the thoughts of many are turning to the rule changes being made for the 58th Eurovision Song Contest and what it could mean for the show. And by ‘rule changes’ we mean the alterations to the running order. To recap, the running order will no longer be determined by drawing lots, but by the production team.
The fans are up in arms and if you ask them now there is still a huge resentment, the running order ‘will decide the winner of the Contest’, ‘it allows a fix’, and ‘it’s not fair’ being typical viewpoints. That’s a viewpoint not shared by Jon Ola Sand, Executive Supervisor of the Eurovision Song Contest, who told ESC Insight that “there is indeed no significant statistical impact of the running order on the result” and “the ESC Reference Group are confident that both artists and viewers will benefit from this decision.”
That’s a line of thought that is hard to understand. Everyone knows the running order has an impact on the result… doesn’t it?
Glen Vella needed just one more point…
Over the holidays the ESC Insight team have been doing some research through the world of statistics and academia to try and determine just how much of an effect the running order could have on the 2013 result. We’ve been crunching numbers, running scenarios, and reading so many papers on statistics we think we’ve finally worked out what a z-value is. The conclusion is a simple one. The running order does impact on the result.
So were the academics right? Or was the EBU?
It turns out that they both were.
The Running Order Myth That Fans Hold On To
Over the last month, I’ve been asking Eurovision fans casually how much influence they think the running order has on the final results. While some people have said as much as 65%, the general consensus is around the 40%-50% mark. Given that belief, is it any wonder there was so much noise and fury when the changes were announced last year?
Part of the problem here is that idea of the influence feels right. The running order, something that fans delight in discussing and using it to project the final result is a time honoured tradition. How many episodes of our Juke Box Jury previews have contained the phrase “if this gets a late draw it’ll do well”? Looking at results such as Lane Moje (singing fifth) and Chiara’s Angel (singing third) both being beaten by songs later in the order, fans have the circumstantial evidence and hunches to back up the myth.
‘Perhaps I should audition for ‘The Killing’?’
The Contest Must Also Be Entertaining Television
First of all, the Eurovision Song Contest is not a fixed format. It continues to evolve, as Sharleen has pointed out in All These Changes Are Good For Eurovision. Do you still want Eurovision to be aired in black and white, with an orchestra attempting to cover Euphoria, to an invited audience in evening wear?
The running order change is one that the EBU believe will improve the show. Jon Ola Sand, Executive Supervisor of the Contest told ESC Insight when the change was announced that “what does impact the Eurovision Song Contest as a whole in terms of ratings and televoting participation, and most likely the result as well, is if numerous ballads or up-tempo songs follow each other up, in other words, if the composition of the program is done randomly.” As to the running order impact, it’s back to our favourite quote of “no significant statistical impact of the running order”.
The key here is the phrase ‘significant statistical impact’. As you’re about to see, there is an impact, but just because something might not be statistically significant, that doesn’t mean the impact can make a significant impact in the real world.
Here Comes The Mathematical Bit
(ESC Insight would like to thank John Egan for his help in navigating these statistical papers).
Much of the heavy statistical work here is derived from three academic papers looking at how events are judged when viewed sequentially, and in two of the papers the data is drawn specifically from the Eurovision Song Contest.
- Recent Developments in Quantitative Methods For Literature Reviews, Rosenthal & DiMatteo 2000.
- Save The Last Dance For Me: Unwanted Serial Position Effects In Jury Evaluations, by W. Bruine de Bruin, 2004.
- Expert Opinion and Compensation: Evidence From A Musical Competition, Ginsburgh & Van Ours, 1993.
After studies by Rosenthal & DiMatteo, followed up by Bruine de Bruin, the scores and results from the Eurovision Song Contest were standardised (allowing different elements to be compared). Analysis these results, and Bruine de Bruin is clear “…the Eurovision Song Contest showed linear order effects of a similar pattern and a similar magnitude: Scores increased with serial position.” The songs nearer the end of the running order score more. In statistical terms, the relationship is:
r = .23
z = .23
95% c.i. for z = .04, .42
p < .001
Which is, to use academic language, a significant statistical impact. In plain English, the running order’s impact on the final result will be around 5% of the final score.
What Does Five Percent Mean For Eurovision?
When you translate that 5% onto the final order of results, the impact is going to be measured in less than single places (perhaps two places in extreme circumstances). If we go to extremes and knock 5% off Loreen’s winning score of 372, and add 5% to the Buronovskiye Babushki’s 259, the final result of 354-272 would still be convincing. With the majority of recent Song Contest winners having a clear margin of victory, the running order would not alter who had won the majority of previous Contests.
Nothing’s gonna stop us now
But the running order’s 5% contribution could impact the Contest. Take the landmark 2003 Contest, where the top three songs scored 167, 165, and 164 points. The running order of those songs were 4th (Turkey), 22nd (Belgium), and 11th (Russia). Switch Russia and Belgium around and you could give Russia the opportunity to make up an extra three points, take the Contest to a tie-breaker, and win the 2003 trophy (more countries had voted for Russia, and they had five douzes, compared to Turkey’s four).
Would Russia have sung later in a producer led draw? There would have been a strong argument to put the biggest named act, with the strongest PR message, into a much later position in the order to keep people watching and allow the tension to build. If t.A.T.u. had sung in the last four songs, the model shows that a few more points would have been scored purely because of their position in the draw.
The Semi-Finals are even more fraught with danger for a fixed running order. Bulgaria and Norway both finished the second semi-final in 2012 with 45 points, but on count-back the final place in the Grand Final went to Tooji from Norway. Norway were drawn 16th in the running order, Bulgaria were drawn 8th. Altering the running order could award an extra point for Bulgaria if they were to sing later, or a dropped point if Norway had been earlier in the draw.
The Semi Final situation for 2013 is (slightly) mitigated because the Countries who will sing in the second half of the draw, arguably where there is more statistical chance of qualifying, are drawn at random. But the producers of the show will still decide on the running order within those two blocks, and of course the entire running order for the Grand Final on the Saturday night.
While it would never derail a ‘Euphoria‘ or a ‘Fairytale‘, in a close contest the 5% impact the running order will have a significant impact on the result.
ESC Insight’s Thought For The Day
Dear fans of Eurovision, relax, the sky is not falling. But that doesn’t quite let the EBU off the hook.
Putting aside the 5% issue illustrated above, there are still a number of issues around the running order that could cause problems because a producer led order is not just a dry issue of statistics and reassuring delegations behind closed doors. As the name suggests, the Eurovision Song Contest is a contest. And running a contest means not just being fair and neutral to every contestant, but being seen by everyone involved as being fair and neutral to every contestant.
Someone is going to sing second on May 18th, that much is obvious. The myth is that the second place song will never win. On the Friday morning, a delegation is going to be told that the production team have put them in slot number two – and the perception will be either that the song has lost what little chance it had for victory, or that the producers do not want that song to win the Contest. The delegations will likely accept this, but will the rest of the world?
Dino Merlin drew second, and had to sit next to Jedward
The perception of the EBU running a fair and equitable Eurovision Song Contest is just as important as the actual running of the Contest. The workings of any contest must involves clear communication with all the stakeholders, and that includes the delegations, the performers, the fans, and the viewers. For as long as I can remember, Eurovision commentators, fans, and delegations have believed that a later position in the running order would give them a better result.
In all of this though, I want to leave you with one thought.
While the data available shows that the running order does have an impact, it also shows that the impact is less than many would believe. In a year with a song that is head and shoulders above the rest that song will win from first, last, fifteenth, or even the ‘dreaded’ second place.
But if 2013’s songs include a number of potential winners, or there is a very competitive semi-final, the producers of the show will have a direct impact on the final result.
Over the next few months, ESC Insight is going to look in detail about the impact of the running order on the judging procedure, the phone line opening debate, how songs are perceived and assessed, how Delegations and the EBU can improve the jury procedure, and more issues around the Contest. Follow us online, via Twitter, Facebook, and RSS, or through our Email Newsletter.
Images and thumbnails courtesy of the EBU / eurovision.tv.
What is the difference between the years of televoters and the jury results for the breakdown, if any? That would be a more interesting view of debate.
The running order is more important than this. Does the 5% effect more the higher points (getting a 10 or 12) or getting a 1 or 2 points on the lower end of the scale. Of course the full televoting breakdown is really needed to fully assess…
The bigger issue is that countries are going to pick high profile artists to get later in the running order as you suggest. Look at Melodifestivalen to see how that works to this effect. Even if it isn’t much of an effect, it is the only real effect a country can make in their selection. Styles of songs that ‘end on a high’ are going to be entered to try and get the ending positions. A smaller effect, but one that will be magnified by the culture of the contest which then doesn’t produce an equal playing field for different types of songs and songwriters.
The other changes through time have opened up new opportunities to songwriters in terms of style, whereas this then starts to limit them.
A similar study looking at Melodifestivalen would be interesting to see as that is the model Eurovision is going to be like, although sadly only on a 8/10 song level and not 25 where additional factors like time difference and country differences also affect results.
Overall, to the average Eurovision viewer, the effect this will have is likely to be 5%, if not less, they will seemlessly not notice a difference I am sure. To the deeper darker inner workings and strategies of winning the world’s biggest songwriting contest (and the political fallout of maybe being ‘drawn’ 2nd in international media) the effect I suspect will be larger.
Maybe that is no bad thing, but I don’t want to be the judge of that.
Excellent point on the (unintended?) consequence of altering the ‘preferred’ style of song at ESC, Ben, thanks.
“The running order’s impact on the final result will be around 5% of the final score”
How can you possibly believe it’s that low? I don’t think it’s as high as 50% and someone will eventually win from 2nd, but 5%? Don’t believe it.
People always save… “the best for last”!!!!
There may also be the feeling now possibly not to send a too upbeat song as you have a feeling that they are gonna stick something lively to open the show and not a snore fest that we had at the start of 2006 and 2012. You can tell that this year there is going to be outcry if a big name country like Russia or Azerbaijan get put in one of the first spots and they come in like 20th place, we all know that Azerbaijan will more than likely go mad at the EBU.
Also here is another thought, who will be placed around Sweden, is there anything stopping the producers from putting a few of the poorer songs just before Sweden to lift their chances?
The statistics quoted are from a paper–ONE paper–that queries whether performing first or last increases the chance of winning the Contest. It doesn’t parse all all possible positions in the running order. It also doesn’t look at doing “well”–be that top 3, 5 or 10–it looks only at winning. And it found that definitely performing first or last in the “douze points” era BEFORE the semi-final system was in place contributed a 5% advantage over all the other placings combined.
But bear in mind a few things:
+many years had less than 15 entries
+this was juries only for most of the Contests
+every year is, effectively, a new Contest: mostly new juries, all new entries, different running orders.
So there’s so many variables at play…it’s surprising there’s any statistically significant relationship between winning and any one element. If I were to guess at another it would be that having a female lead vocallist might have a slightly bigger effect, since between 1975 and 2003 10 winners had male-only lead vocals (versus 18 with lead vocals wholly or partially sung by females).
My argument agains the producers selecting the running order is simple: bias. So long as a draw is random, no one can complain it’s been rigged. As soon as someone–or a small group–choses the running order, the accusations of a fix will start.
BTW between 75 and 03 the winner performed first or last in 75, 76 77 82 83 84 and 89: 8 times out of 28 Contests.
Exploring a different angle:
The fairness of a random draw ends after the draw; Any specific sequence will probably include unfair situations.
The difference in perception is relatively weak consolation as long as the effect remains.
Conjecture:
It is possible that good show management, along with other efforts, can raise interest and participation among televoters that leads them to judge songs more like fans, and potentially resist the effect of running order itself.
If fans believe that they, unlike the masses, are above the effect of running order, then any means to increased fandom must reduce the effect. We could say that voter engagement, rather than running order, is what is really significant.
Deciding the running order poses a tonne of risks against the fairness of the contest, and the EBU seem to have taken this step with almost complete disregard to the myth of the running order. I have a feeling that if they were to see this article, they might find reason for further discussion amongst themselves.
Fans are going to want to know why a song is in a specific place, and are going to come up with a lot of their own kneejerk conspriacy theories which are going to drive us all mad in a heartbeat. The EBU need to try and build a specific model based on something that is measurable and beyond debate in music like BPM and visual differences between adjacent performances, rather than just having a bunch of producers treat it like an iTunes playlist. The producers line of thinking is essential to explaining why they chose to put a certain song in a certain spot, and for complete fairness, they would need to have the same reason for doing so for every entry in any one show. I would be surprised if they have not thought of this… but the EBU do seem to have resorted to casually thinking “most people won’t suspect anything, this song sounds nice after this one, and this livens things up a bit afterwards…” I do hope they’re treating this approach with more responsibility than that.
The 5% statistic is believable, because I do think that a good enough song can win Eurovision from any starting position. If we have another big favourite this year like last and in 2009, I genuinely hope the producers stick them in #2 just to make everyone eat their own hats once they still win, and just maybe this will make everybody calm down for 2014.
John Eagan “My argument agains the producers selecting the running order is simple: bias. So long as a draw is random, no one can complain it’s been rigged.”
As long as commentators tell voters who to vote on, start slot does not matter…
IMHO
It is much better for an artist to be liked by all commentators and start as no 2 than to be disliked and have the last slot.
I do not accept the findings of Bruine de Briun’s (2004) paper as it has a number of short comings, particularly the sample size. It is a well established fact that running order bias occurs frequently in competitions requiring selective judgement (Wilson, 1977; Flores and Ginsburgh, 1996; Page and Page, 2010).
I agree with the findings of Page and Page (2010) which examined running order effects in the “Pop Idol” series in various countries. They found that EACH additional slot in the running order makes it 5% more likely to score points. The cumulative effect would mean a song in the 25th position would be 3.38 times more likely to score a point than the song in 1st position.
I’m also drawn to Page and Page’s (2010) explanations of why this may occur. Memory certainly will play a role, however they also found that when judging contests people are likely to compare the songs unique features to that of the last song. If the producers decide the running order this will actually accentuate the unique features of a song. This could in theory actually make the running order more important.
This in mind I would also recommend to counter act the bias in the running order. The running order should be prepared for the public show but the running order should be reversed for the jury final.
Everyone has valid theories here, but the best thing, and the only thing that we as fans can do is see how it pans out. In many years song’s have lost out through being in a line of similar songs…no doubt that both me and my guitar life looks better in spring both lost a few points through being one after the other. This draw may help the unique features of each song shine to the effect that the contest could be fairer for everyone. Of course bias is still a concern, and I really want to know how and when the running orders will be announced. I feel that they are likely to not be revealed until the contest itself so that the excitement of seeing your country’s song overcomes the feeling of ‘oh we got number 2’