Now we’re into 2013 and Malmo is getting ever closer, the thoughts of many are turning to the rule changes being made for the 58th Eurovision Song Contest and what it could mean for the show. And by ‘rule changes’ we mean the alterations to the running order. To recap, the running order will no longer be determined by drawing lots, but by the production team.
The fans are up in arms and if you ask them now there is still a huge resentment, the running order ‘will decide the winner of the Contest’, ‘it allows a fix’, and ‘it’s not fair’ being typical viewpoints. That’s a viewpoint not shared by Jon Ola Sand, Executive Supervisor of the Eurovision Song Contest, who told ESC Insight that “there is indeed no significant statistical impact of the running order on the result” and “the ESC Reference Group are confident that both artists and viewers will benefit from this decision.”
That’s a line of thought that is hard to understand. Everyone knows the running order has an impact on the result… doesn’t it?
Over the holidays the ESC Insight team have been doing some research through the world of statistics and academia to try and determine just how much of an effect the running order could have on the 2013 result. We’ve been crunching numbers, running scenarios, and reading so many papers on statistics we think we’ve finally worked out what a z-value is. The conclusion is a simple one. The running order does impact on the result.
So were the academics right? Or was the EBU?
It turns out that they both were.
The Running Order Myth That Fans Hold On To
Over the last month, I’ve been asking Eurovision fans casually how much influence they think the running order has on the final results. While some people have said as much as 65%, the general consensus is around the 40%-50% mark. Given that belief, is it any wonder there was so much noise and fury when the changes were announced last year?
Part of the problem here is that idea of the influence feels right. The running order, something that fans delight in discussing and using it to project the final result is a time honoured tradition. How many episodes of our Juke Box Jury previews have contained the phrase “if this gets a late draw it’ll do well”? Looking at results such as Lane Moje (singing fifth) and Chiara’s Angel (singing third) both being beaten by songs later in the order, fans have the circumstantial evidence and hunches to back up the myth.
The Contest Must Also Be Entertaining Television
First of all, the Eurovision Song Contest is not a fixed format. It continues to evolve, as Sharleen has pointed out in All These Changes Are Good For Eurovision. Do you still want Eurovision to be aired in black and white, with an orchestra attempting to cover Euphoria, to an invited audience in evening wear?
The running order change is one that the EBU believe will improve the show. Jon Ola Sand, Executive Supervisor of the Contest told ESC Insight when the change was announced that “what does impact the Eurovision Song Contest as a whole in terms of ratings and televoting participation, and most likely the result as well, is if numerous ballads or up-tempo songs follow each other up, in other words, if the composition of the program is done randomly.” As to the running order impact, it’s back to our favourite quote of “no significant statistical impact of the running order”.
The key here is the phrase ‘significant statistical impact’. As you’re about to see, there is an impact, but just because something might not be statistically significant, that doesn’t mean the impact can make a significant impact in the real world.
Here Comes The Mathematical Bit
(ESC Insight would like to thank John Egan for his help in navigating these statistical papers).
Much of the heavy statistical work here is derived from three academic papers looking at how events are judged when viewed sequentially, and in two of the papers the data is drawn specifically from the Eurovision Song Contest.
- Recent Developments in Quantitative Methods For Literature Reviews, Rosenthal & DiMatteo 2000.
- Save The Last Dance For Me: Unwanted Serial Position Effects In Jury Evaluations, by W. Bruine de Bruin, 2004.
- Expert Opinion and Compensation: Evidence From A Musical Competition, Ginsburgh & Van Ours, 1993.
After studies by Rosenthal & DiMatteo, followed up by Bruine de Bruin, the scores and results from the Eurovision Song Contest were standardised (allowing different elements to be compared). Analysis these results, and Bruine de Bruin is clear “…the Eurovision Song Contest showed linear order effects of a similar pattern and a similar magnitude: Scores increased with serial position.” The songs nearer the end of the running order score more. In statistical terms, the relationship is:
r = .23
z = .23
95% c.i. for z = .04, .42
p < .001
Which is, to use academic language, a significant statistical impact. In plain English, the running order’s impact on the final result will be around 5% of the final score.
What Does Five Percent Mean For Eurovision?
When you translate that 5% onto the final order of results, the impact is going to be measured in less than single places (perhaps two places in extreme circumstances). If we go to extremes and knock 5% off Loreen’s winning score of 372, and add 5% to the Buronovskiye Babushki’s 259, the final result of 354-272 would still be convincing. With the majority of recent Song Contest winners having a clear margin of victory, the running order would not alter who had won the majority of previous Contests.
But the running order’s 5% contribution could impact the Contest. Take the landmark 2003 Contest, where the top three songs scored 167, 165, and 164 points. The running order of those songs were 4th (Turkey), 22nd (Belgium), and 11th (Russia). Switch Russia and Belgium around and you could give Russia the opportunity to make up an extra three points, take the Contest to a tie-breaker, and win the 2003 trophy (more countries had voted for Russia, and they had five douzes, compared to Turkey’s four).
Would Russia have sung later in a producer led draw? There would have been a strong argument to put the biggest named act, with the strongest PR message, into a much later position in the order to keep people watching and allow the tension to build. If t.A.T.u. had sung in the last four songs, the model shows that a few more points would have been scored purely because of their position in the draw.
The Semi-Finals are even more fraught with danger for a fixed running order. Bulgaria and Norway both finished the second semi-final in 2012 with 45 points, but on count-back the final place in the Grand Final went to Tooji from Norway. Norway were drawn 16th in the running order, Bulgaria were drawn 8th. Altering the running order could award an extra point for Bulgaria if they were to sing later, or a dropped point if Norway had been earlier in the draw.
The Semi Final situation for 2013 is (slightly) mitigated because the Countries who will sing in the second half of the draw, arguably where there is more statistical chance of qualifying, are drawn at random. But the producers of the show will still decide on the running order within those two blocks, and of course the entire running order for the Grand Final on the Saturday night.
While it would never derail a ‘Euphoria‘ or a ‘Fairytale‘, in a close contest the 5% impact the running order will have a significant impact on the result.
ESC Insight’s Thought For The Day
Dear fans of Eurovision, relax, the sky is not falling. But that doesn’t quite let the EBU off the hook.
Putting aside the 5% issue illustrated above, there are still a number of issues around the running order that could cause problems because a producer led order is not just a dry issue of statistics and reassuring delegations behind closed doors. As the name suggests, the Eurovision Song Contest is a contest. And running a contest means not just being fair and neutral to every contestant, but being seen by everyone involved as being fair and neutral to every contestant.
Someone is going to sing second on May 18th, that much is obvious. The myth is that the second place song will never win. On the Friday morning, a delegation is going to be told that the production team have put them in slot number two – and the perception will be either that the song has lost what little chance it had for victory, or that the producers do not want that song to win the Contest. The delegations will likely accept this, but will the rest of the world?
The perception of the EBU running a fair and equitable Eurovision Song Contest is just as important as the actual running of the Contest. The workings of any contest must involves clear communication with all the stakeholders, and that includes the delegations, the performers, the fans, and the viewers. For as long as I can remember, Eurovision commentators, fans, and delegations have believed that a later position in the running order would give them a better result.
In all of this though, I want to leave you with one thought.
While the data available shows that the running order does have an impact, it also shows that the impact is less than many would believe. In a year with a song that is head and shoulders above the rest that song will win from first, last, fifteenth, or even the ‘dreaded’ second place.
But if 2013’s songs include a number of potential winners, or there is a very competitive semi-final, the producers of the show will have a direct impact on the final result.
Over the next few months, ESC Insight is going to look in detail about the impact of the running order on the judging procedure, the phone line opening debate, how songs are perceived and assessed, how Delegations and the EBU can improve the jury procedure, and more issues around the Contest. Follow us online, via Twitter, Facebook, and RSS, or through our Email Newsletter.
Images and thumbnails courtesy of the EBU / eurovision.tv.