Judge, Jury, and the Execution of the Eurovision Song Contest Written by Ewan Spence on April 24, 2013 | 12 Comments

Previously on ESC Insight, we  took an academic look at how the public judge entries in the Eurovision Song Contest. Now it’s time to turn our attention towards the other half of the scoring process at Eurovision… the jury vote.

This article is going to talk about how juries work; the thought process of individual jurors; and how elements such as the running order, national bias, ill-defined guidelines, time constraints, and a lack of training all impact on the jury results.

One question you might ask is ‘why go to all this trouble, it’s just a television show?’ And I think that’s the issue. The Eurovision Song Contest is more than a television show. As the EBU remind us, it’s the largest entertainment show on earth, it’s the largest singing contest, and it was set up to not only as a technology test-bed, but also to help bring together a continent ravaged by war and distrust. If you are organising a Contest with that sort of scale and  history, part of the remit should be to provide a fair and neutral platform for everyone to compete on. That is why this is an important topic.

Why Do We Need To Look At The Juries?

Since 2009, the national juries have made up 50% of the final vote awarded by each broadcaster at the Eurovision Song Contest. While many Eurovision pundits pour over the running order and the on-stage performances to try and ascertain what the public will make of them, the impact of the jury does not receive as much attention. In many discussions you might hear ‘the jury will go for that’, or ‘that’s one for the juries, the public won’t vote for it at all.’

The current jury system has a number of flaws that have the potential to introduce unwanted bias into the final result of the Eurovision Song Contest. These flaws are well documented in other judged contests, and the EBU should look to them to help improve the jury system for the Contest.

The Eurovision 2011 Winner… according to the juries.

It’s important to highlight that the juries are human, and will be affected by the running order and the presentation in exactly the same way as someone watching the show on their television at home. Our key findings (from ”) can be summed up as…

• The producer led running order includes a remit to program ‘genre’ songs away from each other so they can ‘stand out’, which assumes the audience is using a contrast model of comparison.
  Mussweiler, T. 2003, Comparison Processes in Social Judgement: Mechanisms and Consequences (Psychological Review 110 (3) 472-489).
  
• Academic studies show that people evaluate songs by comparing them to the previous song, comparing similar elements to decide which is the better song… an assimilation model of comparison, the opposite of the model adopted made by the production team.
  Drs Lionel Page and Katie Page, A Field Study Of Biases In Sequential Performance Evaluation On The Idol Series (Journal of Economic Behavior and Organization (2008).

How The Juries Will Work At The 2013 Song Contest

As in previous years, the juries will assemble to watch the second dress rehearsal of the two semi-finals and the grand final on the night before the live show (Monday 13th, Wednesday 15th, and Friday 17th May 2013). Their votes must be submitted by 2300 CET that night, which means that they can watch the show just the once before deciding their rankings and submitting them to the official voting partner, Digame. There will be no official opportunity to listen back to songs and watch the performance again, perhaps to run two similar songs against each other to see how they compare, to check for vocal flaws, or to simply remind themselves in full of a song from earlier in the running order.

This leaves the five member juries open to the same running order bias issues that affect the public. But remember that the public are simply asked for their one favourite song, while this year the jury are being asked to rank every single song that they hear to come up with a ‘1 to 26 order’, where every position will be important in the final result. Objectively that’s a much more difficult task, made doubly so by the time pressure the jury members will be under.

What Information And Guidance Do The Juries Receive?

So what do the EBU actually ask of the jury members? There are two areas to look at, the process and the criteria.

There are extensive details on the technicalities behind the voting, detailing the process that happens between broadcasters, jury heads, the jury members, and Digame. The EBU places guidelines on who should be in the jury, and advises on the timing, and the process of submitting individual juror’s rankings digitally to Digame.

Jury members are asked to rank the songs from 1 to 26 (or 16 and 17 in the two semi-finals). On the ranking form provided to each juror, they are asked to focus on a number of areas. Following a similar process to previous years, for the 2013 Eurovision Song Contest the EBU will be asking all jurors to focus on the following areas:

• Vocal capacity.
• The performance on stage.
• The composition and originality of the song.
• The overall impression by the act.

This is a good start by the EBU (and it’s worth pointing out that contrary to the belief of many, the EBU does not consider ‘commercial success’ a valid criteria), but if the purpose of the jury is to balance out an impulsive public vote, then the current methodology is not sufficient, as the juries are also open to the same impulses.

Running Wild, Running Free

That’s not to say that the jury are not aware of these issue. EscXtra’s Luke Fisher served on the Jury during the Austrian National Final this year, and Fisher felt that he had a duty to balance out the expected views of the public. “Whilst ORF did not give us any guidance on how to vote, merely ‘to rank them in order of preference, I did feel a responsibility to balance out my votes with how I expected the public to vote,” Fisher told me. “I felt that I should ‘vote up’ a song that I thought wouldn’t appeal to a televoter but deserved to do well”.

There may be a list of areas to focus on, but without guidance from the organisers, jurors are left to decide their own criteria for voting, to add in their own personal thoughts on what areas should be compensated for, and if they should take a personal view of each song from their own viewpoint, or how the song fits into the wider musical industry.

The impact of how jurors judge can be minimised with clear and defined instructions from the EBU. This is a process that other competitively judged contest already employ. One clear example is in the Word Figure Skating Championships.

Looking at the protocols of between 1994 and 2000, the selection and training process is clear. Nine judges are selected at random from an international pool of judges (with the caveat that each judge of the nine must come from a different country). All of the judges in the pool “receive extensive training to achieve high-inter-rater reliabilty, had years of world-level jury experience, and were continuously checked for nationalistic bias.” (Weekley, J. A., & Gier, J. A.,Ceilings in the reliability and validity of performance ratings: The case of expert raters (Academy of Management Journal 1989, 32, 213–222).)

Bolero, Bolero, where the sun is always shining on you…

Further studies by W. Bruine de Bruin (Save the last dance for me: unwanted serial position effects in jury evaluations. Acta Psychologica 2005, 245-260) show up a few more points that I feel are relevant to the Eurovision Song Contest.

The first is that through the training and input from the International Skating Union (the ISU), neither the final scores or the individual juror scores exhibited national bias (de Bruin p256). But this training did not eliminate any bias because of the running order and the serial nature of judging.

Figure Skating relies on ‘step-by-step’ voting, with each contestant receiving their score directly after competing. Other judged sports and contests rely on end-of-sequence procedures, where the scores are allocated after all the competitors have performed. Both methods show a similar linear order effect giving competitors at the end of the contest higher scores.

In other words the running order still impacts on the results from a professional jury, even with training and strict criteria.

How To Make A Fairer Eurovision Jury

The jury system for the Eurovision Song Contest, while technically detailed, lacks the strong direction and rigour that other judged contests have. But with a few changes backed by academic research, the confidence in a jury vote could be improved.

Guidance and training

Firstly the EBU need to supply more guidance to the jurors so they can make informed decisions. Rank the songs from top to bottom, with four criteria to consider, leaves a lot of room for interpretation. There’s an argument that being this hands-off allows jurors to use their own skill and judgement to rank the songs, but there needs to be a distinction made between telling the jurors what  to vote for (which we don’t want), and what the jurors should be looking for (which is currently left as an exercise to the individual).

The EBU focus heavily on the technical process, and that same focus and detail needs to be employed to the psychological process of voting to reduce the human bias that currently unbalances the system.

Reducing the impact of a lone voice

The current juries consist of five jury members. That’s a huge amount of voting power handed to one person, and the impression is they hold 10% of the overall mark. In fact they have a much larger impact. The new scoring system, where jurors are asked to rank all the songs, will reward songs that have a consensus opinion. A song that divides opinion, with some jurors voting it highly, with others ranking it low, will have a lower score a song which is ranked ‘middle of the road’ by everyone. A single low ranking in a jury of five members can have a significant impact on the final jury ranking.

I would increase the number of jurors to at least thirteen jurors, and preferably out to the same number of songs that are performing in the Grand Final. The more members on a jury, the less damage an outlying opinion can have. And if you have a larger number of jurors, the third recommendation becomes even more potent.

Give every juror a different running order

While it is desirable to guide each juror in an attempt to minimise bias, it is also possible to create a level playing field with the running order if you have a large enough pool of jurors. There is no need or requirement for the jury to be entertained, their job is to judge the songs. So why do they need to watch the regular show as the audience do, why do they need to watch the postcards, and why should they watch the same order of songs as the public?

If you have twenty six jurors, every song can have the opportunity to ‘open’ Eurovision, every song gets a shot at closing the Contest, and every song gets a go at being second in the order.

Twenty six is rather unwieldy though, so you could get the same effect with thirteen jurors, and an equal spread of the starting positions for the Grand Final – juror one starts at song one, juror two starts at song three, juror three starts at song five, and so on. For the semi finals the ‘extra’ jurors after this process would also be equally shared out, starting their viewing at every third or fourth song.

Using this method, every song has an appearance at the weaker and stronger points of the running order, and the effects occurring from the running order bias are shared out equally between all competitors.

Which song would be the ‘first’ song for each juror in an offset system?

This would mean that some jurors would not be able to start the judging process until later in the proceedings, which would mean the deadline for submitting a ranking to Digame would need to be extended, but remember this happens a full day before the live show that the public vote on, so there is clearly enough time to accommodate this. And the benefits to the integrity of the jury vote far outweigh the issues in delaying Digame’s collation of the jury result by an hour or two.

Give the jurors more time

Finally, jurors should have enough time to rewatch any song to refresh their memory, to compare two songs next to each other, and not have to rely on their memory to compare a song near the end of their running order to one that they listened to over ninety minutes ago to minimise any remaining serial order bias. Currently the final rankings need to be submitted via Digame to the EBU by 2300 CET. In previous years the deadline has been 0900 the following day, and it would make sense return to this schedule.

Final Thoughts

The 50/50 split between the public and the jury was introduced in 2009 following disquiet at the diaspora influence voting patterns under the 100% tele voting system.. The current system goes some way to reducing national and cultural bias, but introduces unnecessary human bias into the result. Thankfully there is a lot of scope to improve the process. Four small changes should be considered for the 2014 Song Contest, and beyond.

• The EBU should introduce training and evaluations for jurors to reduce bias, and provide  clearly defined judging criteria and instructions.
• The number of jury members should be increased, potentially to thirteen members (half the number of songs in the final).
• Jury members should watch a personalised and edited version of the Contest to reduce the overall impact of the running order.
• The deadline should be extended to allow jury members an opportunity to re-watch performances.

The Eurovision Song Contest is the biggest competitive singing contests in the world. Yes it is a huge light entertainment show that can rival the Superbowl half time show in spectacle and viewership, but it is still an international contest, one of the largest in the world. With that status comes a responsibility to create a level playing field for every contestant and to have each song judged in a fair and equitable manner. With some small changes to the jury system procedures, we would have a fairer Contest.

And that is, surely, an ideal to strive for.