Support ESC Insight on Patreon

Voting Insight: Ten Years Of Different Voting Systems Written by on May 19, 2014 | 5 Comments

While most of the arena was still in euphoria from Austria’s victory in the Eurovision Song Contest 2014, the European Broadcasting Union (EBU) were true to their word and posted the results from this year’s contest as quickly as possible.  

For the first time ever, the individual ranks of each jury member are available as well as the ranking of the televote from first place to last.  This means that it is now possible to run detailed models into the voting of the Contest.  In the first part of a series we’re calling ‘Voting Insight’, Ben Robertson goes through the results to assess what would have been different had the different voting systems of Eurovision over the last ten years been in place.

Why?

Having all of this data from the EBU means that we can see how Eurovision has changed through each of the evolutions, from televoting only to our current system of ranked jury and televote scores.  Firstly let’s make it clear that Austria this year was a clear runaway victory in whatever system we use, and the margin of that victory changes only slightly with each alteration.

However, our full analysis below brings up surprising twists.  Different songs would qualify for the Grand Final on Saturday night depending on the model used – in different calculations Portugal, Macedonia, and Lithuania would have made it, and San Marino, Slovenia and Malta may be on early flights home.

Also, we are able to assess how different the jury and televote are.  This is the first year where full ranked similarities and differences can be spotted between each of the results of the two seperate groups, and included here is statistically significant evidence that our jury votes are more spread out and unpredictable than our ‘unreliable’ televoters.

We do this because it helps us understand how our Eurovision Song Contest works in the way it does.

Conchita Wurst, Ready To Rise (picture: Ewan Spence)

Conchita Wurst, Ready To Rise (picture: Ewan Spence)

What Is Different In The Voting Systems

Since the advent of Semi Finals in the Eurovision Song Contest we have seen various systems used for keeping score.  Classically, all the countries used televoting to announce points from the 12 points down to 1 point as has been traditional in the contest for many years.

In 2008 and 2009, the top 9 songs in each Semi-Final qualified through the televote, with one place left available for a jury wildcard, which was the highest ranked song from the juries scores overall combined score  that had not yet qualified.

In 2010 the voting system changed so that each jury and televote would create scores from 12 points down to 1 point, which would then be merged together to make a total 12 to 1 score from that country.

Our current combination system (which has been in place since 2013), has seen each jury and televote rank each song from first place to last place.  These rankings are then combined and the best combination scores 12 points, the next 10 and so forth down to 10th place which gives 1 point from that country.

We take the data provided by the EBU in order to determine how the Eurovision scoreboard would have looked changed  in the different voting systems.

Our Methodology And Our Assumptions

Using the data provided by the EBU, we have been able to extrapolate quite easily the televoting scores from each country.  A song ranked in 1st place with a televote would be given 12 points, second place would receive 10 points and so on.

The calculations of the jury scores for the old system are not quite as easy however.  EBU have calculated average jury rankings for each country, but these can not be used due to the discrepancy that exists with the new ranking system.  In previous years with juries each jury member ranked only their top 10 songs in each show.  Therefore any negative drag effects from low ratings below 10th from individual jury members in the modern day voting system are removed from our calculations, but are included in the average ranks provided by the EBU.

To complete the calculation as fairly as possible, we have made the assumption that the rating from first to last for each country in the jury scores can be used to work out what each individual jury member would give for a top 10.  For example, a first place score from a song would give it 12 points from that juror, a second place score would give 10 points.  For each of the five jury members these are then combined to form the total 1 to 12 from the jury.  This can then be, if appropriate, combined with the televoting scores in our data.  We also refer back to the official rules document from Eurovision 2012 to help in the case of tie-breaks occurring.  In particular we have had to split jury tie-breaks often using this segment of the rule book

“Should two or more songs receive exactly the same number of points from a National Jury, the order shall be ascertained by a show of hands by the members of the National Jury.”

So when two or more songs have tied for points under the old system, we have manually assessed from their ratings of all of the songs the likely voting preferences from each jury member in this scenario.

For each Semi Final and Final below we have calculated.

  • The results in the case that only televoting was used
  • The results in the case that only televoting was used, and countries that did not meet the threshold used their back-up jury voting from 12 points to 1 point (like methods from 2004 to 2007)
  • The results in the case that only jury voting was used using the 12 points to 1 point scale rather than ranking
  • The results in the case of juries and televoting split 50/50 with each giving using 12 points to 1 point (the system from 2010 to 2012)
  • The average (mean) and the standard deviation (how spread out the results in the full data set are on average away from the mean) of each entry

Semi Final One

If we take the results of Semi Final One on a pure televote, there are derivations from the results announced on the Tuesday night. The Netherlands is still a comfortable winner, 27 points ahead of Hungary, but the big mover is Portugal, moving up to 6th place overall.  San Marino is a comfortable qualifier in 8th place followed by Latvia as our new qualifier in 10th position, 7 points ahead of Belgium.  Montenegro and Azerbaijan would have missed out in this case, most notable would have been Azerbaijan’s televote score of only 15 coming from three countries (8 from Russia, 6 from Ukraine and 1 from The Netherlands).

Televote Semi Final One (including back-up juries)

The Netherlands 148 1st
Hungary 126 2nd
Armenia 123 3rd
Sweden 122 4th
Ukraine 118 5th
Russia 73 6th
Portugal 71 7th
San Marino 53 8th
Iceland 49 9th
Montenegro 42 10th
Azerbaijan 42 11th
Latvia 41 12th
Belgium 40 13th
Albania 23 14th
Moldova 18 15th
Estonia 13 16th

However, in reality a full televote would not have been possible in Semi Final One, as a total of four countries had to resort to their back up juries due to the lack of televotes cast.  These were San Marino, Albania, Montenegro and Moldova.  Including their jury results into the list of televoting results would give us the actual figures as if we were following the Eurovision voting system for semi finals from 2004 to 2007.  The Netherlands is again still a clear winner, but the success stories in this system go to Azerbaijan and to Montenegro, who each received 27 and 16 points respectively from these four jury groups.  This is enough for both to springboard over Latvia (which received 2 points from these four juries) and to tie on equal 10th place.  Montenegro received one 12 points (from Albania’s jury) compared to Azerbaijan’s zero, so would qualify under the rules of the contest at the time (and they also received points from more countries, so they would qualify under the more recent rule revisions as well).  Spare a thought here for Estonia too, which would be moved to last place in this semi final if the EBU decided to use a 100% televoting system for this year’s contest.

Jury Vote Semi Final One

The Netherlands 136 1st
Hungary 127 2nd
Sweden 120 3rd
Armenia 109 4th
Ukraine 91 5th
Azerbaijan 88 6th
Montenegro 68 7th
Iceland 64 8th
Estonia 62 9th
Albania 62 9th
Russia 57 11th
Latvia 31 12th
Belgium 27 13th
Moldova 25 14th
San Marino 23 15th
Portugal 12 16th

The Jury Vote in Semi Final One still shows a very strong top 5 which is in agreement with the televoting top 5.  The Netherlands is a clear winner, but Hungary is closer in just 13 points behind.  The clear losers in a jury vote are Portugal, which moves to last place in the semi final ratings, and San Marino which is second last. Russia would miss out on the final in a 100% jury vote, with Estonia taking the 10th and final place five points clear.  Albania takes the second spot vacated from San Marino’s drop in points from the juries.  If the jury vote as here was used for deciding the 10th and final qualifier from the semi finals, as in 2008 and 2009 when the top 9 progressed to the final followed by the jury wildcard, Montenegro would lose out on its place to Azerbaijan, which was a comfortable 6th with jury voting.

Combined 50/50 voting method from Eurovision 2010-2012 for Semi Final One

The Netherlands 148 1st
Hungary 135 2nd
Armenia 127 3rd
Sweden 122 4th
Ukraine 115 5th
Azerbaijan 65 6th
Russia 60 7th
Iceland 58 8th
Montenegro 55 9th
Portugal 39 10th
Latvia 38 11th
Estonia 36 12th
Belgium 35 13th
San Marino 33 14th
Albania 28 15th
Moldova 13 16th

When we combine together the jury and televote scores for each country, as was done for semi finals between 2010 and 2012, we again find that The Netherlands is a clear winner with 148 points while Azerbaijan, Iceland and Montenegro cement themselves as secure qualifiers. Qualification at the bottom end though is even tighter, with just 6 points separating songs from 10th to 14th.  With virtue to a 7 from France (as opposed to the 5 it received in the current system), Portugal takes the final spot to qualify with 39 points.  This is one point above Latvia, who are themselves two points clear of Estonia on 36.  Belgium collected 35 points and San Marino only 33 and therefore 4 places off qualifying for the Grand Final.

Semi Final One Mean and Standard Deviation Results (Televote)

  Mean Ranking of Mean Standard Deviation
The Netherlands 2.79 1st 2.26
Hungary 3.71 2nd 1.38
Sweden 4.21 3rd 1.76
Ukraine 4.57 4th 2.68
Armenia 5.29 5th 4.66
Portugal 6.71 6th 3.52
Russia 7.36 7th 3.97
San Marino 8 8th 2.62
Iceland 8.43 9th 3.63
Latvia 8.71 10th 3.22
Belgium 9.36 11th 3.15
Montenegro 9.67 12th 2.47
Estonia 11.36 13th 3.08
Azerbaijan 12.14 14th 3.76
Albania 12.73 15th 2.19
Moldova 13.8 16th 2.34

Semi Final One Mean and Standard Deviation Results (Jury Vote)

  Mean Ranking of Mean Standard Deviation
The Netherlands 4.44 1st 3.63
Sweden 4.94 2nd 4.09
Hungary 5.06 3rd 4.19
Armenia 6.17 4th 4.45
Ukraine 6.17 4th 2.92
Azerbaijan 6.22 6th 3.61
Montenegro 7.28 7th 3.58
Albania 7.83 8th 3.11
Iceland 8.28 9th 4.51
Estonia 8.33 10th 4.17
Russia 8.61 11th 4.1
Latvia 10.89 12th 3.55
Belgium 10.94 13th 3.35
San Marino 10.94 13th 3.7
Moldova 11.44 15th 3.62
Portugal 11.78 16th 2.96

There are no great contrasts when we present the average scores in Semi Final One as averages, which is how the EBU chose to present the split results last year.  We can though note that our winner The Netherlands was much more broadly supported by televoters than Emmelie De Forest was last year, her semi final score from televotes and jury were 3.33 and 3.58 respectively, whereas The Netherlands peaked with televoters at a average ranking of 2.79.

Here is Aram MP3... alone (picture: Ewan Spence)

Aram MP3… he’s alone here (picture: Ewan Spence)

The standard deviations are a measure of how much variance there is either side of this average score across all of the voting countries.  This will naturally follow lower numbers for songs that are universally ranked highly and lowly.  Nevertheless the high score here of Armenia is notable, showing how Armenia got high points from some countries and low points from others.  Part of this can be seen as the drag effect due to the last place from Azerbaijan’s televote for Aram MP3, however even removing this from the data still gives a score of 3.73.  This would be the third highest spread in the semi final, with only Russia and Azerbaijan splitting opinions of viewers across the continent as much.  With the jury groups it proved to be Iceland who divided their professional opinion the most.

Semi Final Two

In Semi Final Two we are able again to perform the same analysis.  Looking at the qualifying results if they were based purely on televoting, we can see that Austria would still have been a clear winner.  Austria received a minimum of 8 points from all televotes in each semi final, except for only receiving 5 points from Lithuania.  The top 5 in the televotes in this semi final received points from all the other competing nations if we look at televotes alone.  The one change to the qualifiers in this system is that Malta drastically drops out. Malta only received 16 points from televoting, the highest being 4 points from the United Kingdom.  Georgia is still last, but the 15 points they received include an 8 from Lithuania and a 7 from Greece.

Televote Semi Final Two (including back-up juries)

Austria 165 1st
Romania 134 2nd
Poland 116 3rd
Switzerland 98 4th
Greece 91 5th
Belarus 86 6th
Finland 60 7th
Norway 55 8th
Ireland 47 9th
Lithuania 44 10th
Slovenia 39 11th
Malta 36 12th
Macedonia 30 13th
Israel 26 14th
Georgia 15 15th

Again however, this does not tell the full pictures because two countries, Georgia and Macedonia, were unable to meet the threshold required for a full televote.  Adding their back-up juries to the scores shows that Lithuania, thanks to 4th place from the Georgian jury, leapfrogs Slovenia for the final place and would have qualified under 2004 to 2007 rules therefore.  Malta received 20 points from these two juries lifting it away from Georgia in last but not enough to qualify. The country that takes Malta’s place in the Eurovision Final, under 100% televoting, would have been Ireland.  Slovenia ends up in 10th place, just one point above Lithuania. Slovenia received points from 9 countries in televoting in contrast to just 6 for Lithuania.

Jury Vote Semi Final Two

Austria 135 1st
Finland 116 2nd
Malta 112 3rd
Norway 102 5th
Romania 102 4th
FYR Macedonia 72 6th
Belarus 68 7th
Slovenia 53 8th
Switzerland 51 9th
Greece 48 10th
Lithuania 41 11th
Georgia 40 12th
Poland 36 13th
Israel 33 14th
Ireland 33 15th

Assuming that only juries were voting in the competition sees a shock qualifier in that Macedonia would have easily made it through, with only Lithuania and the UK juries not awarding Tijana points, leaving Macedonia comfortably in 6th place.  Interestingly, last place, shared by Ireland and Israel, is given with a comparitively high 33 points each.

Should the Jury Vote have been used this year to determine the final qualifier like in 2008 and 2009, Malta would have been in the Saturday night show.  Ironically in both 2008 and 2009 Macedonia finished 10th in their semi final but did not get the jury support to go through into the Grand Final, this year they would have had jury support to lift themselves above 10th place Lithuania, but not enough to rise above Malta’s 3rd place jury score.

Combined 50/50 voting method from Eurovision 2010-2012 for Semi Final Two

Austria 168 1st
Romania 126 2nd
Finland 103 3rd
Belarus 83 4th
Switzerland 83 5th
Poland 82 6th
Norway 80 7th
Greece 71 8th
Malta 64 9th
Macedonia 42 10th
Slovenia 37 11th
Lithuania 36 12th
Ireland 31 13th
Israel 18 14th
Georgia 18 15th

When these jury scores are combined with the televoting our one change is that Macedonia takes the final spot which is vacated by Slovenia.  Austria flies away with a lowest score of 6 points from Belarus, Macedonia and Germany (in contrast to a 4 Germany gave to Austria in the new system) and qualified by over 100 points.

Semi Final Two Mean and Standard Deviation Results (Televote)

  Mean Ranking of Mean Standard Deviation
Austria 2.13 1st 1.36
Romania 4.07 2nd 2.74
Poland 4.13 3rd 2.92
Switzerland 4.6 4th 1.99
Greece 5.33 5th 2.64
Belarus 6.47 6th 3.4
Finland 7.53 7th 1.46
Norway 8.27 8th 3.13
Slovenia 8.27 8th 2.99
Ireland 8.8 10th 3.19
Israel 9.87 11th 2.26
Lithuania 9.87 11th 4.53
Malta 10.4 13th 1.88
Macedonia 11.5 14th 4.32
Georgia 12.19 15th 3.51

Semi Final Two Mean and Standard Deviation Results (Jury vote)

  Mean Ranking of Mean Standard Deviation
Austria 3.88 1st 3.59
Malta 4.88 2nd 3.46
Finland 5.33 3rd 4.28
Norway 5.53 4th 3.64
Romania 5.65 5th 3.5
Macedonia 7.12 6th 3.24
Belarus 7.65 7th 4.36
Slovenia 7.88 8th 3.55
Switzerland 8.18 9th 3.05
Greece 8.59 10th 3.66
Lithuania 9.29 11th 3.6
Israel 9.71 12th 3.12
Ireland 9.88 13th 3.67
Poland 10.06 14th 3.58
Georgia 10.47 15th 4.11

The average scores for televote suggest Israel was much closer than it actually was.  Israel appears 11th in the televote average list, however because of the low standard deviation of Israel’s points few of these scores got Israel high enough on the scoreboard to make a difference (Israel got ranked 5th from the Finnish televote, but 8 of the countries had Israel marked 11th or 12th).

A similar effect occurred for Malta, where the televotes ranged from a highest of 7th place to a lowest of 13th, and very closely grouped together.  Juries did not have to pull up the song so far to get it to qualify.  In contrast Lithuania in televotes was wildly divergent.  We note that Lithuania received 32 of those 37 points in a televoting only system from Norway, Ireland and the United Kingdom, countries that have a well-known and active Lithuanian diaspora.

TIjana Macedonia 2014 (picture: Ewan Spence)

TIjana is going To The Departures Lounge (picture: Ewan Spence)

On an aside, note too the average score from Conchita Wurst of 2.13 is the highest ever in a semi final televote, higher than Alexander Rybak’s 2.37 from 2009, although admittedly that was a 19 song semi final as opposed to the 15 song semi that Conchita qualified through.  Also note how Conchita, for a song that won the jury vote, has a remarkably high standard deviation from their scores.  This is further evidence of the polarising effect the Austrian entry had on some jurors, and we note the big difference in standard deviation scores between the jury and the televote.

The Grand Final

Our Grand Final has a different make up if we consider it from a jury only and a televote only perspective, as fellow Insight writer John Egan has already published.

When we consider this in the combined format from 2010 to 2012, with the individual jury rankings into consideration, unsurprisingly there is little change at the top of the leaderboard.  Austria though is able to break the 300 point barrier, as Conchita is less hurt by damagingly low jury scores from some countries.  Our top five overall remain unchanged, although Hungary is much more distant from the pack as previously and Poland takes the 6th place spot as again damaging jury scores are removed in the old combination.

Combined 50/50 method from Eurovision 2010-2012 in the Grand Final

Austria 304 1st
The Netherlands 229 2nd
Sweden 201 3rd
Armenia 181 4th
Hungary 118 5th
Poland 110 6th
Russia 104 7th
Ukraine 95 8th
Romania 90 9th
Finland 70 10th
Norway 64 11th
Denmark 59 12th
Switzerland 58 13th
Belarus 54 14th
Azerbaijan 53 15th
Malta 53 16th
Spain 51 17th
Iceland 48 18th
Montenegro 46 19th
Germany 41 20th
United Kingdom 37 21st
Greece 31 22nd
Italy 28 23rd
Slovenia 9 24th
San Marino 9 25th
France 3 26th

Overall the combination of scores has little impact for most entries in the middle of the scoring.  Azerbaijan is a notable positive mover up to mid-table instead of bottom five however due to low televote positioning.  Azerbaijan was indeed last in the televoting averages this year in the 26 song final.

Grand Final Mean and Standard Deviation Results (Televote)

  Mean Ranking of Mean Standard Deviation
Austria 2.76 1st 1.63
The Netherlands 5.5 2nd 4.63
Sweden 6.18 3rd 2.82
Armenia 7.44 4th 6.32
Poland 7.53 5th 5.45
Switzerland 7.97 6th 2.98
Ukraine 9.32 7th 5.38
Russia 9.91 8th 6.73
Hungary 10.29 9th 4.87
Romania 11.44 10th 6.85
Greece 12 11th 4.08
Norway 12.65 12th 4.73
Spain 12.88 13th 4.37
Belarus 13.76 14th 6.62
Denmark 13.79 15th 5.9
Finland 14.68 16th 5.51
Iceland 15.09 17th 6.34
United Kingdom 15.18 18th 4.83
Germany 15.91 19th 5.58
San Marino 16.69 20th 5.37
Malta 18.29 21st 5.55
Italy 19.5 22nd 6.17
Slovenia 19.71 23rd 5.49
France 20.35 24th 3.84
Montenegro 21.15 25th 6.36
Azerbaijan 21.74 26th 5.76

Grand Final Mean and Standard Deviation Results (Jury Vote)

  Mean Ranking of Mean Standard Deviation
The Netherlands 7.06 1st 6.19
Austria 7.14 2nd 7.1
Sweden 7.71 3rd 7.07
Hungary 8.94 4th 6.16
Malta 9.4 5th 5.54
Finland 10.26 6th 6.32
Azerbaijan 10.86 7th 6.37
Armenia 10.97 8th 7.39
Norway 11.03 9th 6.85
Spain 11.23 10th 5.95
Denmark 11.31 11th 5.72
Ukraine 12.49 12th 6.3
United Kingdom 13.17 13th 5.55
Germany 13.51 14th 6.25
Iceland 13.94 15th 7.3
Romania 14.71 16th 6.65
Slovenia 15.03 17th 5.18
Montenegro 15.09 18th 6.71
Russia 15.14 19th 7.53
Switzerland 15.94 20th 6.01
Belarus 16.03 21st 6.9
Greece 16.34 22nd 7.47
Italy 17.29 23rd 6.57
Poland 17.74 24th 6.66
France 18.97 25th 5.19
San Marino 20.4 26th 6.18

In terms of the spread of data we can observe how Switzerland was universally well liked around the whole of Europe, no country put ‘Hunter of Stars’ less than 12th place in the final, which is a feat that even The Netherlands, Armenia, Sweden and Hungary can not claim on their entries.  The Romanian song has the highest spread of televote data, which when looking at where the high points come from we can infer is the country that has had the most benefit in the final from diaspora voting.

The juries overall show a much wider spread of results across all of the competing songs.  We can note from the standard deviations that entries from Greece and Russia divided jury opinion the most.  In Greece’s case they were in the top 10 of 11 different jury groups around Europe, but were ranked from 20th to 26th from 17 of the 37 jury groups.  Russia was in the top 10 in 10 different countries with juries, and ranked from 20th to 26th with 13 countries in the contest.

Also in case you missed it, if we use the method EBU used last year to release the jury/televote split then the negative votes Conchita received from certain juries ensured that The Netherlands is the ‘winner’ of the jury vote, their average of 7.06 just beating Conchita’s 7.10.

Overall Conclusions

Each of the methods investigated shows how different voting systems would have created differences in what was played out on stage in the Eurovision Song Contest 2014.  However it appears in all of our results that Austria would be the winner in every voting system used by the Song Contest over the last ten years, and barring some noted exceptions most songs would have seen insignificant benefit or loss depending on the system used.

Sum of Televote Standard Deviations
(Semi 1/Semi 2/Final)
Sum of Jury Vote Standard Deviations
(Semi 1/Semi 2/Final)
% increase of Standard Deviation in Jury Vote compared to Televote
46.69 59.94 28%
42.31 54.4 29%
134.17 167.13 25%

What this data does note, officially for the first time, is that our juries are less predictable than the televotes.  Comparing the average standard deviations of juries and televotes across Europe in all three shows, we clearly see that the juries are significantly more spread across the continent in their voting compared to viewers at home.  With a sample size of only five jurors in each country, compared to a huge televote, this is of course no big surprise.

What is notable though is the spread of results from jurors for numerous songs is larger than even those countries we hypothesise benefitted most from a huge diaspora.  In the Grand Final only Romania and Slovenia appeared to have bigger spreads from televoters than with jury points.

Stay tuned to ESC Insight for further analysis of the voting statistics.  Episode Two will investigate the impact running order has on the Eurovision Song Contest.

Many thanks go to Alison Wren and the students studying Eurovision at Internationella Engelska Skolan i Täby for their help in data analysis.  Many calculations were made by hand and although they have been checked please forward any irregularities to Ben via [email protected]

About The Author: Ben Robertson

Ben Robertson has attended over twenty Eurovision's, Junior Eurovision's and National Finals for ESC Insight. He uses statistics to explain the Song Contest aims to educate readers about what the Song Contest means to do many different people.

Read more from this author...

You Can Support ESC Insight on Patreon

ESC Insight's Patreon page is now live; click here to see what it's all about, and how you can get involved and directly support our coverage of your Eurovision Song Contest.

Share This Post

Have Your Say

5 responses to “Voting Insight: Ten Years Of Different Voting Systems”

  1. Matt says:

    This is a mind bender if you are not an expert in this field, you do this as a job or you are very good at it as a hobby. Ben, you and everyone who had assisted you should tip their hats. But boy, it’s tough to get your head around this at near midnight in Sydney.

  2. cathal says:

    Okay, the juries have got to go they are constantly voting for malta even when your song is bad. e.g everyone said that malta and switzerland had similar songs… how is switzerland bottom 5 in the jury in the final and malta is 6th? they must be gotten rid off pronto!

  3. João Teles says:

    Malta is one thing but Azerbaijan is far worse, to be honest.

Leave a Reply