What Happened On Friday And Saturday Night?
Belarus’ jury vote reveal during the Eurovision Song Contest’s 2019 Grand Final was going to be one to keep an eye on. Earlier in the week it had been revealed that the EBU had disqualified the Belarussian jury from voting on the Grand Final. This was because members of the jury had revealed publicly the songs they preferred during the first Semi Final. Jury members should keep how they voted secret until after the Grand Final.
In confirming this to Eurovoix, the EBU also confirmed that for the Grand Final the Belarussian jury vote would be calculated by an ‘aggregated result approved by the auditors’:
The Belarussian jury voting has been revealed in an interview contravening Eurovision Song Contest rules. In order to be compliant with the ESC voting regulations, the EBU has taken action and has dismissed the Belarussian jury from the Grand Final on Saturday. An aggregated result approved by the auditors will be used in order to determine to whom the Belarussian votes will be allocated.
It was therefore a shock for many to see Belarus award twelve points during the jury voting on Saturday night to Israel. Not only was this Israel’s only twelve points, they were Israel’s only points of the jury voting. For this to be an aggregated result means somewhere else Israel must have placed high with the juries…which was visibly not the case.
Something clearly went wrong.
The Belarussian ‘Jury’ vote as broadcast on May 18 2019 (YouTube/Eurovision.tv)
Members of the Eurovision community were quick to suggest what the problem was. Taking countries in the same pot as Belarus used to split up similar voting countries for the Semi Finals (namely Armenia, Azerbaijan, Russia and Georgia) and averaging their jury rankings resulted in a jury score that was a perfect match for how Belarus voted.
…if you took their combined last place and flipped it around to be their first place.
Israel was ranked 15th, 21st, 24th and 25th by those four respective countries. This average placment of 21.25 was the lowest scored by any country, but somehow rewarded Israel with 12 points on Saturday night.
A ‘human error’ can be inferred from these numbers; the calculated average was ranked from highest to lowest rather than lowest to highest. The lowest average ranking song (which would have been ‘Chameleon’ from Malta) should have received the 12 points, not Israel.
The EBU have now acknowledged that this is the mistake, and on Wednesday May 22nd revealed a corrected ranking for the Grand Final before updating the Grand Final scoreboard.
How The Results Have Changed
The Belarussian jury vote is only 1/92nd of the total vote in the Eurovision Song Contest, and thankfully there is no controversy over our winner… it’s still The Netherlands, although ‘Arcade’ is now a slightly more impressive winner with a total score of 498 points compared to the 492 points that were revealed on Saturday night.
The changes that exist are further down the table. Sweden becoming the top Scandinavian country – leapfrogging the televote winner Norway into fifth – is the most notable swap. The biggest alteration is in the mid-table. Cyprus and Malta improve two places to 13th and 14th respectively, forcing Slovenia and France down by the same amount.
From a production perspective, the most significant change is that North Macedonia win the jury vote. With 10 points from the Belarussian aggregated score, Sweden’s last minute steal of first place on Saturday night would not have happened. Tamara Todevska would have held the limelight and be leading the Song Contest at the half way mark, and the final head-to-head in the new voting announcement procedure would have been Todevska vs Lawrence.
A Need To Catch Human Error In The Future
That takes care of the facts from Eurovision 2019, but what lessons need to be learned?
The wrong scores were announced on the night, and these could have had a much larger impact. Imagine for a moment a closer result where The Netherlands was awarded the Eurovision Song Contest victory on screen, but the true winner was Italy…
The most troubling aspect has to be the number of checks that missed the error. As the EBU press release takes time to explain, it is Digame who produced the aggregated result; then voting monitor Ernst & Young approved the final results; and at the final level it goes to Executive Supervisor Jon Ola Sand and the EBU team who give the final all clear live on the Saturday night broadcast. Nobody in this chain spotted spotted the upside-down mistake… although when it was announced Eurovision fans online raised digital eyebrows on social media.
Credit must be given to @Euro_Bruno for his analysis of the jury scores which raised the issue in the community which was subsequently picked up by the world’s media.
@Euro_Bruno investigates the Belarus vote (Twitter)
This is the first time an entire jury score has been replicated in this way since 2016, when jury scores and tele votes were split in the Saturday night presentation, but it is not the first mistake in process that has happened.
From individual jurors we have seen evidence of incorrect ranking orders numerous times. Arguably the most notable example is Hilde Heick, a Danish juror from 2016, who intended to place eventual winner Ukraine second last, but instead voted ‘upside down’ and placed Ukraine second. 14 of the 23 point advantage ‘1944’ had over ‘Sound of Silence’ that year was due to that mistake. Other individual jurors, including a number from this year’s Song Contest, have voting patterns that suggest they have voted the wrong way around. Ranking ‘upside down’ has been a problem for years and it remains a possibility that there has been a material impact in this year’s Contest in terms of countries who qualified for the Grand Final:
While the process to determine the Belarussian ‘jury’ vote is different to the ranking from an individual juror, the same mistake has occurred… Eurovision voting rewards those who get the highest points, but jury rankings start from the smallest number. It is easy to see how human error can result in these mistakes. This must be the catalyst to ensure there is increase clarity to the process.
One immediate quick fix would be for each juror and jury chairperson to write down which song is their favourite and input it into the computerised voting form to ensure the ordering has been completed correctly as a ‘check’ on the ranking order
Ensuring Integrity to The Eurovision Song Contest Voting
The Eurovision Song Contest voting is now a battle of two halves, with both jury and televote having equal value. After the EBU’s corrections to the vote, this year we have an unusual situation where neither televote nor jury winner are in the final top 5… and there is nothing wrong with that.
What a split outcome requires though is the need for a demonstrably clean, transparent and fair voting system for the Song Contest. This year we didn’t get that until four days after the live show. The Belarussian jury vote was an embarrassment for the EBU, for the artists involved, and also for the Belarussian broadcaster.
There are also ethics which need to be openly discussed.
Is it ethically correct to use other juries in different countries to ‘simulate’ a Belarussian jury? The countries used to assemble the points may be in a group of similar countries, but their results showcase huge differences. Russia for example received a 1st, 6th and 26th place ranking from those used to construct the vote. Azerbaijan scored a 1st, 2nd and 26th place ranking. Even politically detached Sweden received a 1st place and 9th place…combined with a 15th and 18th place. By accepting that you should take replacement jury points from countries that have a similar voting history, does that mean the result you calculate just perpetuates the perceived issues of political voting?
Certainly one argument to make that would have been to not include a Belarussian jury at all. This has issues for the TV broadcast – as we would therefore not see a Belarussian spokesperson on screen. However making Maria Vasilevich read out a jury score that was quite obviously made up has its own moral dilemma.
The Black Box Of Exponential Weighting
This is not an abstract question for the 2019 presentation. There are a number of issues around voting and point calculations that need to be exposed to public scrutiny to ensure the long term integrity and confidence in the Eurovision Song Contest results.
Firstly, in 2018 the EBU revealed an ‘exponential weight model’ to calculate the points from the 5 jurors. While good in that it mimimises the ability of one juror to destroy the voting power of one song, no set formula for this exponential curve has been formally revealed by the EBU.
ESC Insight’s Ellie Chalkly took a closer look at the system when it was announced last year:
This change to the jury scoring system is a welcome step forwards to a more competitive Contest. It rewards positivity, it diminishes the power of a single juror to negatively impact a song, and it allows strong but divisive songs the opportunity to achieve a respectable jury score ahead of the televote the following night.
Eurovision 2018 scoring chart, Ellie Chalkley
While these changes are welcome, the formula requires a number of constants that have not been revealed. Without these numbers it is impossible for a third party to confirm the model is working as advertised. Essentially there is a ‘black box’ between the juror scores and the jury points awarded. If the Belarussian jury vote can be incorrect, can we have confidence in the rest of the jury voting process?
Returning to the suggestion that a juror has ranked incorrectly during Semi Final One, we cannot be certain of the impact on the qualifiers because we do not know the exact workings of the black box that calculates the jury points.
It is impossible for a broadcaster and member of the public to themselves calculate and ratify that the result is valid alone. We can model what the exponential curve acts like, but this is not the same as a clear and transparent system. One key model of integrity is ensuring that the model is replicable by others.
The San Marino Televote
As noted, it was a simple average of the juries in the same pot as Belarus that resulted in a correct calculation of the Belarussian jury vote. Each year since 2016 we have had to use a similar aggregate method to calculate not a jury, but a televote. Because San Marino uses the Italian telephone system, it is not possible to guarantee a televote is only made up of San Marinese voters. Instead San Marino’s televote system is “an average result of a representative group of televote results of other countries.”
San Marino’s televote construction sounds eerily similar to how Belarus’ jury result has been constructed.
JESC 2013 singer Michele Perniola reading out the points from the San Marino jury in the Eurovision Grand Final of 2014.
Yet whereas the Belarussian jury score appears to have been mathematically simple to construct, it has not been possible to re-construct a San Marinese televote model since 2016. Even their Head of Delegation does not understand how the vote is constructed. Poland’s Tulia were two points away from reaching the Grand Final, and received five points from this constructed San Marino televote. It is easy to imagine if a different group of countries were selected Poland could have scored two more points and qualified for Saturday.
Hiding the process does not protect the integrity of the televote, it simply creates more mystery and raises more questions around the validity of the vote.
Improve The Design By Expecting Mistakes
Mistakes and human error happen, we all accept that. The key is to expect them and design systems that can allow them to be discovered and corrected. Having to change the results after the live show has embarrassed the EBU, Digame and Ernst & Young. Improvements to the design of the jury vote are needed to make sure it delivers the robust result that is expected by the millions of viewers and listeners to the Eurovision Song Contest every year.
However the EBU can not and should not stop now the Belarussian jury vote has been addressed.Clarity and integrity throughout the Song Contest’s voting systems are needed. This year’s incidents can be the catalyst for much needed improvements both behind the scenes and in the package that is presented to the public.
Which is why San Marino should not be in the Contest: no capacity for a televote, no entry. And…San Marino has its own country code and at least three local telcos offering mobile service. So if the local public are invested enough in sending a Sammarese entry they should be invested enough to participate in the televote.
https://en.wikipedia.org/wiki/Telecommunications_in_San_Marino#Mobile_network_operators
Thanks so much for this explanation.
I don’t understand why juror’s still need to rank songs from 1-26 still. If all jurors from a country placed Russia last for example, and the televote 12 went to Russia then the jury vote doesn’t cancel out the televote in this new system. Why not just get each Juror to list their 10 favorite entries and not worry about the 10 entries they placed at the bottom. This would of solved the problem for Hilde Heick from Denmark and others.
Also as for small countries like San Marino – couldn’t they run an SMS vote if they have their own telco? If only a 100 vote were received they are indicative of the small audience certainly. Or an internet vote of favourites could be taken into consideration?
Also the Belarussian jurors may have discussed their favourites but the votes were not actually revealled were they? If the fan media didn’t report breathlessly on everything this may of gone unnoticed. The jury votes could of been scrutinised in anycase. This outcome seems worse than the reason the jury was dismissed and has attracted more media coverage than the initial misdemeanour.
Some people bet thousands on Eurovision. The error around who finished fifth could potentially have major financial implications. Does anyone know how bookmakers reacted to the scoreboard correction?
@Eurojock: I know someone who bet that Sweden would be the best Scandinavian country – theoretically he was right, but the general terms of the bookmaker state that results are used as officially anounced and later changes are not taken into consideration. I expect they all do it this way. Another fan reported somewhere that he already had his (false) winnings transfered to his bank account – it’s probably not practicable to get the money back, which is why later changes to the results are simply ignored. One more reason the EBU should have been more careful. 😉
On the San Marino televote, I don’t understand the argument that San Marino share the Italian telephone system so the votes can’t be distinguished. I fail to understand why San Marino cannot simply have different numbers to call. San Marino broadcaster could pay to setup their own separate numbers to call to vote for each of the counties and then the votes are identified by the number called (unique for San Marino). Working in a call centre, I know it is very easy to identify both numbers who called and the number that they called. The only reason I can imagine this isn’t done is because San Marino do not want to pay for the numbers or the voting numbers would be so small heavy weight is potentially placed by the population that do vote. But we should then be questioning whether they are equipped financially and/or suitably to be taking part at all.
The Belarusian jury vote absolutely should not have been used! The EBU need to move away from these hidden means of calculating votes and making them up as they see fit. It makes absolutely no sense to disqualify a jury but then make up their votes so the jury *seem* to still be there. If a juror was disqualified in an Olympic event, they IOC wouldn’t make up that jurors score as each diver goes. This is also important considering news that the declared winner on the night would be kept as the *winner* even if at a later date votes were identified as being incorrect. So if the Belarusian jury score had actually caused a different winner, we’d not actually see the rightful winner declared at a later date. It’s important where televotes as are identified as being inaccurate or jury votes disqualified, they are just not used and not replaced with numbers calculated by the EBU.
Finally, what IS the point of checking in with Jon Ona Sand for a valid result of he’s not staying that there has been a replacement of a jury result?
Excellent article. I would also encourage the EBU to start reviewing jury scores that have the same country last, when that country is top 10 with nearly every other jury. Georgia placing Russia last with all 5 jury members has nothing to do with the voting criteria; The same applies for Armenia v Azerbaijan. There are other examples that need addressed. A close points finish is about to happen.
1) The system is too complicated. It’s too much to expect jurors to rank 26 songs in order. Even as a Eurovision fanatic for decades, I couldn’t come up with a fully meaningful ranking of that depth. Top 10 yes.
2) EBU / Digame / E&Y / Sand should be ashamed of themselves in allowing an error through which most of us suspected as soon as Israel got 12 points
3) The televote reveal is flawed in any case, spoilable by anyone who can do basic maths. So the pretence that Sweden might get “253” and win was bad enough, but now we know that even the fake suspense was based on a mistake.
4) I can’t wait to see how all this will be explained on the official DVD. Surely they won’t still be pretending that Israel got 12 and Sweden won the jury vote