The Continuing Mystery Of San Marino’s Complicated Televote Written by Ellie Chalkley on May 28, 2019 | 2 Comments

Do you trust the result of the Eurovision Song Contest in 2019?  Why has the community’s confidence been damaged since the credits rolled over ‘Arcade’? Mistakes in the recalculation of Belarussian jury vote have been acknowledged, other issues are left unanswered, and questions remain over the San Marino televote. For the long term health of the Song Contest, it is time for the EBU to publicly show how the process works. Security by obscurity is a fantasy.

Since the curtain came down on Tel Aviv, the Eurofandom’s elite squad of spreadsheet enthusiasts and statisticians have been hard at work highlighting errors and raising questions around the scoring. In this article we take a deeper view at how the system for calculating backup votes works.

• We finally found out how (one of) the systems for creating emergency backup Eurovision votes works, and it has had some impact on this year’s scores.
• The system used to create the backup Belarussian jury vote is not the same as the one used to simulate the San Marino televote.
• Why would there be two different systems for simulating emergency backup votes?
• In a year where the winner was decided by 26 points and qualification was decided by single point margins, transparency concerning scoring is hugely important

Sorting The Belarussian Vote

Portuguese Eurovision fan @Euro_Bruno tracked down the error in the aggregated score that replaced the dismissed Belarus jury, resulting in a correction of the scores that affected placings. Several other community members have done work showing that some jurors likely ranked the entries backwards, and showed the comparative margins of victory for the Top 50 songs of the decade.

@Euro_Bruno investigates the Belarus vote (Twitter)

Thanks to Bruno’s analysis, we now have a very good working model how an aggregated ‘emergency’ jury score is calculated. It makes sense to see whether this model can be used to calculate the aggregated televote score that is attributed to San Marino.

It would make organisational sense for there only to be one system for generating emergency scores – you could use it for the case you’re expecting every year (San Marino, which has no independent phone system); cases that crop up from time to time (when the televote in a country is too small to constitute a representative sample); and emergencies (when a jury fails to act with the required discretion and must be dismissed). Then you could be sure that you’re applying the same maths in the same way to anyone who needed an emergency score. Applying the same method to everyone makes the system auditable, gives the appearance of fairness and avoids the need to scramble together a last minute solution.

When You Need To Create A Televote

The San Marino televote has been a long-running conundrum. Back in 2016 Ben Robertson and I investigated the aggregated score that replaces San Marino’s televote. We had been told that the score was composed of an aggregation of other real televote scores. I used a data fitting technique called non-negative least squares to try and reverse calculate which televotes were part of the aggregation.

The answer was inconclusive. No matter what combinations of televote rankings we tried, it was not possible to verify how the televote was created. It is impossible to know if the San Marino televote is correct or if there are mistakes similar to the Belarussian jury vote.

The Belarus Jury incident gave us insight into one EBU aggregation method, thanks to the methodology worked out by @Euro_Bruno. This method combines countries in the same semi-final draw allocation pot as the country you’re calculating for. The individual jury rankings are combined to produce an average value for each country. The average values are then ranked to produce the aggregated ranking.

Let’s look at Semi Final 1, where San Marino had 58 televote points to distribute and the margin between qualification and missing out on the Grand Final was 2 points.

The countries from San Marino’s pot competing in Semi 1 were Greece and Cyprus. Let’s look at what the rankings look like if you just use Greece and Cyprus. The San Marino 12 and 10 points were awarded to Greece and Cyprus, which works out just fine as Greece and Cyprus both came highest in each other’s televote. But looking further down the rankings, it’s clearly more complicated than that. Georgia was ranked 2nd in both the Cypriot and Greek televotes, but was ranked only 10th by the San Marino process. Poland should be ranked 13th if the aggregator just includes Greece + Cyprus, but instead they were ranked 6th and obtained 5 televote points.

San Marino’s 2019 Televote – Greek and Cypriot combination (image: Ellie Chalkley)

So we are clearly looking for another constituent for the aggregation. In order to produce the San Marino ranking, that extra ingredient in the ranking must look like this:

• Greece is ranked more highly than Cyprus, but not necessarily 1st or 2nd.
• Georgia has an extremely low ranking, possibly last.
• Poland has a good ranking, possibly top 3.
• Iceland and Estonia both have good rankings, likely top 5.

It would make sense to combine one of the Big 5 + Host nations into the simulated televotes. For 2019 Semi 1, this is Spain, France or Israel. Looking at the table below, let’s check these off:

• Israel – 1: False. 2: False. 3: False. 4: False.
• Spain – 1: True. 2: True. 3: False. 4: True.
• France – 1: True. 2: False. 3: True. 4: False

San Marino’s 2019 Televote – French/Spanish/Israeli potential (image: Ellie Chalkley)

Next, we look through the rest of the semi final for countries which fit these criteria.

…It appears that no single country fits all four of our criteria. The San Marino televote does not appear to be an aggregate of Greece + Cyprus or Greece + Cyprus plus any one of the other countries in Semi Final 1. Let’s just run the numbers including Finland and Belgium just to make absolutely sure.

San Marino’s 2019 Televote – Poland and Georgia High/Low combinations (image: Ellie Chalkley)

No, it’s not Finland

San Marino’s 2019 Televote – Cyrpus and Greece with Finland (image: Ellie Chalkley)

It is also very much not Belgium either.

San Marino’s 2019 Televote – Cyrpus and Greece with Belgium (image: Ellie Chalkley)

Wait though, we don’t seem to be getting closer. Can we construct a single set of rankings that would give us the correct result? We know roughly what values we’re aiming at, so we can try and solve it like a logic puzzle.

The two biggest clues are Poland and Georgia. To see if it’s possible to get Poland ranked 6th and Georgia ranked 10th, I’ve started by ranking Poland 1st and Georgia 16th. This results in average ranks of 8.67 for Poland and 6.67 for Georgia, which cannot result in Poland ranking higher than Georgia, whatever the other ranks from a single real televote are.

What is clear is that the system used to generate the San Marinese televote is different to the system used to generate the Belarussian televote.

I asked an EBU spokesperson to clarify how San Marino’s televote was calculated [Updated 29 May –  Following publication of this article, the EBU provided a more detailed statement]:

The calculation for the San Marino televote is based on the standardized calls and SMS of countries with the same voting pattern. The formula for the calculation is confidential and agreed between the ESC Reference Group, the EBU, and its voting partners digame to generate the fairest substitute result. This process was audited by two different and independent auditors as previously stated. The EBU can therefore confirm that the televoting result in its entirety has been verified, rechecked, and is correct, including the calculation of the San Marino result.

A Voting Frustration

The San Marino televote remains a mystery, and this remains a problem.

For the Eurovision Song Contest to remain as successful as it is, viewers, delegations, broadcasters, and performers must have confidence in the results of the Song Contest. If you cannot trust the scoreboard, why enter the Contest. If you cannot trust the scores why watch the Contest?

Without a transparent explanation of how the aggregated votes used in the Song Contest are generated, there will always be uncertainty over the veracity of the scoring. Following the Contest in Tel Aviv the ‘generated’ Belarussian jury vote was proven to be wrong, investigations are continuing over the Italian televote discrepancy, and various individual jury rankings are being questioned.

When qualifications are won and lost over a single point, it is important for everyone involved to be able to track where every single point comes from – a sign-off by an auditor is no longer enough when the auditor has been proven to miss a glaring error in the 2019 Grand Final. Giving everyone the data to work out where these aggregated scores come from (at least after the event), would enable those in competition and the extremely engaged audience to have confidence in the results.

Confirming the San Marino televote process should be one of the EBU’s first steps in rebuilding the loss of trust in the results of the Eurovision Song Contest.