James Ball sent me the data for the Russian election vote counts this morning and asked me to test whether it deviates from Benford’s law, a test that can give a hint at whether numbers are the product of fraud. Posted below is my analysis, and also a check for last digit preference, which is another method for spotting sneakiness.
You might remember Benford’s Law from this post, it’s a good way to check if data has been faked, and was used on shady Greek economic data submitted to the EU:
Benford’s Law: using stats to bust an entire nation for naughtiness.
Imagine you have the data on, say, the population of every country in the world. Now, take only the “leading digit” from each number: the first number in the number, if you like. For the UK population, which was 61,838,154 in 2009, that leading digit would be “six”. Andorra’s was 85,168, so that’s “eight”. And so on.
If you take all those leading digits, from all the countries, then overall, you might naively expect to see the same number of ones, fours, nines, and so on. But in fact, for naturally occurring data, you get more ones than twos, more twos than threes, and so on, all the way down to nine. This is Benford’s law: the distribution of leading digits follows a logarithmic distribution, so you get a “one” most commonly, appearing as first digit around 30% of the time, and a nine as first digit only 5% of the time.
The next time you’re waiting for a bus, you can think about why this happens (bear in mind what leading digits do when quantities repeatedly double, perhaps) but reality agrees with this theory pretty neatly, and if you go to the website testingbenfordslaw.com you’ll see the proportions of each leading digit from lots of real-world datasets, graphed alongside what Benford’s law predicts they should be, with data from Twitter users’ follower counts to the number of books in different libraries across the US.
Benford’s law doesn’t work perfectly: it only works when you’re examining groups of numbers that span several orders of magnitude, for example, so for age, in years, of the graduate working population, which goes from around 20 to 70, it wouldn’t be much good, but for personal savings, from nothing to millions, it should work fine. And of course, Benford’s law works in other counting systems, so if three-fingered sloths ever develop numeracy, and count in base-6, or maybe base-12, the law would still hold.
I analysed for election fraud using the stats package Stata, beloved of economists and epidemiologists. James has posted graphs from the analysis I did on his blog, where you can also (I assume!) download the data he sent me:
www.guardian.co.uk/news/datablog/2012/mar/05/russia-putin-voter-fraud-statistics
The bottom line is this: the data massively do not conform to Benford’s Law, which would make you think that fraud is at work; but I think we shouldn’t expect the data to conform to Benford’s law, as the numbers don’t span several orders of magnitude. Essentially, this tool won’t work on these numbers. Anyway, here are the graphs and tables. Firstly, a graph showing the leading digit for all vote counts, in blue, against Benford’s distribution in red.
But here’s the distribution of the vote counts, not spanning multiple orders of magnitude:
Here are the tables with the Benford figures, in case you want them, and the second table reports a chi-squared test for whether the real figures deviate from Benford’s distribution (they do, but we could have guessed that from eyeballing the data!):
Lastly, I looked at last digit preference, which is a crude tool for looking at whether someone has made up figures. If they’re all made up by the same human, you might expect them to have some preference for specific digits, as humans are quite bad at generating random numbers. There’s no evidence of digit preference in the data. You might say that’s not surprising, as if there was fraud it would not have been centralised. In any case, I’m only really posting this as an illustration of how you can take data and play around with it.
Lastly, bear in mind that all these tests are only to check if people have made up numbers after the votes have been counted, and won’t detect people stuffing dodgy voting slips into ballot boxes.
For completeness, here’s the Stata code I used. Opensource dorks: in the future, I’m planning to do more public data analyses and detailed walk-throughs, for which I’ll use R, the open source statistics package, but because I did this in a hurry between work this morning I had to use Stata (I know it well so can code without thinking too hard).
Here’s the Stata code:
* housekeeping
clear all
cd /Users/bens/Documents/academia/projectscompleted/benford/
insheet using CandCount.csv, names* eyeball
summ votecount, detail* benford action
benford votecount
firstdigit votecount
firstdigit votecount, by(candidate)
digituse votecount* digit preference
gen lastdigit = mod(votecount,10)
tab lastdigit* nice pics
hist votecount, bin(100) freq
eqprhistogram votecount, bin(100)* install packages if needed
ssc install benford
ssc install firstdigit
ssc install digits
chrisbeeleyimh said,
March 5, 2012 at 1:49 pm
Well, chalk one up to Stata for having short code and decent output.
But yes please on the Open Source R!
digitrev said,
March 5, 2012 at 2:15 pm
Hmm. You’ve shown that the data does not follow the distribution expected by Benford’s law. But what if you ran the same analysis on what one can assume to be non-rigged elections. Say analyse the most recent national level elections in several countries. If none of them follow Benford’s law, then it’s probably safe to assume that election data cannot be expected to follow Benford’s law.
Ben Goldacre said,
March 5, 2012 at 2:22 pm
yeah, the data don’t follow the distribution of benford’s law because the numbers generated by this election don’t span several orders of magnitude. if other elections don’t produce numbers that span several orders of magnitude, then i wouldn’t expect them to conform to benford’s law distribution either.
alexscott292 said,
March 5, 2012 at 2:28 pm
Certainly, UK election data doesn’t match Benford’s law – blog.jgc.org/2009/06/does-benfords-law-apply-to-election.html
But then our elections are “fixed” in the sense that the first past the post system generates an entirely different distribution.
soulinafishbowl said,
March 5, 2012 at 3:35 pm
@digitrev:
thanks to google, it is easy to discover that the idea of matching vote data with Benford’s law is already tested and discussed. Some people at the Voting Technology Project (Caltech/MIT) has even done some simulation tests (vote.caltech.edu/drupal/node/327):
Joseph Deckert, Mikhail Myagkov, Peter C. Ordeshook, Caltech
The Irrelevance of Benford’s Law for Detecting Fraud in Elections
Abstract:
With increasing frequency websites appear to argue that the application of Benford’s Law – a prediction as to the observed frequency of numbers in the first and second digits of official election returns — establishes fraud in this or that election. However, looking at data from Ohio, Massachusetts and Ukraine, as well as data artificially generated by a series of simulations, we argue here that Benford’s Law is essentially useless as a forensic indicator of fraud. Deviations from either the first or second digit version of that law can arise regardless of whether an election is free and fair. In fact, fraud can move data in the direction of satisfying that law and thereby occasion wholly erroneous conclusions.
gawley said,
March 5, 2012 at 3:45 pm
Digitrev, I wrote about Benford’s law recently (wrt to the Greek data) and have a [simple] explanation in the post: marcgawley.com/2011/09/12/you-couldnt-make-it-up/.
Essentially, the Russian data only covers a few orders of magnitude, and even then it’s mostly in the range 0 to 200, so you wouldn’t expect Benford’s Law to hold.
Doire said,
March 5, 2012 at 4:04 pm
What happens if you convert the data to different bases?
I suppose you gain in orders of magnitude but lose sensitivity with fewer points to compare.
For a country wide single-question-election polling stations don’t have to be similar sizes, but might show size chunking based on physical needs, say, 1 box per 300 registered voters. I wonder if there is more information in the mis-match of voters registered and votes posted. It might have a wider range.
jt512 said,
March 5, 2012 at 6:18 pm
No they do not. A chi-squared test of goodness-of-fit with the Benford distribution has a p-value of .23. The data are not inconsistent with the Benford distribution, according to this test.
jt512 said,
March 5, 2012 at 6:21 pm
Ugh. My first sentence, above, was ambiguous. Ignore it. The statistical test shows that the data do not significantly depart from the Benford distribution. Hope that’s clearer.
jt512 said,
March 5, 2012 at 7:09 pm
Never mind. I used the wrong test. I recalculated it correctly, and you’re right. The data are a “massive” departure from Benford.
I’ll shut up now.
bsmith89 said,
March 6, 2012 at 4:46 am
Since the data do not cover several orders of magnitude in base-10, and since Benford’s law should work in other bases, have you considered transforming the data into a lower base and checking again?
pwaller said,
March 6, 2012 at 1:09 pm
This Arxiv paper covers the topic and shows voter fraud in the 2011 Duma election. They can also purportedly detect ballot stuffing. They make a 2D plot of the voter turnout versus the fraction of votes for a winner over all districts. Fair elections typically have a smooth-ish continuous distribution. The differences in these distributions are quite stark, and there are comparisons of the distribution from a few different countries.
arxiv.org/abs/1201.3087
Might be interesting to repeat this, if you have the data handy.
andrewjoy said,
March 7, 2012 at 8:44 am
Interesting. However I would like to point out a small error in the use of the term open source , open source misses the point of free software www.gnu.org/philosophy/open-source-misses-the-point.html
Regression To Norm said,
March 8, 2012 at 4:48 pm
As these statistical tools are not secret would you not expect and reasonably organised electoral fraud to have pre-checked there numbers.
Chris Neville-Smith said,
March 10, 2012 at 2:20 pm
I think a more telling indication of fraud is in Chechnya – you know, that bit of Russia that they flattened not long ago. 99.5% turnout and 99.5% support for Putin.
Yeah, right.
aram said,
March 11, 2012 at 1:18 am
bsmith89: That would only work if the people committing fraud were 3-toed sloths, or for some reason preferred the base you transformed to.
Ben: Can you correct for the limited range of the numbers as follows? Discard the bottom and top 5%, take the log, and compare with a uniform distribution on that interval. That seems preferable to saying that it sometimes work and sometime doesn’t.
gorshkov said,
March 12, 2012 at 6:41 am
Don’t know if I’m missing anything, but I guess the only way to find out is to ask.
You can’t use Benford’s law because it doesn’t cover orders of magnitude. But – if you used a base other than 10 – say, octal, or something even smaller, so that you *did* have the required changes in magnitude – would I be right in thinking that it *would* work then?
Eponymouswhitey said,
March 12, 2012 at 1:19 pm
Is it just me, or don’t the Russian boast some of the finest mathematicians around ? That being the case, I would have thought they were more than capable of producing data undetectable by any of the possible fraudulent detection methodologies ?
Now I know what Perelman has been up to recently….
kylemcinnes said,
March 14, 2012 at 4:20 pm
Like the other commenters, I’m having a hard time understanding how Benford’s law can be invariant of base, while requiring data that spans “many orders of magnitude”.
AndrewKoster said,
March 14, 2012 at 10:13 pm
I’m not a statistician, but isn’t it quite simply that Bedford’s law works for data that you expect to draw from a uniform distribution and people who try to fiddle the books usually do this in a manner that messes up this distribution?
In that case, it only works if the distribution is also messed up in whatever base you’re changing to. When you reduce the base, you also reduce the sample size for testing whether it follows a power law. Instead of having 9 starting numbers to work with, you have less. For instance, if you reduce it to binary, your starting number is ALWAYS a 1 and Bedford’s law is useless.
I expect that in the case of this data, which only barely spans 3 orders of magnitude according to the graph, the base would have to be so low that testing Bedford’s law becomes impossible.
susu.exp said,
March 15, 2012 at 1:19 am
Benfords Law doesn´t work for data drawn from a uniform distribution. It does apply for data sets where the logs of the data are uniformly distributed (over some interval). The more orders of magnitude you span, the less difference the size of the interval makes and for empirical data with more than 5 orders of magnitude you generally recover Benford. Or -to make that a bit more precise – if the logs of the data are uniformly distributed on an interval of integer lenght, then Benfords law holds precisely. If that condition isn´t met then the relative error you make in that assumption is |RND(x)-x|/RND(x), where x is the lenght of the interval and RND rounds to the nearest integer. RND(x) thus corresponds to the number of orders of magnitude the original data span and that error goes down as RND(x) goes up (since the dividend is <1).
But apart from that, your point is valide Andrew: You move to a lower base and will recover benford, but get less data points for comparison (for base 2 the probability for getting a starting 1 is the base two log of 2, which is 1. It´s worth noting that there is one integer that doesn´t start with a 1 though… It´s only almost surely the case that each number starts with a 1).
gabbyjulie said,
March 19, 2012 at 3:05 pm
Hi Ben,
This is totally off topic, but have you seen this yet?
www.latentexistence.me.uk/heavy-handed-police-threaten-nhs-protest/
There’s been absolutely no report of this in the mainstream media, despite the fact that armed police were at this demo.
Filias Cupio said,
March 20, 2012 at 1:27 am
@Eponymouswhitey
Given that you can’t convince Perelman to leave his flat to collect a million dollars, I can’t see how you’d convince him to fix an election for you.
GrahamB said,
March 22, 2012 at 12:33 pm
I haven’t tried it on these data, but with the disputed Ukrainian presidential election a few years ago, I assumed that most rigging tactics will increase apparent turnout, so I plotted declared turnout against the percentage for each candidate. It showed that there was a strong relationship between areas of high turnout and a higher vote share for one candidate. Of course, you could argue that his voters were more motivated to get out and vote. It was just a bit suspicious when turnout at a polling station was around 15% higher than any nearby station. It would be interesting to know if Mr Putin did better where turnout was highest.
neoteny said,
March 24, 2012 at 12:37 pm
@aram: Benford’s law is independent of base. If it’s applicable in base 10 it will be applicable in other bases, too. bsmith89 and gorshkov’s suggestions are perfectly sensible. See terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/
OperationBarbapapa said,
May 2, 2012 at 1:41 pm
How about we statistically analyse the geopolitical leanings of the researchers concerned, and see if there is some kind of correlation to the countries chosen to analyse? Or possibly the susceptibility of the researchers to mass-media opinion-forming, advertising, etc. if we are being slightly more kind.
Let’s face it, elections are normally are said to be “disputed” when other countries don’t get the outcome they wanted, even after vigorously funding the usually hopelessly unpopular opposition (hint-that’s why they’ŗe the opposition). Then you call their elected leader a “strongman” until the next election comes around, What does that even mean, visions of handlebar moustaches, stripey victorian gym outfits and moulded black iron dumbells? I notice this didn’t ever happen to Sarkozy 😉
annoporci said,
March 21, 2013 at 7:29 am
How about taking a sample of election results with known fraud, if available, and analyzing that to see what sort of distribution emerges, and see if that can be used. We know elections in many countries in Africa are rigged. Chechnya is another example that comes to mind. Not sure if the data is available though.
Patrick Toche.