Saturday September 6 2008
Britain’s happiest places have been mapped by scientists, according to the BBC: Edinburgh is the most miserable place in the country, and they were overbrimming with technical details on exactly how miserable we are in each area of Britain. The story struck a chord, and was lifted by journalists throughout the nation, as we cheerfully castigated ourselves. “Misera-Poole?” asked the Dorset Echo. “No smiles in Donny,” said Doncaster Today.
From the Bromley Times, through Bexley, Dartford and Gravesham, to the Hampshire Chronicle, everyone was keen to analyse and explain their ranking. “Basingstoke lacks any sense of community or heart,” said Reverend Dr Derek Overfield, industrial chaplain for the area. And so on.
Exactly what kind of data is the good reverend explaining there? The Times had some methodological information. “Researchers at Sheffield and Manchester universities based their findings on more than 5,000 responses from the annual British Household Panel Survey.” According to the BBC it was presented in a lecture at some geographical society. “However,” they said quietly, “the researchers stress that the variations between different places in Britain are not statistically significant.”
Here, nestled away, halfway through their gushing barrage of data and facts, was an unmarked confession: this entire news story was based on nothing more than random variation.
There are many reasons why you might see differences between different areas in your survey data on how miserable people are, and people being differently miserable is only one explanation. There might also be, of course, the play of chance: 5,000 people in 274 areas doesn’t give you many in each town – fewer than 20, in fact – so you might just happen to have picked out more miserable people in Edinburgh, and miss the fact that misery is, in fact, uniformly distributed throughout the country.
This is called sampling error, and it quietly undermines almost every piece of survey data ever covered in any newspaper. Although the phenomenon has spawned a fiendish area of applied maths called “statistics”, the basic principles are best understood with a simple game.
Dr Deming was a charismatic management guru who railed against performance-related pay on the grounds that it arbitrarily rewarded luck.
Working in a theatrical field, he demonstrated his ideas with a simple piece of stagecraft he called The Red Bead Experiment.
Deming would appear at management conferences with a big trough containing thousands of beads which were mostly white, but 20% were red.
Eight volunteers were then invited up on stage from the audience of management drones: three to be managers, and five to be workers.
“Your job,” Deming explained solemnly, “is to make white beads.”
(Fast forward to 4m30s in the video below…)
He then produced a paddle with 50 holes cut into it, which was passed to the each worker in turn. They dipped the paddle into the trough, wiggled it around, and tried to produce as many white beads as they could manage, through this entirely random process.
“Go and show the inspectors,” he would say, sternly. “Only five red beads, well done!”
“14 red beads? I think we need to re-evaluate your skill set.”
Workers were sacked, promoted, retrained and redeployed, to great amusement.
We ignore basic principles like sampling error at our peril, because the illusion of control, which we all carry around for the sake of sanity, is more powerful than we think, and countless workers have had their lives turned to misery for the simple crime of pulling out 15 red beads.
Back in the world of misery, were the journalists blameless, and guilty only of ignorance? For any individual, nobody can tell.
But Dr Dimitris Ballas, the academic who did the research, had this to say: “I tried to explain issues of significance to the journalists who interviewed me. Most did not want to know.”
· Please send your bad science to
Gimpy bothered to email the BBC, and they emailed him back. Bad BBC.
Meanwhile here is the fun you could be having in the Bad Science Forum with fellow nerds, pedants, cranks and geniuses: