Ben Goldacre
The Guardian
Saturday July 5 2008
Anyone would think the cold war was still on, with all this top secret scientific data that journalists constantly seem to be writing about. In last week’s column, as you will remember, we saw the Sunday Express front page claiming that a scientist and government adviser called Dr Coghill had performed scientific research, and found that the Bridgend suicide cases all lived closer to a mobile phone mast than average: this was an issue of great public health significance, but when I contacted the researcher, he wasn’t a doctor, he wasn’t really a government adviser, he couldn’t tell me what he meant by “average”, and he had, in a twist of almost incomprehensible ridiculousness, “lost” the data.
This week we have the same thing, from the insurance company Esure, and their agents Mischief PR. They’ve done a very good job of getting publicity for some survey figures. “Fornightly bin collections spark rat plague” was the headline in the Express this time. According to the Daily Mail “the number of pests plaguing homeowners has gone up by more than a fifth in a mere three years.” What caused it? “The rise in unwanted visitors coincides rather neatly with the introduction of fortnightly rubbish collections in half the country.” They all quote reams of detailed data. “Household reports of wasps have risen by 39 per cent, squirrels by 23 per cent, mice by 17 per cent and rats by 12 per cent,” and so on. Similar figures were reported in the Telegraph, on GMTV, and in the Daily Mirror.
I contacted Esure and Mischief to ask about the figures. It’s fairly standard practice to make your data publicly available on this kind of survey, as far as I know. Esure refused to give me the numbers. Have they lost it, perhaps, like Dr Coghill? Apparently not. They do not send out raw data (“this is company policy” is an eerily familiar phrase from insurers). They are, however, happy to answer individual questions.
This presents us with an interesting challenge: can you interrogate a statistical dataset through a letterbox, in a chat with a PR person? It might take a while.
Starting with the easy stuff: you will already have noticed that all the figures quoted are what statisticians would call “relative risk increases“: there is a “39% increase”, but 39% more than what? A very rare thing? A very common thing? The figures for “absolute risk increases” would be nice, please, Esure, and I’d be happy to calculate them myself, from your top secret data.
Then there are the basics of what information was gathered: Esure are claiming a change over time, but there’s no indication of what was measured in the past, when, and how it is being compared with current data. Or did they rely on recall, which is human and flawed, and prone to substantial biases (known in the trade as recall bias)? “Ooh yes.” “Really?” “Mmmm now you come to mention it since they changed the bins I do think I’ve definitely seen more rats…” There are the basic data analysis issues – like “did they only ask people whose rubbish collection patterns have changed about vermin patterns changing, or did they ask everyone?”
But then there are the fascinating statistical issues. Did they just cherry pick the biggest figures? Did they do a “correction for multiple comparisons“? After all, if you measure a huge number of different things, some of them are bound to change, or be different, or appear to be statistically significant, simply by chance: because if you toss a coin enough times, you’ll perfectly easily get five heads in a row, simply by chance. In fact, speaking of statistical significance, what tests did Esure and Mischief do to make sure that their results weren’t simply due to the play of chance? A chi-squared test, perhaps, and if so, on how many subjects? Did their data fulfil the assumptions of the chi-squared test? Was there other numerical information? What was the variance in the data? Where are the p-values? And so on.
I’m very happy to analyse a dataset by playing twenty questions through a letterbox with a PR person, but it might well require yes/no answers to several hundred thousand questions until we have the actual numbers. I don’t know how many, because I can’t even know what they’ve collected.
This research has received blanket media coverage, it’s clearly the subject of great public concern, it speaks to us of vitally important issues of public health, and once again the data is hidden from the public, preventing anyone from analysing its contents and significance. Nobody on the Mirror, the Mail, the Telegraph or the Express seems bothered by this. Clearly it’s you and I who are wrong.
mjs said,
July 5, 2008 at 2:53 am
Awwww, ratties!
(There were so many pictures I couldn’t resist.)
What if they had surveyed pest control companies and compared their business earnings over time? Or the number of calls for each pest?
At least the data would be cleaner than a phone survey of apartment dwellers and homeowners, in the sense that there would be perhaps numbers.
“Company X made $— in 2006, $— in 2007, and $–,— in 2008 from wasp removal? Call the newspapers!”
Though, as stated in the article, I’m not quite sure why Esure would be interested in this sort of thing. Is it a parameter for determining home insurance premiums?
All quite odd.
Moganero said,
July 5, 2008 at 7:22 am
If, and it’s obviously a big if, the fortnightly collections do coincide with an increase in the number of rats, then it is much more likely to be people putting their bags out before the day of collection or not keeping them in a bin until collection day.
And I bet over the same period there has been an increase in the fast-food market and the casual dropping of unfinished junk-food in the streets. That certainly seems to have increased around where I live.
Maybe the increase is because there are more dirty bastards around than previously, but I’m not sure how that could be researched – maybe we could get an insurance company to do it!
phayes said,
July 5, 2008 at 8:21 am
Hmm… looks like a rat king forming on the left hand side of that bowl of milk. Very bad news – especially if it gains access to the ‘net:
en.wikipedia.org/wiki/Rat_king
Dr* T said,
July 5, 2008 at 8:23 am
It’s such a shame, as the Daily Mail is normally so good.
Dangerous said,
July 5, 2008 at 9:09 am
Calm down Dear!!!
It seems like esure’s research is a load of rat sh!t…
I agree with Moganero. Put your rubbish in a bin.
Only really clever and strong rats can climb a wheely bin, open it, eat stuff and get back out… (Not that I’ve done any research)
Oh yeah, eat food don’t throw it out.
Maybe people are blaming rats for stealing electronics from their houses and esure are having to pay out???
icthyophile said,
July 5, 2008 at 9:36 am
It almost seems churlish to mention this, but do you think it’s about time to fix the typo that appears on every post for people who aren’t logged in? It has a certain charm, and I’m sure people don’t mind being told that they need to “bee” logged in to post a comment – but sooner or later I expect someone is going to tell you that a man who can’t use a spellchecker can’t be taken seriously as an evangelist for good science…
bpdlr said,
July 5, 2008 at 10:29 am
Data is cheap to store and publish online. Maybe everyone could petition their MP for a bill to require all data for surveys that are reported in the press (and that supposedly are in the “public interest”) to be made available for scrutiny online? A sort of “open source surveys” movement. That would get rid of a lot of crap…
Paul Crowley said,
July 5, 2008 at 10:31 am
Do the Press Complaints Commission care about this sort of thing? Or any other such body? Could we somehow make them care?
I’ve read the “transparent nonsense pushed by people with an agenda and easily punctured by the most basic questions presented as serious science by lazy journalists after cheap story that reinforces their prejudices and those of their readers” story quite a few times now, and it’s starting to feel like all of those receptive to understanding that this is how it works already know. Is there anything else we can do to hold these people accountable for publishing this nonsense? Can we have them very publically dinged for ignoring the guidelines? Or if not, can we get the guidelines tightened until we can?
Diversity said,
July 5, 2008 at 6:37 pm
I had a look at the Esure website. Their PR people do not list this press release; but they do note that they are not responsible for “paid-for marketing activity.” I deduce that the press reliese via Mischief PR was paid-for marketing, and is therefore reportable to the Advertising Standards and/or similar PR Industry people. The Press Complaints Commission come in if the papers have been presenting un-verified paid marketing material as news. What is more, since the Government are in favour of fortnightly bin collections, officials can look after the reporting.
Incidentally, are Mischief PR by any chance staffed by cockroaches? Or is there some other reason why those quick multiplying rubbish-eaters got left out of this imaginative story?
JQH said,
July 5, 2008 at 10:18 pm
Maybe in-jokes should be avoided hairnet. They can upset some prickly characters.
pv said,
July 5, 2008 at 10:24 pm
‘speshly on teh Internets. 🙂
Robert Carnegie said,
July 6, 2008 at 1:53 am
I’ll be embarrassed if I missed anything in the article by, say, not reading between the lines: Ben doesn’t even suggest that the research may have been done incompetently or simply invented as pure fiction. After Esure told Ben that “They are, however, happy to answer individual questions”, the rest of the article is merely speculating about the questions he could ask, and that in an ideal world, all journalists should and would apply to research that they were reporting on, whether by checking the numbers themselves, relying on a suitably trained colleague to do so, or depending on the scientific peer review process. (Which, however, as I’m told, allows a lot of junk to pass in.)
We can suspect that Esure are up to no good although I am stll unsure what it will get them.
Despard said,
July 6, 2008 at 10:50 am
It won’t get them my custom, for a start!
David44 said,
July 7, 2008 at 5:01 am
OK, so here’s a silly question, and probably one that’s been answered countless times here. What exactly going on in these journalists’ heads? Some PR person sends them a press release, and they – what? Simply put it onto the front page of the paper without performing any elementary checks on its credibility? If I sent the Daily Mail and the Daily Telegraph a press release saying (e.g.) that my statistical research has proved that people who take beach holidays have worse sex lives than those who go hill walking, wouldn’t anyone bother to check whether I had actually done any statistical research, or whether the research I had done actually proved what I said it proved?
Perhaps we should be amazed not that there is so much garbage in these papers, but that there is so little (relatively speaking), since they appear to be laying themselves open to every con-man in the country.
simonturner said,
July 7, 2008 at 9:16 am
@David44
May I suggest you read Nick Davies’ excellent Flat Earth News www.amazon.co.uk/Flat-Earth-News-Award-winning-Distortion/dp/0701181451/ref=sr_1_1?ie=UTF8&s=books&qid=1215418475&sr=8-1
In short, the answer is “yes” they put it on the front page without checking, but the book answers the more complex question “why?”.
emilypk said,
July 7, 2008 at 4:44 pm
Around this time of year various northern hemisphere location have a rat panic, like clockwork. Like they never saw rats before and haven’t thought deeply about what summer does to the food supply and the amount of time all animals (human and vermin) spend outside noticing each other. If you ever get the alleged data I’d be happy to look it over too–I spent a good decade studying rats and they are, overall, very sensible and predicatble creatures easily deterred by a rubbish bin with a lid. It’s the people that cause problems.
SpiderJ said,
July 7, 2008 at 5:18 pm
At the risk of being ostracised from this place, I should explain that I work in PR, and have run surveys to highlight issues for my clients.
However, I always advise clients to make the raw data available, and most of them do, if requested to do so by a journalist.
What is interesting to note – is how very rare it is for journalists to ask to see the raw data.
While certain PR companies (and their clients) may well be trying to pull a fast one by using poor data (or poor analysis of the data), this is only made possible by the extremely lax attitude of the majority of the media.
It has become a vicious circle – journalists see surveys as an easy way to fill column inches with accompanying scary headlines (you never see a survey giving you good news, do you?) and PR people see it as an easy way to get coverage for their clients.
The only real way to stop this is for journalists to become more rigorous in their questioning (or even asking ANY questions). But don’t hold your breath.
Toenex said,
July 8, 2008 at 9:12 am
I think it fairly true to say that few journalists, our host excluded, have little if any formal scientific training. This, together with the lure of the catchy headline makes it all too easy for them to forget that the plural of anecdote is not data. Couple this with an uncritical readership, themselves lost in a sea of opinions all of which are presented in pseudo-jargon rendered using neat fonts and you really have to worry about our future. Particularly, according to the Express, if you’re a dustbin man.
I’m interested in this idea of ‘Household reports’. It has never occured to me that there might be an organisation who would be interested in the number of wasps I’ve had to deal with in the kitchen. Should I be keeping some kind of record and how do I submit this data?
benb said,
July 8, 2008 at 11:15 am
@icthyophile: “…I’m sure people don’t mind being told that they need to “bee” logged in to post a comment – but sooner or later I expect someone is going to tell you that a man who can’t use a spellchecker can’t be taken seriously as an evangelist for good science…”
a spellchecker wouldn’t flag up bee as a problem, as it’s a valid word
emilypk said,
July 8, 2008 at 2:12 pm
A grammar checker might.
mjs said,
July 9, 2008 at 4:23 am
to JonM, #34:
Oh, you know. That whole plague thing kind of gave them a bad reputation. Especially in London.
Granted, it is really a bacterium which is responsible. But the rat brings the flea what brings the bacteria, so you can’t exactly ignore that.
They are vectors for other diseases, as well.
*
I suppose they do function (sloppily) as a self-propelling garbage disposal.
Sorry to put too fine a point on it, but a rat will also urinate & defecate after consuming food. “One rat produces 20-50 droppings per day and excretes 14 ml of urine per day.”“One rat produces 20-50 droppings per day and excretes 14 ml of urine per day” (source). So I’m personally not so sure how clean streets will be where a large number of rats dine well.
*
That said, tame rats can make wonderful pets. If you get them young and handle them well, they can be very sweet.
*
All of which is kind of beside the point of the original post (which looks to be more about freedom of information).
But I thought that, if you were serious about not knowing why rat increases get media play? It might help in case you were ever tempted to rent or buy a place with an actual rat problem.
….Plus I couldn’t resist… Forgetting the Plague? Tsk, tsk… 🙂
JonM said,
July 9, 2008 at 11:53 am
To MJS
Right, yeah, the plague. I was being slightly flippant, mainly because the article was illustrated by a very nice picture of a cute rat.
You know you can get plague from squirrels in the USA? (Seriously big warning signs about the dangers of squirrels in Utah/Arizona)
No one ever makes the same fuss about squirrels as they do about rats. I guess my point was why does this “study” focus on rats? (And not cockroaches as someone else pointed out)
Pigeons apparently carry 37 diseases which are communicable to humans. And I seem to see far more pigeons than rats whenever I go into the city centre.
And yes, despite liking rats, I wouldn’t really want to share my house with them.
SpiderJ said,
July 9, 2008 at 12:42 pm
JonM
Why does it focus on rats?
Er… because they are way more scary than pigeons, squirrels or cockroaches. This isn’t about reality – it’s about getting publicity for an insurance company.
If they could have come up with a story that said man-eating alligators are going to come up from the sewers and eat your left-over pizza, before tucking into your baby for dessert, then I suspect they probably would – just as long as they got a plug for Esure in there.
mjs said,
July 10, 2008 at 12:56 am
JonM,
How do you know the study itself focuses on rats? The only bit in the Guardian article that even pretends to be quoted by the newspapers from the study itself is this sentence:
“Household reports of wasps have risen by 39 per cent, squirrels by 23 per cent, mice by 17 per cent and rats by 12 per cent.”
Rats are included in the study. But I haven’t a clue what animal (if any) is the “focus” of it. Since only MischiefPR knows what was measured, only they can say what the focus really was.
Not that it stops the Express and the Daily Mail from focusing on the rats.
—–
Anyway. Sloppy figures aren’t limited to science news stories. A couple days ago, I read that a certain American city has seen “double-digit increases in murder, rape, burglary, larceny and shooting rates” so far this year.
I still don’t know what this double digit increase really means. +15/wk? +12 per 3 months? Only when all those crimes are combined? Or in each catagory individually?
If I only read that one story, I’d never know. It would not have even been that difficult to give out useful information. Someone simply chose not to, in service of making a splashy statement.
—–
So, since we can’t count on the newspapers to tell us anything useful, if we want to know more about murder rates (or rats, or wasps), then we would need access to other sources.
Oh, wait. That’s what the Guardian article was about. Access.
bob_calder said,
July 11, 2008 at 2:59 am
SpiderJ, alligators don’t like babies. But they do like women.
During 2006, women were attacked three times as often as men in Florida.
The lesson: you should always take a woman with you into the swamp. muah ha ha.
One year they apparently had a preference for boy scouts, another time it was german shepherds. Oh well…