Publish or perish

Ben Goldacre
Thursday May 13, 2004
The Guardian

Talk about Bad science here

· If you visit the Royal Society – even women are allowed in, contrary to what you may have been told – you’ll see their motto proudly on display: “Nullius in verba” or on the word of no one. And what I like to imagine they’re referring to, in my geeky way, is the importance of publishing proper scientific papers, if you want to be trusted. This obviously comes as a surprise to most journalists and alternative therapists, so I’ll explain exactly why. Let’s say that you’ve decided to accept satan’s shilling and write for the Daily Mail. You want to write an article on MMR and how bad it is. Luckily, Dr Arthur Krigsman has been claiming for years now that he has found evidence linking MMR to autism and bowel disease, so you write about that. He may well have done so. But since he didn’t publish his findings, he can claim them until he’s blue in the face, because until we can see exactly what he did, we can’t see what flaws there may be in his methods. Maybe he didn’t select the subjects properly. Maybe he measured the wrong things. If he doesn’t write it up formally, we can never know.

· This is what scientists do: write papers, and pull them apart to make sure the findings are robust. We look for flaws in the experimental methods that may cause flaws in the results, that may cause flaws in the conclusions. Science is not about absolute facts from authority figures: you describe exactly what you did in the methods section, what your results were, and how wide the error margin was: then you describe your theory, contingently built on this fragile, contentious data. Without all that information, the findings and the figures are worthless.

Two weeks ago, you will remember, the biggest science story in the UK was Kevin Warwick’s study, showing that watching Richard and Judy improved IQ performance in 200 subjects. Four years ago the same result was reported, but with a study sample of 120 people, implying that this ridiculous finding was robust enough to be replicated. The study has never been properly published in a peer-reviewed academic journal, and Warwick says he doesn’t want to do so. But the study was misreported this time round: a press release incorrectly stated that there were 200 subjects, when in fact there was just one study, of 120. Which just goes to show that press releases on unpublished data are a rubbish basis for a report on a scientific experiment. If the experiment was properly published, and if journalists knew enough to trust only properly published data, stories like this would never run.