Cherry picking is bad. At least warn us when you do it.

September 29th, 2011 by Ben Goldacre in academic pr, academic publishing, aric sigman, schools, systematic reviews | 9 Comments »

Ben Goldacre, The Guardian, Saturday 24 September 2011

Last week the Daily Mail and the Today programme took some bait from Aric Sigman, an author of popular sciencey books about the merits of traditional values. “Sending babies and toddlers to daycare could do untold damage to the development of their brains and their future health,” explained the Mail.

These news stories were based on a scientific paper by Sigman in The Biologist. It misrepresents individual studies, as Professor Dorothy Bishop demonstrated almost immediately, and it cherry-picks the scientific literature, selectively referencing only the studies that support Sigman’s view. Normally this charge of cherry-picking would take a column of effort to prove, but this time Sigman himself admits it, frankly, in a PDF posted on his own website.

Let me explain why this behaviour is a problem. Nobody reading The Biologist, or its press release, could possibly have known that the evidence presented was deliberately incomplete. That is, in my opinion, an act of deceit by the journal: but it also illustrates one of the most important principles in science, and one of the most bafflingly recent to emerge.

Here is the paradox. In science, we design every individual experiment as cleanly as possible. In a trial comparing two pills, for example, we make sure that participants don’t know which pill they’re getting, so that their expectations don’t change the symptoms they report. We design experiments carefully like this to exclude bias: to isolate individual factors, and ensure that the findings we get really do reflect the thing we’re trying to measure.

But individual experiments are not the end of the story. There is a second, crucial process in science, which is synthesising that evidence together to create a coherent picture.

In the very recent past, this was done badly. In the 1980s, researchers such as Celia Mulrow produced damning research showing that review articles in academic journals and textbooks, which everyone had trusted, actually presented a distorted and unrepresentative view, when compared with a systematic search of the academic literature. After struggling to exclude bias from every individual study, doctors and academics would then synthesise that evidence together with frightening arbitrariness.

The science of “systematic reviews” that grew from this research is exactly that: a science. It’s a series of reproducible methods for searching information, to ensure that your evidence synthesis is as free from bias as your individual experiments. You describe not just what you found, but how you looked, which research databases you used, what search terms you typed, and so on. This apparently obvious manoeuvre has revolutionised the science of medicine.

What does that have to do with Aric Sigman, the Society of Biologists, and their journal, The Biologist? Well, this article was not a systematic review, the cleanest form of research summary, and it was not presented as one. But it also wasn’t a reasonable summary of the research literature, and that wasn’t just a function of Sigman’s unconscious desire to make a case: it was entirely deliberate. A deliberately incomplete view of the literature, as I hope I’ve explained, isn’t a neutral or marginal failure. It is exactly as bad as a deliberately flawed experiment, and to present it to readers without warning is bizarre.

Blame is not interesting, but I got in touch with the Society of Biology, as I think we’re more entitled to have high expectations of them than Sigman, who is, after all, some guy writing fun books in Brighton. They agree that what they did was wrong, that mistakes were made, and that they will do differently in future.

Here’s why I don’t think that’s true. The last time they did exactly the same thing, not long ago, with another deliberately incomplete article from Sigman, I wrote to the journal, the editor, and the editorial board, setting out these concerns very clearly.

The Biologist has actively decided to continue publishing these pieces by Sigman, without warning. They get the journal huge publicity: and fair enough. I’m no policeman. But in the two-actor process of communication, until they explain to their readers that they knowingly present cherry-picked papers without warning – and make a public commitment to stop – it’s for the reader to decide whether they can trust what they publish.

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

9 Responses

  1. Richard Gadsden said,

    September 29, 2011 at 3:20 pm

    I have a question.

    Would it make sense to have a peer-review process for entire journals (not issues of a journal) to determine whether that journal’s peer-review processes are adequate for it to continue to be part of the scientific literature?

  2. deevybee said,

    September 29, 2011 at 6:59 pm

    Oh the irony. The Society of Biology has a blog today in support of the Sense About Science campaign for evidence: they “don’t want people to continue getting away with making claims without supporting evidence. We want things to change, so that anyone making a claim expects to be challenged, and is made to stand by their words. The only way to do this is to get more people asking for evidence”

  3. Anglica said,

    October 3, 2011 at 11:00 am

    Great article or post what you exactly call it! Any way it was my good luck to read such excited phrases today morning. well writer had a deep knowledge about it that’s why wrote such a post.

  4. oakdale said,

    October 5, 2011 at 11:18 am

    The dispute boils down to how much emphasis on opinion vs fact there was in the press release, see the response in today’s Guardian. But it would be disingenuous to pretend to be unaware of the press’s tendency to seize on sensation.

  5. p_tipper said,

    October 7, 2011 at 5:22 pm

    Isn’t it normal practice for a journal to send prospective paper to respected/experienced reviewers prior to publication to ensure the quality of the paper? Square peg for a square hole mentality!

  6. Deliscience said,

    October 10, 2011 at 1:53 am

    The new niche-market of science: separating the chaff from the wheat and making hay from it.

  7. Stuartp said,

    February 29, 2012 at 9:47 am

    This crazy is presenting ‘the facts about alcohol’ to the sixth-formers at PGS next week. I hope that they ‘destroy’ him.

    That’s ‘destroy’ as in the UK teen vernacular, not the Dept of Homeland Security’s…

  8. Brain Jim said,

    April 8, 2012 at 10:11 pm

    BTW, when you remove the real spam, get rid of my mock spam (and this post) too! I’m not trying to clog up the discussion.

  9. joey89924 said,

    November 16, 2012 at 2:21 am

    oh , disingenuous to pretend to be unaware of the press’s tendency to seize on sensation.