The power of anecdotes

August 28th, 2010 by Ben Goldacre in anecdotes, bad science, big pharma, media, statistics | 53 Comments »

Ben Goldacre, The Guardian, Saturday 28 August 2010

For simpletons and amateurs, there are good research methods, and bad research methods. In reality, different tools are valuable in different situations, and sometimes, even very tiny numbers of people can give you a meaningful piece of information: even an anecdote can be informative.

For example, if you produced a research paper about just two people who had gone into space, in a rocket or in a space shuttle, and then an extra eye had physically opened in the centre of their forehead, I would be concerned. Thats’s because going into space is a very rare lifestyle risk exposure: maybe a thousand people in total. And I’ve never seen a third eye physically open in the centre of someone’s forehead.

In this context, even though there are just two anecdotes, even though they are unsystematically gathered, it’s still interesting information, because the exposure and the outcome are both very rare, and because this is the first information available on the issue. I would want to look into it more carefully, and more systematically, but these anecdotes are still useful evidence.

That was a special case. This week the newspapers were filled with stories about NICE’s recommendation not to fund Avastin, a bowel cancer drug that costs £21,000 per patient, in draft guidance. This drug has been studied in a large randomised trial of 1401 patients receiving either chemotherapy with Avastin, or chemotherapy with placebo.

The trial isn’t perfect – no trial is, you can read the details online –  but it gives the best estimate of the true benefit of this drug, and overall, it shows that Avastin extends survival from 19.9 months to 21.3 months, which is about 6 weeks. Some people might benefit more, some less. For some, Avastin might even shorten their life, and they would have been better off without it (and without its additional side effects, on top of their other chemotherapy). But overall, on average, when added to all the other treatments, Avastin extends survival from 19.9 months to 21.3 months.

The Daily Mail, the Express, Sky News, the Press Association and the Guardian all described these figures, and then illustrated their stories about Avastin with an anecdote: the case of Barbara Moss. She was diagnosed with bowel cancer in 2006, had all the normal treatment, but also paid out of her own pocket to have Avastin on top of that. She is alive today, four years later.

Barbara Moss is very lucky indeed, but her anecdote is in no sense whatsoever representative of what happens when you take Avastin, nor is it informative. She is useful journalistically, in the sense that people help to tell stories, but her anecdotal experience is actively misleading, because it doesn’t tell the story of what happens to people on Avastin: instead, it tells a completely different story, and arguably a more memorable one – now embedded in the minds of millions of people – that Roche’s £21,000 product Avastin makes you survive for half a decade.

Rationing healthcare resources is a soul-destroying and unavoidable horror, in which some people who are dearly loved will always die, and this makes it an irresistible magnet for questionable behaviour from people who are happy to release themselves from the burden of being realistic about difficult decisions.

Journalists can exploit these impossible decisions for outrage, and the pleasure of leading a popular campaign, but so can politicians: the Conservatives in opposition even invented a £50m fund to pay for the drugs that will save your life, in a scheme that is no more realistic than Barbara Moss’s unrepresentative anecdote. With drugs that cost £21,000 per person, your £50m will buy you precisely 2,381 patients on Avastin living an extra 6 weeks, and then it’s gone forever.

£50m clearly gets us nowhere at all. You’re always going to draw the line somewhere, and if you paid £200,000 for 6 weeks of life there would still be more you could do. Whoever draws that line, wherever it falls, is always going to be pilloried and despised. When you’re writing about such an incredibly easy and emotive target, it might be fair to at least use a representative anecdote for illustration, instead of Barbara Moss.

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

53 Responses

  1. steveg said,

    September 2, 2010 at 4:20 pm

    This relates to Herceptin rather than Avastin, but I think it’s relevant to the discussion.

    Demands for ‘off-licence’ access to trastuzumab (Herceptin): content analysis of UK newspaper articles

    Background  Sensationalized reporting styles and a distorted framing of health-care issues in newspapers may trigger inappropriate commissioning decisions. We evaluated UK press coverage of pre-licensing access to trastuzumab (Herceptin) for early breast cancer as a case study.

    Methods and findings  Content analysis of newspaper articles published between April 2005 and May 2006 were coded by two researchers for interest groups represented, claims made and sensationalized reporting. Disagreements in coding were resolved by a third researcher. One thousand and ninety published articles were identified in the study period and a 20% sample (n = 218) was included in the content analysis. Most articles (76%, 95% CI 71–82) included claims about the clinical benefits of trastuzumab, and this was significantly higher than those expressing the uncertainty surrounding such benefits (6%, 95% CI 3–9) or those that discussed the potential harms (5%, 95% CI 2–8). Articles were significantly more likely to feature claims made by a breast cancer survivor or family member than any other interest group (P < 0.0001). Almost half of the articles carried some message to the effect that trastuzumab would make the difference between life and death (47%, 95% CI 40–53). Over a quarter (28%, 95% CI 22–34) suggested that trastuzumab is a ‘miracle drug’ or similar.

    Conclusions  The benefits of drugs are highlighted, frequently using sensationalist language, without equal consideration of uncertainty or risks. Health-care purchasers should express decisions in opportunity cost terms; journalists should give fairer coverage to such arguments.

  2. amcle said,

    September 3, 2010 at 1:14 pm

    This blog from the BMJ summarises things nicely, and answers some of the cost questions posed above.

  3. johnkooz said,

    September 14, 2010 at 5:22 pm

    @bengoldacre Finally a leader who’s not interested in being part of the baby-boomer generation “boat”. Good times. (that is a conclusion derived not just from this article but seeing ben in that long interview with the guy with odd hair and peculiarly large head where ben and he each had three beverages). jolly good!