Ben Goldacre, The Guardian, Saturday 28 August 2010

For simpletons and amateurs, there are good research methods, and bad research methods. In reality, different tools are valuable in different situations, and sometimes, even very tiny numbers of people can give you a meaningful piece of information: even an anecdote can be informative.

For example, if you produced a research paper about just two people who had gone into space, in a rocket or in a space shuttle, and then an extra eye had physically opened in the centre of their forehead, I would be concerned. Thats’s because going into space is a very rare lifestyle risk exposure: maybe a thousand people in total. And I’ve never seen a third eye physically open in the centre of someone’s forehead.

In this context, even though there are just two anecdotes, even though they are unsystematically gathered, it’s still interesting information, because the exposure and the outcome are both very rare, and because this is the first information available on the issue. I would want to look into it more carefully, and more systematically, but these anecdotes are still useful evidence.

That was a special case. This week the newspapers were filled with stories about NICE’s recommendation not to fund Avastin, a bowel cancer drug that costs £21,000 per patient, in draft guidance. This drug has been studied in a large randomised trial of 1401 patients receiving either chemotherapy with Avastin, or chemotherapy with placebo.

The trial isn’t perfect – no trial is, you can read the details online –  but it gives the best estimate of the true benefit of this drug, and overall, it shows that Avastin extends survival from 19.9 months to 21.3 months, which is about 6 weeks. Some people might benefit more, some less. For some, Avastin might even shorten their life, and they would have been better off without it (and without its additional side effects, on top of their other chemotherapy). But overall, on average, when added to all the other treatments, Avastin extends survival from 19.9 months to 21.3 months.

The Daily Mail, the Express, Sky News, the Press Association and the Guardian all described these figures, and then illustrated their stories about Avastin with an anecdote: the case of Barbara Moss. She was diagnosed with bowel cancer in 2006, had all the normal treatment, but also paid out of her own pocket to have Avastin on top of that. She is alive today, four years later.

Barbara Moss is very lucky indeed, but her anecdote is in no sense whatsoever representative of what happens when you take Avastin, nor is it informative. She is useful journalistically, in the sense that people help to tell stories, but her anecdotal experience is actively misleading, because it doesn’t tell the story of what happens to people on Avastin: instead, it tells a completely different story, and arguably a more memorable one – now embedded in the minds of millions of people – that Roche’s £21,000 product Avastin makes you survive for half a decade.

Rationing healthcare resources is a soul-destroying and unavoidable horror, in which some people who are dearly loved will always die, and this makes it an irresistible magnet for questionable behaviour from people who are happy to release themselves from the burden of being realistic about difficult decisions.

Journalists can exploit these impossible decisions for outrage, and the pleasure of leading a popular campaign, but so can politicians: the Conservatives in opposition even invented a £50m fund to pay for the drugs that will save your life, in a scheme that is no more realistic than Barbara Moss’s unrepresentative anecdote. With drugs that cost £21,000 per person, your £50m will buy you precisely 2,381 patients on Avastin living an extra 6 weeks, and then it’s gone forever.

£50m clearly gets us nowhere at all. You’re always going to draw the line somewhere, and if you paid £200,000 for 6 weeks of life there would still be more you could do. Whoever draws that line, wherever it falls, is always going to be pilloried and despised. When you’re writing about such an incredibly easy and emotive target, it might be fair to at least use a representative anecdote for illustration, instead of Barbara Moss.