Saturday April 26 2008
And so our ongoing project to learn about evidence through nonsense enters its sixth improbable year. This week, the assembled celebrity community and vitamin pill industry will walk us through the pitfalls of reading through a systematic review and meta-analysis from the Cochrane Collaboration, an international not for profit organisation set up 15 years ago to create transparent, systematic, unbiased reviews of the medical literature on everything from drugs, through surgery, to community interventions. Last week Cochrane produced a comprehensive, gold standard review, looking at 67 trials describing the experiences of 230,000 people, which showed that antioxidant vitamin pills do not reduce deaths, and in fact may increase your chance of dying. [Summary here, full document here, plain language summary on page 2].
In the Health Food Manufacturers Association press release, both Gloria Hunniford and Sir Cliff Richard issued their own definitive refutations. Carole Caplin explained: “It must be obvious to everyone who hasn’t got a vested interest in supplements that this review is absolute rubbish, it contains fundamental flaws.” In a press release issued on behalf of the food supplement industry. Criticising an academic collaboration which does not accept any corporate funding.
So what were these flaws? The entire pill community were worried by the way that trials were selected for inclusion in the group analysis. Dr Rajendra Sharma, ex-head of the Harley Street-ish Hale Clinic – a man who advertises his use of a “bioresonance” machine called the Quantum Xrroid Consciousness Interface to diagnose his patients – explained science to the nation on More4 News. “The writers of this study started with 16,000 studies, and we’re asking the question why did it go down to 68, clearly there’s a bias that we’re not yet quite sure about.”
Let the mystery be revealed. The answer to his question can be found in figure 1 on page 156 of the Cochrane report (which of course he read). Of the 16,111 studies which the Cochrane authors found – by using various search terms in various databases – 12,703 were duplicates, 983 were in children and so not applicable to this review’s predescribed remit, and so on.
Carole Caplin picks up the ball. “With nearly 750 studies to choose from, why did the researchers manage to focus on just 67? That’s less than nine per cent of the total number of clinical trials on antioxidants available.” Well, Carole. Many of those trials were excluded simply because they were not the type of trials being looked at in the study – in accordance with the standard protocol for Cochrane reviews – but 400 trials were excluded because there were no deaths in them. This Cochrane review was a study of deaths, comparing deaths on antioxidant pills to deaths on placebo pills. You need deaths to be reported to put a trial in such an analysis.
These deathless trials were mostly small and brief, representing only 40,000 people in total (page 5, kids). But in any case, as a precaution, the Cochrane authors did a re-analysis of their 230,000 people data, adding in hypothetical fake data, for the 40,000 people, with one death in each of the vitamin and placebo groups: it made no difference. Carole may have skipped that part of the 191 page Cochrane report which she read in full and understood [it's on page 7].
“This isn’t even a new study,” continues Caplin: “it’s simply a re-hash of old work which was widely criticised in 2007 for its inaccuracies.” Interesting point, Carole. This was indeed a reworking of an earlier review published in the journal JAMA, but updated, and in the format required by Cochrane, resulting in a report about 10 times longer than the original journal article. I should hope it did incorporate criticisms from previous work in the same area: this is the whole point of publishing papers, and opening them to informed criticism.
A key question with all research is how to apply it in practice. Are the interventions in the study comparable to the decisions in your real life? “The analysis largely focused on extraordinarily and atypically high doses of antioxidant vitamins,” explained David Adams of the HFMA: “Supplement users would have some trouble trying to replicate this kind of daily intake.”
Well, not that much trouble. I went to Holland and Barrett. In the Cochrane review, the mean dose of betacarotene was 18mg: H&B sell 100 capsules of 15mg betacarotene for £7.49. So practically identical, then. The mean dose of vitamin E was 570 IU: H&B sell 100 capsules of 1000IU – twice that dose – for £19.99. Perhaps David Adams thought nobody would check.
But my favourite insight comes from the actress Jenny Seagrove, and it is a valid one. “I’m not going to be bullied by this dismal research paper – I am 100% confident that the vitamins and mineral supplements I use are safe and effective and I will continue to use them when I choose.” This is the key move. Do you put evidence into practice? Yes: not as an automaton, but at your discretion, taking into account not just the best possible evidence, but also the individuals’ preferences. That is the core of evidence based practice, so munch away, Jenny, and good luck to you. But the publics’ understanding of evidence is far more important than your vitamin pills, so show a little consideration.
· Please send your examples of bad science to firstname.lastname@example.org
Mostly in the text above, but because they’re important…
Here is the silly HFMA press release:
Here is the Cochrane review:
Here is the definitive early document on the nature of evidence based practice:
And here is a video of Sir Cliff Richard. I think I was just slightly sick in the back of my throat.
In many ways what I found most striking were the similarities between the dissembling of the $60bn food supplement pill industry in this case, and the well established strategy of “manufacturing doubt”, first seen in the tobacco lobby’s war with epidemiology over 40 years ago.
The history of this corporate strategy is very well covered in David Michaels’ new book, Doubt is Their Product: How Industry’s Assault on Science Threatens Your Health, on which there is an excellent essay here.
The sabotage of science is now a routine part of American politics. The same corporate strategy of bombarding the courts and regulatory agencies with a barrage of dubious scientific information has been tried on innumerable occasions — and it has nearly always worked, at least for a time. Tobacco. Asbestos. Lead. Vinyl chloride. Chromium. Formaldehyde. Arsenic. Atrazine. Benzene. Beryllium. Mercury. Vioxx. And on and on. In battles over regulating these and many other dangerous substances, money has bought science, and then science — or, more precisely, artificially exaggerated uncertainty about scientific findings — has greatly delayed action to protect public and worker safety. And in many cases, people have died.
Tobacco companies perfected the ruse, which was later copycatted by other polluting or health-endangering industries. One tobacco executive was even dumb enough to write it down in 1969. “Doubt is our product,” reads the infamous memo, “since it is the best means of competing with the ‘body of fact’ that exists in the minds of the general public. It is also the means of establishing a controversy.”