Ben Goldacre, The Guardian, Saturday 24 July 2010
There is a pleasing symmetry in the ropey science you get from different players. When GlaxoSmithKline are confronted with an unflattering meta-analysis summarising the results of all 56 trials on one of their treatments, as we saw last week, their defense is to point at 7 positive trials, exactly as a homeopath would do. Politicians will often find a ray of positive sunshine in a failed policy’s appraisal, and promote that to the sky. Newspapers, similarly, will spin science to fit their political agenda, with surreal consequences (the Telegraph have claimed recently that shopping causes infertility in men, and the Daily Mail reckon housework prevents breast cancer in women).
But does the same thing happen in formal academic research?
Isabelle Boutron and colleagues set out to examine this problem systematically. They took every trial published over one month that had a negative result – 72 in total – and then went through each trial report to look for evidence of “spin”: people trying to present the results in a positive light, or distract the reader from the fact that the trial was negative.
First they looked in the abstracts. These are the brief summaries of the academic paper, and they are widely read, either because people are too busy to read the whole paper, or because they cannot get access to it without a paid subscription (a scandal in itself).
Normally, as you scan hurriedly through an abstract, you’d expect to be told the “effect size” – “0.85 times as many heart attacks in patients on our new super-duper heart drug” – along with an indication of the statistical significance of this result. But in this representative sample of 72 trials with negative results, only 9 gave these figures properly in the abstract, and 28 gave no numerical results for the main outcome of the trial, at all. It gets worse. Only 16 of these negative trials reported the main negative outcome of the trial properly, anywhere, even in the main body of the text.
So what was in these trial reports? Spin. Sometimes the researchers found some other positive result in the spreadsheets and pretended that this was what they intended to count as a positive result all along. Sometimes they reported a dodgy subgroup analysis. Sometimes they claimed to have found that their treatment was “non-inferior” to the comparison treatment (when in reality a “non-inferiority” trial requires a bigger sample of people, because you might have missed a true difference simply by chance). Sometimes they just brazenly banged on about how great the treatment was, despite the evidence.
There a lots of things in place to stop this kind of stuff from happening. Trials are supposed to be registered, before they begin, with their protocol described in full, so that highly motivated individuals can go back and check if researchers changed their minds about what constituted a positive result, retrospectively, after the results came in. There are also reporting guidelines, such as CONSORT, which formalise the information that is supposed to appear in any scientific paper resulting from a trial.
But there is no enforcement for any of this, everyone is free to ignore it, and commonly enough – as with newspapers, politicians, and quacks – uncomfortable facts are cheerfully spun away.