Saturday March 8 2008
It was gratifying to see – after only a one-week delay – the government announcing that they would follow my suggestion on the comment pages last week, and demand that drug companies disclose all trial data, to make sure they’re not hiding anything. This has been pegged to the issue of undisclosed side-effects of antidepressants, because a drug company hiding side-effects is intuitively evil.
This is unfortunate because – as I have repeatedly argued – much more worrying is the tendency to only publish results which show your drug performing well, and to leave the less flattering trials in a desk drawer. This happens much more commonly, it makes drugs look better than they are, it wastes money, it exposes people to risks of unnecessary side-effects, and it leaves doctors prescribing on inaccurate information.
But how can you tell if the research literature on a given subject has been rigged? It’s a tricky problem, because you’re chasing evidence for the existence of trials you cannot see. One option is to use mathematical tools, and something called a funnel plot, one of the cleverest ideas of the last century. It’s so clever that you might need to concentrate for the next bit.
Let’s imagine that there are 30 trials on a given drug. Some are big, and more accurate. Some are small and less accurate, with more random noise. You’d expect that the big, accurate trials should all cluster together around the true finding, all giving similar results for the efficacy of a drug. Meanwhile the smaller, rubbish trials – because they are less accurate measures of the drugs efficacy – will be scattered about randomly, some showing the treatment to be better than the good big trials indicate, some showing that it is worse.
You could then plot all your trials on a graph, one dot for each trial. On the x-axis, left to right, is “how good the drug was shown to be by this trial” and on the y-axis, “how methodologically sound and large the trial was”. If there is no publication bias, you should get a triangle shape: at the top of your graph, you will see all your good-quality, accurate trials, clustered together around the true answer. At the bottom of the graph, you will see a broad smear of results, the poor quality trials showing random variation.
But if there is publication bias, you will see a distorted triangle: the small, poor-quality trials at the bottom will be smeared over to the right, because small trials with unwelcome results are much more likely to be overlooked, and dumped in desk drawers, than huge multicentre collaborative studies involving dozens of academics and tens of thousands of participants, which are almost definitely going to get published. If you get a distorted triangle, you know there are some interesting negative trials missing.
This is a funnel plot demonstrating possible publication bias in a set of studies looking at… publication bias. Seriously.
This happens repeatedly, in too many fields to list, and it doesn’t just happen because of big pharma evil. If you’re an academic, and you get a negative result, you’re less likely to get round to publishing it, because it’s not going to get in a big journal, so it’s not going to buff your department’s “research assessment exercise” score for this year, nobody’s going to invite you to give lectures about it, the whole thing feels like a disappointing waste of time.
And even though you know in your heart that a negative finding is still an interesting piece of evidence, there are undergraduates that need teaching and you hate doing that and the references for that other paper need reformatting before submission to a third journal and before you know it five years have passed and nobody’s even mentioned the negative finding at the departmental meeting since the last prof retired so you can probably get away with leaving it for another year at least and possibly even ideally until you die.
I’ve said it before: all trials should be registered before they start, no trial – by anybody – should be passed by any ethics committee without a firm commitment to publish. No exceptions, because bad data costs lives.
· Please send your bad science to firstname.lastname@example.org
Publish your negative results here:
PLoS Clinical Trials was launched specifically with the aim of accepting for publication all properly reported, correctly conducted randomized trials, irrespective of outcome. it’s part of PLOSOne now www.plosone.org clinicaltrials.ploshubs.org/)
Trials www.trialsjournal.com is big on trial methodology, but also publishes results, whether positive or negative, and is edited by Doug Altman, king of propriety, god of Stats, and overlord of the CONSORT guidelines. Like any decent journal they will only accept your trial if it has been registered.
Try also BMC Research Notes: www.biomedcentral.com/bmcresnotes/
it’s new, not just trial data, a quiet hole to get your results onto the record.
and there’s the old Journal of Negative Results in BioMedicine www.jnrbm.com/
In case you missed it, here is the letter from Kent Woods of the MHRA where he announces the new legislation and, if I’m not mistaken, calls the chief exec of GSK an “arse”, to his face:
“Such a course of action should be unnecessary in an industry which relies so heavily on public trust and aspires to high ethical standards.
“I would have thought it self-evident that such information should be made available promptly to the regulator in order that action can be taken to protect public health. However, that moral responsibility now needs to be to be insisted upon by the unambiguous force of law.”