Ben Goldacre
Saturday April 14, 2007
The Guardian
[oh, I love the subs, but there was a slightly bonkers headline in the paper today on this column, as sometimes happens… this is why I don’t mock people for what’s written by someone else in their headlines…]
So here’s an interesting question. Lots of us wander around quite happily with a “dolphins good, drug companies bad” morality in our heads; and this is entirely reasonable, they are quite bad. But how easy is it to show that drug companies kludge their results, and to explain what they’ve done to a lay audience?
On an individual level, it is sometimes quite hard to show that one trial has been deliberately rigged to give the right answer for the sponsors. Overall, however, the picture emerges more clearly. The issue has been studied so frequently that in 2003 a systematic review found 30 separate studies looking at whether funding affected the findings, and overall, studies funded by a pharmaceutical company were four times more likely to give results that were favourable to the company than independent studies.
But one classic review of bias tells a genuine Alice in Wonderland story. They found 56 different trials looking at NSAID painkillers: drugs like ibuprofen, diclofenac and so on. People often invent new versions of these drugs in the hope that they might have fewer side effects, or be stronger.
Now, these were trials in which one painkiller was compared against another, rather than against a placebo. Because if you think about it, people in pain tend to look at you a bit impatiently when you offer them a placebo. And in every single trial, the sponsoring manufacturer’s drug came out as better than, or equal to, the others in the trial. On no single occasion did the manufacturer’s drug come out worse.
Philosophers and mathematicians talk about a phenomenon known as “transitivity“. If A is better than B, and B is better than C, then C cannot be better than A. To put it bluntly, the results of this review of 56 trials exposed a singular absurdity: these drugs, surely, cannot all be better than each other?
But people then began to study the methodological flaws in big pharma studies, and to their astonishment found that industry-funded trials turn out to have no worse research methods, or better research methods, on average, than independent trials.
The most they could pin the drug companies down on were some fairly trivial howlers: using inadequate doses of the competitor’s drug as a control, for example, or making claims that somewhat lyrically exaggerated a positive finding in the results. Both heinous and cheeky, of course, but in both cases, at least, these were transparent flaws. You only have to read a trial to see that they have given a miserly dose of a painkiller, and you should read the “results” section to decide what the “results” of a trial are, because the discussion and conclusions section, at the end of the paper, is like the comment pages in a newspaper. It’s not where you get your news from.
The real action in how drug company data seems to be biased overall seems to come largely from the burying of bad results, known as “publication bias“, and as I’ve geekily argued many times before, having a clinical trials database, where people are forced to register their trial before they start, knocks this on the head.
But there is a bigger political backstory to all this. We don’t like big pharma, because it makes money out of what we think should be a “helping people” industry: a wider, unspoken dislike of brutal capitalism, perhaps. But more than that, we don’t like big pharma because we have disempowered ourselves and our states.
Everybody has some kind of interest in their results; and everyone makes mistakes, perhaps – we’ll say unconsciously – more often in their own favour. So science relies on independent replication; but drug trials are so expensive, and state funding of research so miserly, that pharmaceutical research is rarely independently funded. By which, of course, we mean it’s rarely state funded.
I’m totally up for the stuff about the dolphins being good, and big pharma bad. But if only 10% of pharmaceutical research is funded outside the pharmaceutical industry, I’m not convinced that’s entirely the industry’s fault.
Please send your bad science to bad.science@guardian.co.uk
References:
In the text as links above, but for formality’s sake…
A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, et al. (1994) Arch Intern Med 154:157–163.
Pharmaceutical industry sponsorship and research outcome and quality: systematic review. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) BMJ 326:1167-1170
neoteny said,
April 14, 2007 at 8:33 am
I don’t think making money is wrong, in principle, whether it’s for helping people, feeding people, clothing people, or providing them with a newspaper.
Azimov said,
April 14, 2007 at 8:39 am
Well said, its never just as simple as black and white.
jackpt said,
April 14, 2007 at 9:10 am
This is interesting and the things that immediately spring to mind are the costs of research and whether there are effeciency implications. The two things I’ve heard often in this debate are “pharmaceutical research is very expensive” (suggesting that state funding is not enough) and “state funding is less efficiently spent than private funding” (which seems paradoxical given there’s less of it).
What does phamaceutical research cost and what are the pros/cons of state funded research?
Beermonkey said,
April 14, 2007 at 9:53 am
It’s estimated that it costs round about eight hundred million pounds to get a drug from phase I to market, there or there abouts. However, only about a tenth of those drugs getting to trials ever make it through, so it’s a spectacularly expensive business. I have to say that although big pharma isn’t exactly whiter than white, I have no real issue with them making profits – each drug has a 14-year profitable window before the patent runs out, and each one has to make a billion in that period for the industry to be in any way sustainable.
Beermonkey said,
April 14, 2007 at 10:23 am
Fair enough, currency confusion on my part. In any case, a dirty great big bag of cash.
psychopharm said,
April 14, 2007 at 12:29 pm
Not all pharmacological research is in the format of “clinical trials” and not all business investment is bad news. We need to be a little more open minded here, I suspect, in terms of the Public Good?
Robert Carnegie said,
April 14, 2007 at 1:54 pm
Not all good treatments are pharmaceutical! The pharmaceutical industry biases healthcare towards handing out pills – something that can be bought and sold as a commodity!
Something like “publication bias” will arise where results of preliminary trials do or do not make the product look good. Commercially interested money will follow the promising results. Independent money may go wherever abstract scientific interest lies. So the venture capital will be put into the good bets, and will get the clinically better results.
That isn’t to say that they don’t bury bad news, too!
A public clinical trials database will not eliminate commercial game-playing with the system. It only calls for significantly more skilful players. So, it will make it harder to make pharmaceutical profits dishonestly – but how much harder? Will playing the system become an area of specific expertise, just as patent law is?
Ben Goldacre said,
April 14, 2007 at 2:01 pm
“Not all good treatments are pharmaceutical! The pharmaceutical industry biases healthcare towards handing out pills – something that can be bought and sold as a commodity!”
that’s absolutely true, and it cuts for both the pharmaceutical pill industry and the quack pill industry. i don’t want to go giving out future columns before they’re in the paper, but last week the BMJ published a robust trial demonstrating the efficacy of a parenting program to improve all the things which the fish oil pill people claim their fish oil pills help. nobody in the press reported it, as far as i know.
the media love pill stories.
the pill manufacturers – supplement and pharma – love pill stories.
the public love pill stories.
especially for complex social problems.
Bob O'H said,
April 14, 2007 at 5:33 pm
On pills.
Why a trials database won’t solve the problems of bias. (but it will help. At least it will help create jobs for us statisticians)
I suspect that what is needed is a system where Phase III trials are carried out independently, but paid for by the company that developed the pill. Of course, ensuring probity might be difficult (and will require more statisticians to be employed).
Bob
Ben Goldacre said,
April 14, 2007 at 5:40 pm
trials database almost completely eradicates publication bias, as i said.
the Harlot paper you link to is very good, and a christmas BMJ comedy classic, but i’m not sure david sackett who authored it would agree with your assertion about clinical trials databases.
the other problems with trial methodology that they flag up in that paper are a key issue, but if you remember, industry trials – surprisingly – do not seem to be of lower methodological quality (and in any case when they are it’s easy or at least possible to spot in the paper itself and therefore comment upon and account for). publication bias and missing data seem to be the key issues and so a universally enforced clinical trials database, and vigilance on protocols, is essential.
Teek said,
April 14, 2007 at 7:47 pm
the point about 10% of pharmaceutical research being non-industrial hits the nail on the head – on several counts…
firstly, this means that drugs for conditions afflicting white middle class affluent urban folk will be developed over those afflicting black poor country-dwelling folk – because of the drive for profits.
secondly, it means that industry can say “it costs X to develop drug Y” and then set the market value for that drug at any arbitrary value, without any legitimate challenge to the figure. if 40% of drugs came from public-sponsored drugs, the cost of industry-sponsored drugs would come down. as an example, the NHS gave close collaborators of my research group less than a million to develop a gene therapy trial for immune deficiency, they cured over ten patients thus far – the drugs that Pharma companies charged for previous to this treatment cost almost £100K per patient per year.
we need more goverment-funded small molecule research, but which govt is brave enough to stick its neck out…?!
JQH said,
April 15, 2007 at 11:29 am
At least Big Pharma doesn’t adopt the homeopathic stance ie
“Double-blinded trials aren’t suitable for testing our product so we’re going to ignore the result and push these pills anyway.”
Evil Monster said,
April 15, 2007 at 7:00 pm
I work in pharma. Good article. A couple of points.
Pharma companies employ a great many people to design, conduct and analyse clinical trials, and they have the money and motivation to do it well. It should be no surprise that the quality of them tends to be better than independently run trials. Presumably, it is “astonishing” to some people because they think PHARMA = EVIL. I think that pharma has earned some of its bad reputation, but it shouldn’t be generalised to everything they do.
I agree with the principle of a clinical trials database. Something else that pharmas are sometimes accused of is publishing trials they don’t like in low-circulation journals; burying them another way. However, big-circulation journals won’t publish trials that show nothing of interest. They only want to publish exciting results. So the journals also have to take some responsibility for trials appearing to get buried.
On # 14 above. If a trial is randomised and well conducted (usually meaning double blind and so on), it should matter who does it. Pharmas have the infrastructure for conducting the trials, so having pharma pay for them and someone else conduct them doesn’t make much sense to me. Also, the argument that an independent statistician should do the analysis seems sensible, but doesn’t stand up to scrutiny. Pharmas have to describe in advance (and often in agonisingly tedious detail) how the analysis is going to be done, either in the trial protocol or in a separate statistical analysis plan. So if an independent statistician does it, the results will be identical (down to the precision of the computer). I think JAMA insists on an independent statistician reanalysing pharma trials before they’ll publish them. It’s purely a exercise in seeing if two different computers can produce the same numbers from the same data.
Robert Carnegie said,
April 15, 2007 at 9:42 pm
There isn’t an absolute popular bias towards new-drug stories above other approaches, I think. I know because I read it here. Lifestyle-type disciplines also make good stories – whether it’s physical exercises, feng shui, even eating special fruit or having your neck bent or needles stuck into you. Exotic education systems are particularly popular. And of course religion – when the press prints a picture of the nation’s sports hero’s injured operative member for us all to pray over.
Unfortunately, the effective ones are boring hard work with rather too much common sense in them. It works but people don’t do it because it’s inconvenient. Like a calorie controlled diet.
jay said,
April 16, 2007 at 1:08 pm
Re: transitivity and how these drugs can all be better than each other…
What has been missed here is that it is possible for A to be better than B, B better than C, C better than A, if the criteria for “better than” is redefined in each instance.
The Pharma companies know perfectly well how their products are superior and also inferior to a competitors drug.
Trials are designed to reflect this: drug A reduces symptoms better than pill B, pill B has less side effects than C, patients taking C have greater long term survival rates. Everyone wins.
While a clinical trials database would prevent burying of poor results, it would require the MHRA or similar to independently design trials rather than just approve them to guard against these apparent non-transitive results.
Dr Aust said,
April 16, 2007 at 3:32 pm
jay’s comment is pertinent here, and also see previous thread about surrogate outcomes.
Obviously a trial funded by the company selling drug A tends to “accentuate the positives” of drug A vs drug B… and vice-versa for a trial funded by the company that owns B.
This, of course, is exactly why bodies like NICE / NIHCE were set up, as there needs to be some advice somewhere free of all the string-pulling and vested interests. It is hard to manage this, though, because the ties of many (most?) leading “opinion former” clinicians to the Pharmas industry are so strong. Not to mention the public outcry that can result if new drugs AREN’T waved through following their launch fanfare. (see herceptin passim ad nauseam).
…which is why it was so disappointing that the NHS decided to stop paying for doctors to be sent the Drugs and Therapeutics Bulletin (which publishes independent advice on drugs).
Also in this vein, note that Eisai and Pfizer have been given leave to have a judicial review (Lawyers! Fees! £££!) of the process by which NICE reached it’s negative view about the use of the cholinesterase inhibitor Aricept in the early stages of Alzheimer’s.
Try reading the report on this:
www.pmlive.com/pharm_market.cfm?showArticle=1&ArticleID=5246
– and then for a subtle contrast in reporting, try the earlier press release on Pfizer’s website under the studiedly non-emotive headline
“Alzheimer’s patients betrayed by NICE”
www.pfizer.co.uk/template2.asp?pageid=381
BrickWall said,
April 17, 2007 at 9:44 am
Ben in post 12 (I think its a 12, got a 2 in it anyway!) you refer to BMJ papers on parenting strategies.
Interesting to note that J. Hutchings author of paper is also on eo fthe few “mentors” in the technique tested/proposed. As such if the programme was bought by the Govt. and rolled out she may well benefit.
Of course this doesn’t mean there was anything wrong with the trial at all, it may well be a very good study etc I haven’t read it yet. Its just a bit ironic that you bring it up in the same thread as referencing the appearance of bias in pharma tests etc.
Shirleybobs said,
December 4, 2009 at 12:18 pm
I’m a natural sceptic and perhaps not a good patient to my GP or my rheumatologist. (Define “good”) I want to know everything about what I am prescribed and read all the leaflets that come in the box. I take methotrexate and don’t like the possible side effects and the continual blood tests that go with it so when a friend, having used it herself, genuinely recommended CMO, (google it, there’s masses of it) I thought it merited investigation. I wasn’t about to just fire off a cheque. Finding unbiased info from people other than those trying to flog the stuff was difficult but I did turn up one trial by Dr H Siemandi et al which I read carefully. It seemed plausible, maybe it’s entirely true but having read Bad Science just last week I’m beginning to wonder if this trial is genuine.
I bought the CMO and took it as directed and it didn’t work for me. Statistically possible that I’m one of the unlucky ones of course.
However, despite knowing that you are not to continue with the methotrexate it is not clear that you have to have six months free of this drug before taking the CMO. You only find that out when the bottles arrive. Although I don’t like the stuff it does control the disease well for me so the prospect of not using it and enduring 6 months of pain, immobility and joint damage is not attractive. I stopped taking it for a week before starting the CMO and as I said saw no long term benefit after the two courses I tried. Is there any way of assessing the data in the trial mentioned above to see if it’s a clever ruse to draw in sceptics like me or if it’s a valid trial? I could find no link between Dr Siemandi and the manufacturers but then I don’t really know how to look.
matty said,
March 9, 2010 at 10:30 am
Re Transitivity:
Sorry to be picky, but that isn’t quite what transitivity means. For example “equality” is transitive, and if A=B and B=C it doesn’t follow that C=A must be false (quite the opposite). Follow the wiki link if you like. I could give you a definition here, but suffice to say the type of relation for the reasoning you are using would be both antisymmetric and irreflexive (sometimes, strangely simply called asymmetric) which together with transitivity this would allow you to reason as you have. The relation “better than” that you describe is also an example of a strict total order, which suffices for your purpose.
Oh dear, I’ve bored myself.