More worse trials…

August 11th, 2007 by Ben Goldacre in bad science | 23 Comments »

Ben Goldacre
The Guardian
Saturday August 11 2007

Last week Alan Johnson unveiled his new plan of forcing Big Pharma to accept less money for the drugs we buy off them. The major pharmaceutical companies have now responded to this financial threat – very much in the style of right wing comedian Jim Davidson – by threatening to leave the country. It’s a compelling response: Big Pharma only get 2% of their revenue from UK sales, but they spend 10% of their research budget here.

So how, in the style of a 1970s super-hero, can Alan use science to defeat his opponent? The answer is paradoxical: more bad trials. Bear with me, I’m experimenting here.

It costs £500m to bring a drug to market. Much of that is spent on randomised trials. We have made these trials so spectacularly complicated and expensive that they are beyond the reach of governments, academia, and even small companies: only huge international pharmaceutical corporations can afford to run drugs trials now, and so corporations are in complete control of the information. This is bad, as the problems with Vioxx, SSRI’s, and other drugs have shown.

We could whinge about that: or, while they’re all on holiday in Provence with their au pairs, we could castrate the ethics committees, on the grounds – ironically – that they do more harm than good. Imagine there are two drugs called Sixofone and Halfadozen. Nobody knows which one is better for treating Sickitis. The drug companies periodically do trials, but funnily their own drug always seems to come out the best.

In our clinic, the doctor sees 200 patients in a month with Sickitis. If she gives Sixofone or Halfadozen to her patients entirely at random, on a whim, as a reflection of her own indecisiveness, then the regulation is the same as for any prescription: give reasonable verbal information about risks and benefits, thus obtaining informed consent, and so on.

But let’s say the same doctor, in a spirit of enquiry, recognises that there is massive clinical uncertainty – nobody knows which of these drugs is better after all – and she wants to test her hunches, rather than simply act on them: then she’s in trouble.
She could run a speculative little trial, giving 100 patients Sixofone, and compare them with the others on Halfadozen; but she’d be struck off, unless she went through so much red tape that the idea becomes entirely unmanageable. After passing through the endless obstructive and bureaucratic hoops of the ethics committees, from each and every patient she would need to obtain elaborate written consent, and give huge amounts of information.

Remember, nobody knows which is best, and if she wanted to hand over the same drugs, entirely at whim, outside of her “trial”, there’d be no problem. This is a huge double standard, and it costs us knowledge.

If we ever had a scientist in charge of health, instead of tinkering with payments to big pharma, they would do one simple thing: move hell and high water to collect and collate the best and cheapest evidence on healthcare. First you would give huge amounts of money to the Cochrane Collaboration, which collects and collates data independently on all healthcare interventions (and is quietly one of the most subversive organisations ever to be created, because it blows the lid on false commercial claims). Doctors waste money by irrational prescribing decisions, so give them clean, clear, accessible information, and that will stop.

But more than that, we are in a unique position to generate data: we have inherited an extraordinary information resource, in this vast monolith of the NHS. Nobody knows what the best treatment for stroke is, but if we randomised every single new stroke patient in the UK, over one week, into a rough trial, we’d have our answers in a couple of years.

If you have an ethical objection to that, it better be more important than finding out what is best for stroke victims, because the entire health service is like the clinic I described above. Where there is uncertainty, the patients, and their data, are unique national assets: if we can find a way to use them, systematically, efficiently, ethically, as a resource for research and information, then it wouldn’t matter about Big Pharma’s power any more. But as long as knowledge is expensive, multinational corporations will perpetuate their stranglehold on it, and we will be beholden unto them. It’s just an idea for your cornflakes.

If this kind of thing interests you I heartily recommend the genuinely stunning book Testing Treatments by Iain Chalmers, Hazel Thornton, and Imogen Evans.


Or this classic paper on the power of information by Richard Smith and Iain Chalmers:

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

23 Responses

  1. jackpt said,

    August 11, 2007 at 1:00 am

    I’m largely in agreement with this, although I wonder to what degree patients would be targeted by companies. Because I think there is a greater imperative to do so when data is going to be collected. In order to make patients with sickitis more likely to argue with their doctor for a certain drug. Much like some of the medicine advertising in the US. Given they’re already doing that somewhat in the UK by aiding campaigns against NICE decisions I’m not sure whether this is a valid worry, but the financial reasons for bugging patients would increase. I think bearing in mind the drugs were going to be prescribed anyway that your idea is entirely ethical.

  2. Kimpatsu said,

    August 11, 2007 at 2:39 am

    The solution to the threatened big pharma blackmail of disinvestment is to tax them at 100% if they try it.
    If a wife supports her husband financially while he goes to university to study, and he then becomes mega-rich and leaves his wife for a trophy blonde half his age, the courts say that he has to repay his wife for her financial support and long years of loyalty. Similarly, big pharma has gooten rich through tax breaks and other corporate welfare over the years they have been based in the UK. (And let’s not forget “non-domiciled millionaires” for tax purposes.) So, if they want a divorce from the UK, they can have one–but they must not profit from it. Hence the 100% taxation.
    Simple, really: stand up to bullies and give them a bloody nose. They’ll soon learn.

  3. AitchJay said,

    August 11, 2007 at 6:39 am

    Is part of the massive expense of a drug trial to do with insurance? I’m curious what you’d do to cover your arse legally in this system.

    Don’t get me wrong, I like the idea and it’s worth thinking about, but that could be a big problem.

  4. Bob O'H said,

    August 11, 2007 at 7:06 am

    My cornflakes are already marching on Parliament.


  5. csrster said,

    August 11, 2007 at 7:41 am

    If only the post of Chief Scientific Adviser to the Government was directly elected, we could all nominate you Ben.

  6. Primogenitor said,

    August 11, 2007 at 8:13 am

    I thought the biggest cost with clinical trials was proving that they don’t do more harm than good; remember that trial a couple of years ago that hospitalized all the (perfectly healthy) volunteers? Almost all drugs failing before that stage in animal trials, toxicity tests, etc. So how does doing a randomized trial on the entire NHS help with that?

  7. Arthur Dent said,

    August 11, 2007 at 10:03 am

    Bens process would potentially work for those drugs currently available, indeed so-called Phase 4 trials effectively generate much new data on drugs in use in almost the way described.

    However it doesn’t help with the development of a new drug which by it’s nature is not actually in use in the NHS whilst under development but, as pointed out , is going through the ever more complex hurdles of the cliniacal trials process.

    For those not familiar, there are 4 stages to clinical trials: Phase 1 trials use a small number of healthy volunteers to confirm that the pharmacokinetics of the drug in man are those that have been predicted in computer and animal models. The dose used is small compared to the eventual therapeutic dose and these trials (first time in man) are used to demonstrate safety. Phase 2 clinical trials use a much larger population of affected patients and are rune either against a current treatment or against a placebo (depends on the ethics) these are to demonstrate efficacy – does the drug work as expected and is it an improvement on the existing traetment or no treatment at all. Phase 3 trials involve a much larger group or groups of affected patients (sometimes 10,000 or more) and these are to establish appropriate dosing levels and to identify side effects of high frequency. Phase 4 relates to post launch studies.

    Note a me too drug costs just as much to develop as a first in class, although the risk of failure due to efficacy issues may be lower.

    The big Pharma comapanies are indeed the only entities now big enough to do the trials but this is not because they sought this position. Society gets the development costs that it insists on via its regulators and those costs are increasing rapidly as regulators become more and more risk averse.

  8. Evil Monster said,

    August 11, 2007 at 11:45 am

    “The drug companies periodically do trials, but funnily their own drug always seems to come out the best.”

    Where’s the evidence for that? Given the large number of non-inferiority trials that take place, how does this “always” happen. And what’s the mechanism? Is it because they’re fiddling with the data? If so, where’s the evidence?

    “She could run a speculative little trial, giving 100 patients Sixofone, and compare them with the others on Halfadozen”

    An undersized, open label trial without proper control and without proper collection of safety data. Doesn’t seem like a strong argument against ethics committees.

  9. adavida said,

    August 11, 2007 at 2:03 pm

    I suspect many people stuck with the dilemma that the GP in the article has would decide to conduct an audit. Provided that there was no intent to publish the results then most of the red tape, consent forms and ethics committees could be avoided.

    And then after one month (and 100 patients getting each drug) analyse the results and base future prescribing decisions on this non-randomised, non-blinded, non-controlled non-study.

    I suppose this may give an impression as to which drug is better, but I am not sure that it is much better than guessing. A pseudo-evidence based hunch, perhaps?

  10. Tim Worstall said,

    August 11, 2007 at 2:18 pm

    While I like the general idea I’m not overly convinced that it’ll have much effect.
    The US is the largest market, the one with the highest prices, the one where the profits are made. You’d have to convince the FDA of the merits of this type of testing.
    Not, IMHO, going to happen.

  11. Quixotematic said,

    August 11, 2007 at 2:38 pm

    Ben, RECs deal with more than just the CTIMPs. They also review all other studies within the purview of their hosting SHA.

    Much of the committee’s time is taken up with ensuring that the communication between researchers and prospective participants is clear and that their consent is properly informed.

    Committee members, drawn from a cross-section of clinical and non-clinical backgrounds, frequently have insights into issues of participant welfare which the researchers have overlooked.

    I would be less quick to dismiss their (unpaid) work as red tape or bureaucracy.

  12. Geoff said,

    August 11, 2007 at 5:59 pm

    Ben is right about Big Pharma. I quote a letter I sent to the Observer (most of it was published):

    Dear Sir

    Your story (How the drugs giant…4th Dec 2005) is likely to be repeated in new settings in the near future.

    General practitioners’ research work will shortly be forced to change. The Department of Health funding, on which much of the work is based, will be faded out and researchers will be pushed into relying on commercial money.

    The bulk of this money is likely to come from major pharmaceutical companies. In general they will be willing to pay in exchange for the use of GPs patients as a basis for their drug trials. Thus, instead of most research coming from doctors’ assessments of their patients’ needs it will come from drug companies in search of cheap trials fodder.

    This does not simply raise the problems of access to and use of data involved in your story. It means that patients will not be able to trust that their doctors are involving them in research for honourable motives. This will undermine the broader patient doctor relationship.

    The transformation in GP research has led me to resign as a patient from the network of GPs undertaking research in Devon and Cornwall. I cannot be a party to these damaging changes, which were not discussed in the general election. We, as patients, were never asked our opinions about this change. Yet this government claims that patient opinions matter.

    Yours faithfully

  13. Dr Aust said,

    August 11, 2007 at 9:44 pm

    I suppose for those of us slightly suspicious of the PharmaCos, Arthur, the key would be in the phase 2 trials (i.e. funded by PharmaCo vs. pubically funded). One could argue that, because there would not be quite as much “agenda” to push the drug, we might get outcomes less susceptible to funder and publication bias.

    It is a widespread belief that the PharmaCos “load” the key trials for new drugs in myriad ways. One example is, specifically, by trialling the new compound against a placebo rather than against “current best available drug treatment”. I have heard of examples where trials are deliberately run in countries that allow this placebo comparison, rather than in e.g. the US or UK where this kind of trial design is often ethically unacceptable. Other examples of sleight-of-hand would be by heavily promoting and using “surrogate outcome measures”, on which the new drug looks better than older cheaper compounds, but not looking rigorously at more “real-world” clinical outcomes.

    The suspicion is that post a PharmaCo-funded phase 2 trial, the question being asked will be “can we get this through to market and sell it effectively?” just as much as “Is this compounds really more efficacious than the current agents?”

    And if the answer to Q1 is “yes”, then the full machinery will go into phase 3 and push-to-market overdrive, even if the answer to Q2 is “not really”.

  14. Ben Goldacre said,

    August 12, 2007 at 12:42 am


    “The drug companies periodically do trials, but funnily their own drug always seems to come out the best.” Where’s the evidence for that?

    The funniest example is this paper

    A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, et al. (1994) Arch Intern Med 154:157–163.

    In which every NSAID drug is either equal to or better than every other, depending on who funded the study, desdcribed here:

  15. Tim Worstall said,

    August 12, 2007 at 12:06 pm

    #13. No, the FDA requires proof of effectiveness as well as safety.

  16. Ben Goldacre said,

    August 12, 2007 at 12:40 pm

    tim: i guess you mean efficacy, not effectiveness; and you only need a pretty weak demonstration of efficacy to get registered. there’s lots of stff in the BNF which you’d be unlikely to prescribe, but its licensed.

  17. Wonko said,

    August 13, 2007 at 11:43 am

    I would turn Ben’s proposition on its head and ask, why WOULDN’T you want to test the relative effectiveness and side effect profile of the whole series of lisenced me-too drugs. Once a drug is on the market, shouldn’t we be demanding that the NHS routinely monitors its effectiveness?

  18. The Master said,

    August 13, 2007 at 2:31 pm


    the DoH has done a lot of work with Oncology where research is concerned, they created the national cancer research network which has provided dedicated teams in most hospitals for supporting research, also the UKCCG (CCLG) has done a lot of work in respect to childrens cancers. Oncology has CRUK which provides a lot of support for academics and clinicians to run large multicentre trials. interestingly the NCRN has been so successful that big pharma are now negotiating to use the infrastructure in place to run their trials (and thereby save money)

  19. Charles Copeland said,

    August 14, 2007 at 2:26 pm

    Ben Goldacre writes:

    I’m not aware of medical ethicists writing about the ethical impact of ethics committees ….

    Neither am I. Because ethicists would be out of their depth.

    Ethics committees haven’t even got their act together about such scientifically cut-and-dried issues as the morals of the snake oil business. Of course, they write about such topics as ‘the ethics of alternative medicine” — but that sounds like discussing “the ethics of credit card fraud” or “the ethics of shoplifting.” They discredit and debase themselves by taking crap seriously.

    So does anybody REALLY expect these moralistic charlatans to grasp the complexity of ‘patient consent’ issues? They fail to understand that almost EVERY medical intervention is in some way an ‘experiment, whose documented outcome can constitute a contribution to knowledge, if only a marginal one. The patient gives her consent when she walks into the surgery waiting room.

    My impression is that the concepts of ‘consent’ and ‘personal autonomy’ have become Trojan horses via which every conceivable medical quackery can be smuggled into the medical citadel on the grounds that ‘that’s the treatment the ‘sovereign’ patient wanted.’

    So no — that’ll be the day when the ethicists start writing about their ‘ethical impact’.

  20. RS said,

    August 14, 2007 at 5:37 pm

    Re: Rochon et al, it isn’t quite as cut and dried as Ben suggests, e.g. Barden et al 2006

    “There was no evidence that industry sponsorship affected the outcome of clinical trials in acute pain, and only limited evidence for any difference in migraine trials…The obvious comparison was between trials that were industry-sponsored and those that were not. Only two trials declared a funding source that was definitely non-profit compared with 125 trials that were stated as being industry-sponsored. The comparison was therefore between industry-sponsored trials and trials that predominantly failed to give any information about funding, severely limiting this approach. The only significant difference between industry-sponsored trials and the rest was in the direction of a worse result (higher NNT) with industry sponsorship for ibuprofen 400 mg (Table 1). There were simply too few trials that declared a non-profit sponsorship to permit confidence. Here there was a definite absence of evidence…Our pre-specified alternative approach (Fig. 1) was to recognise that any bias that might occur is going to be specific rather than generic. We tested this hypothesis by examining efficacy measures according to whether, in industry-sponsored trials only, a drug was the prime subject of investigation, i.e., the test drug, or used as a comparator. The anticipated bias was that the test drug would perform better in trials sponsored by its manufacturer and worse as a comparator in trials funded by a competitor. Here, there was sufficient evidence to make a valid comparison, but there were no differences…even in recent trials of rofecoxib where the interests of one major pharmaceutical company clearly conflicted with those of another…There was evidence of absence of industry bias rather than absence of evidence.”

  21. Martin T said,

    August 15, 2007 at 2:10 pm

    re comment 26 (and some others) As someone who sits on an ethics committee (albeit one concerned with healthcare research involving students as distinct from medical research) and who teaches to a wide range of students on healthcare law and ethics, I take great personal and professional exception to the comment about ‘moralistic charlatans’.
    One of reasons for the exsistence of ethics committees was as a result of some research which harmed subjects for little benefit and as a result an understandable loss of trust in the ability of the medical establishment to regulate research without someone ensuring that research subjects were not put at undue risk.
    As for the comments about patient autonomy and consent, I am amazed that anyone could think ‘The patient gives her consent when she walks into the surgery waiting room.’ represents either the current legal or ethical postion. I would suggest that anyone reading those comments may wish to consult an appropriate up to date text or at least consult to DoH website, (

  22. RS said,

    August 15, 2007 at 5:53 pm

    Martin T – if there really is equipoise then ethically speaking I can completely see Charles Copeland’s point that any additional consent requirements smack of fetishisation of patient autonomy.

  23. Cedders said,

    August 16, 2007 at 10:39 am

    Certainly there should be more small-scale trials of existing drugs initiated by clinicians, even off-label uses for non-life-threatening conditions. Some legal indemnity would be needed (really just a fine-tuning of the law), but I don’t think it’s patient consent or ethics committee issue – one-person doctoral research has no problem with those.

    Drug companies have a duty to make a profit for their shareholders, and only about thirteen years of patent life to make it. It’s no wonder that the vast margins they make on novel compounds are spent on copycat drugs, finding trivial ways of extending patent life, and numerous ways of biasing the scientific process from selective publication and re-publication and conferences to public relations and hiring pet GPs to push the company line. It’s not just bad science, it’s an enormously wasteful system, and the money which originally came from the taxpayer via the poor patient, would be far better spent in pure university research, and doing Cochrane-style analyses of the (non-randomized) GP Research Database and small trials where the data is in the public domain.

    As for the bullying reaction to anything that threatens the status quo, I don’t believe it. Drug companies have too much invested in the UK to pull out, and it wouldn’t make any financial sense. Their fear is once it is seen to be successful ni one country, a more public and open research mdoel will spread to Europe and worldwide. Despite the corporate pressure, one has to appeal to the intelligence, public-spiritedness and better nature of individual drug company directors, many of whom originally got into the business with the ideal of helping people.