Beau Funnel – updated with list of academic journals which actively encourage submission of negative results

March 8th, 2008 by Ben Goldacre in bad science, regulating research | 21 Comments »

Ben Goldacre
The Guardian,
Saturday March 8 2008

It was gratifying to see – after only a one-week delay – the government announcing that they would follow my suggestion on the comment pages last week, and demand that drug companies disclose all trial data, to make sure they’re not hiding anything. This has been pegged to the issue of undisclosed side-effects of antidepressants, because a drug company hiding side-effects is intuitively evil.

This is unfortunate because – as I have repeatedly argued – much more worrying is the tendency to only publish results which show your drug performing well, and to leave the less flattering trials in a desk drawer. This happens much more commonly, it makes drugs look better than they are, it wastes money, it exposes people to risks of unnecessary side-effects, and it leaves doctors prescribing on inaccurate information.

But how can you tell if the research literature on a given subject has been rigged? It’s a tricky problem, because you’re chasing evidence for the existence of trials you cannot see. One option is to use mathematical tools, and something called a funnel plot, one of the cleverest ideas of the last century. It’s so clever that you might need to concentrate for the next bit.

Let’s imagine that there are 30 trials on a given drug. Some are big, and more accurate. Some are small and less accurate, with more random noise. You’d expect that the big, accurate trials should all cluster together around the true finding, all giving similar results for the efficacy of a drug. Meanwhile the smaller, rubbish trials – because they are less accurate measures of the drugs efficacy – will be scattered about randomly, some showing the treatment to be better than the good big trials indicate, some showing that it is worse.

You could then plot all your trials on a graph, one dot for each trial. On the x-axis, left to right, is “how good the drug was shown to be by this trial” and on the y-axis, “how methodologically sound and large the trial was”. If there is no publication bias, you should get a triangle shape: at the top of your graph, you will see all your good-quality, accurate trials, clustered together around the true answer. At the bottom of the graph, you will see a broad smear of results, the poor quality trials showing random variation.

But if there is publication bias, you will see a distorted triangle: the small, poor-quality trials at the bottom will be smeared over to the right, because small trials with unwelcome results are much more likely to be overlooked, and dumped in desk drawers, than huge multicentre collaborative studies involving dozens of academics and tens of thousands of participants, which are almost definitely going to get published. If you get a distorted triangle, you know there are some interesting negative trials missing.

This is a funnel plot demonstrating possible publication bias in a set of studies looking at… publication bias. Seriously.

This happens repeatedly, in too many fields to list, and it doesn’t just happen because of big pharma evil. If you’re an academic, and you get a negative result, you’re less likely to get round to publishing it, because it’s not going to get in a big journal, so it’s not going to buff your department’s “research assessment exercise” score for this year, nobody’s going to invite you to give lectures about it, the whole thing feels like a disappointing waste of time.

And even though you know in your heart that a negative finding is still an interesting piece of evidence, there are undergraduates that need teaching and you hate doing that and the references for that other paper need reformatting before submission to a third journal and before you know it five years have passed and nobody’s even mentioned the negative finding at the departmental meeting since the last prof retired so you can probably get away with leaving it for another year at least and possibly even ideally until you die.

I’ve said it before: all trials should be registered before they start, no trial – by anybody – should be passed by any ethics committee without a firm commitment to publish. No exceptions, because bad data costs lives.

· Please send your bad science to

Publish your negative results here:

PLoS Clinical Trials was launched specifically with the aim of accepting for publication all properly reported, correctly conducted randomized trials, irrespective of outcome. it’s part of PLOSOne now

Trials is big on trial methodology, but also publishes results, whether positive or negative, and is edited by Doug Altman, king of propriety, god of Stats, and overlord of the CONSORT guidelines. Like any decent journal they will only accept your trial if it has been registered.

Try also BMC Research Notes:
it’s new, not just trial data, a quiet hole to get your results onto the record.

and there’s the old Journal of Negative Results in BioMedicine

Oh, and

In case you missed it, here is the letter from Kent Woods of the MHRA where he announces the new legislation and, if I’m not mistaken, calls the chief exec of GSK an “arse”, to his face:

“Such a course of action should be unnecessary in an industry which relies so heavily on public trust and aspires to high ethical standards.

“I would have thought it self-evident that such information should be made available promptly to the regulator in order that action can be taken to protect public health. However, that moral responsibility now needs to be to be insisted upon by the unambiguous force of law.”

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

21 Responses

  1. Nicolo said,

    March 8, 2008 at 4:06 am

    If we’re looking into a protocol that will help us flag companies potentially concealing unfavorable studies, eyeballing scatterplots won’t be an acceptably reliable method of detecting ‘distorted triangles’. Are there some sort of statistical methods we could use?

    And even if there were, this only addresses the problem of entire studies being concealed. What about fudged data? Maybe it’s too conspiracy theory, but pharmaceuticals could anticipate what the scatterplot of their studies’ sample sizes vs. scatterplots should look like and manipulate their data accordingly.

  2. mjs said,

    March 8, 2008 at 5:24 am

    no [clinical] trial – by anybody – should be passed by any ethics committee without a firm commitment to publish.

    sorry to ask this again, but be published by whom?

  3. stever said,

    March 8, 2008 at 10:02 am

    It was badscience wot won it

  4. RS said,

    March 8, 2008 at 11:05 am

    I’m not convinced that a funnel plot is informative for studies of publication bias – or indeed any meta-analytic technique – because we already have reasons a priori to believe that there is publication bias – not because people are hiding small negative studies in their desk draw – but because people tend to do studies of publication bias in fields where they think there might actually be some bias.

    A similar point was made about the ‘systematic review’ of meta-analyses of the relevance of animal studies to human health by Pound et al – there were already a priori reasons to believe that people only did studies of the relationship between basic animal studies and human trials when they believed there was a difference.

  5. RS said,

    March 8, 2008 at 4:01 pm

    Therefore the funnely plot is assuming a single population effect size which each study is sampling, which is not a tenable assumption in this case – funnily enough this a point made in the Kirsch et al paper on anti-depressants.

  6. thom said,

    March 8, 2008 at 6:04 pm

    A funnel plot is a nice idea, but not the best way to detect publication bias. It is a nice exploratory tool and should be viewed as such rather than a diagnostic test. For a start it detect assymmetries in the plot and can’t tell you what causes those. Publication bias is one possible cause, as is a non-fixed effect, or indeed a genuine relation between effect size and study size (e.g., because small studies stick to the protocols better). I’m guessing Ben knows this because of the term “mathematical tools”. (Also because of his advocacy for registration of all trials.)

  7. mjrobbins said,

    March 8, 2008 at 8:29 pm

    Incidentally, weird I know, but has there ever been a case of publication bias going against a new drug? I can think of a few situations where it might happen…

  8. tom-p said,

    March 10, 2008 at 1:34 pm

    Camp Freddie,

    They already have to at the time of submission, but not for post-marketing, therefore this should include at PSUR time (every 6 months to 5 years, depending on the age of the product).

  9. emilypk said,

    March 10, 2008 at 1:42 pm

    “And even though you know in your heart that a negative finding is still an interesting piece of evidence, there are undergraduates that need teaching…”

    I don’t think you can put it predominantly on the researchers lack of enthusiasm as you have. I am sitting on a negative result from 1997 that I think is probably the most important work I every did. I included positive controls and extensive testing to assure it was a real negative. It was flatly ejected by 3 different journals. I am preparing to rewrite it for the fourth time. It is depressing how disinterested the entire research machinery really is in the reality of important null hypotheses.

  10. K9 said,

    March 10, 2008 at 3:29 pm

    As a biomedical researcher, I have to publish my work on a regular basis. Furthermore, there is pressure to submit work to high ranking journals. The argument that is used is that quality of a publication counts much more than the quantity of papers one publishes. Now, whilst I may not be an advocate of this approach, my kids still need food and shoes. The majority of international science works like this meaning there is, in effect, a permanent competition to get into the better journals and not necessarily those most suited to your line of work. Now, the kind of publications you propose would not get into high impact journals and hence would essentially just be a repository for dredged up facts. I cannot think of a commercial publisher (they all are commercial now, including those owned by respected scientific societies) who would be able to afford to publish this stuff.

    I also have my doubts about funnel analysis, it seems to make too many presumptions about why people do experiments and how they get designed and implemented.

  11. Geeb said,

    March 12, 2008 at 8:01 pm

    Camp Freddie said,

    “Who is going to publish studies that just confirm a null hypothesis?
    The readers of ‘Nature’ don’t want to see hundreds of reports that just say “Novel drug-candidate #1689 doesn’t work”.”

    It really worries me that people don’t find these results interesting. If it’s just a signal-to-noise ratio problem, there being oodles of negative results for each “interesting” positive, they could just stick them in an appendix or something.

    Anyone fancy starting a free-access online journal to act as a repository for high-quality research with negative findings? Surely there are enough people who understand the importance of these results that they’d be willing to do a bit of peer review…

  12. Despard said,

    March 13, 2008 at 11:09 am

    I thought there were already several Journals of Null Results, for various fields.

  13. Andrew Clegg said,

    March 14, 2008 at 9:44 pm

    Actually there is a sense in which it is highly important where the data is published, which everyone’s missing.

    From the perspective of evidence-based medicine, the gold standard of availability for a result is publication in a journal that’s indexed by a standard literature database such as MEDLINE.

    This means that a systematic literature review for the purposes of setting policy or developing clinical guidelines will retrieve the data by standard search methods. It doesn’t actually matter whether it’s published in Nature or the Journal of Negative Results in BioMedicine — PubMed and Ovid searches are blind to such distinctions.

    If it’s available from the researchers (freely or under duress), but doesn’t actually appear in the databases, then it’s ‘under the radar’ to systematic reviews — not invisible but unlikely to be detected unless you know it’s there. This is a significant risk.


  14. Ben Goldacre said,

    March 15, 2008 at 2:17 am

    hi, sorry, not been around in the comments the past few days.

    tom-p: as you know, because we’ve discussed it in our email chats, EUdraCT to which you refer is an oddly secretive setup. show me the list of trials and protocols. show me even the old ones. prove to me that you chase them up. I don’t think trials being placed on a secret register is enough. the SSRI story demonstrates that fairly clearly, those studies were declared to the FDA but it took investigative minded academics to chase them up, using the freedom of information act.

    the respectable journals for negative results which a few of you have asked for do in fact already exist.

    as you know i’m committed to the open access movement.

    PLoS Clinical Trials was launched specifically with the aim of accepting for publication all properly reported, correctly conducted randomized trials, irrespective of outcome. it’s part of PLOSOne now

    is big on trial methodology, but also publishes results, whether positive or negative, and is edited by Doug Altman, king of propriety, god of Stats, and overlord of the CONSORT guidelines. Like any decent journal they will only accept your trial if it has been registered.

    Try also BMC Research Notes:
    it’s new, not just trial data, a quiet hole to get your results onto the record.

    and there’s the old Journal of Negative Results in BioMedicine

    since this article has had a lot of traffic i’ll also post these as an addendum to the main post.

  15. Robert Carnegie said,

    March 24, 2008 at 2:05 am

    It seems a pity that you have to have your hypothesis ready before you go collecting data. If your hypothesis is not quite right then you do the work and you’re stuffed. It seems to me that with data you should be able to see what the hypothesis ought to be. It!s that naive? Sherlock Holmes thought you shouldn’t theorise without data but he also used drugs, played with guns, and thought the sun orbited the earth, so perhaps he isn’t the best witness. Oh, and he didn’t exist.

  16. MattJH said,

    March 26, 2008 at 3:04 pm

    Fergus, you will probably be interested in the open science movement – there are two particularly pertinent threads, one being open data and the other being open notebook science.

    Open data means giving other researchers access to your raw data. This isn’t as common as it should be.

    Open notebook science (a term coined by Jean-Claude Bradley) means recording your experiments and results online and in the open, so anyone can see your research as it proceeds. This isn’t quite the same as a research registry. A few scientists are now doing open notebook science; Heather Piwowar is a good example, Cameron Neylon is another.

  17. Robert Carnegie said,

    March 29, 2008 at 4:52 pm

    Ferguskane: sometimes it’s between research paper and, say, newspaper report that an epidemiological “fishing expedition” (trawling) is turned into a “scientific discovery” that blondeness is a cause of left-handedness, with the proposition that it’s because more light shines in your eyes if you have fair hair. That is, the actual paper would say that theyre are unusually many left-handed blonde people (which I think I have made up), but otherwise nothing much that’s interesting, and incorporate speculation about credible mechanisms, but if it turns up on your breakfast table then those details have become facts.

  18. ferguskane said,

    April 6, 2008 at 12:46 am

    RC… you’re right of course. It’s definitely important not to get too cynical, which includes not assuming that poor secondary reporting equals poor science.

    So when I’m feeling too cynical, my favourite paper:

    Oh… and one specially for you:
    ‘Associations of handedness with hair color and learning disabilities.’ Neuropsychologia. 1987;25(1B):269-76.

    Actually, like the sound of this as well:
    ‘Dark hair and light eyes in female college students: a potential biologic marker for liability to psychopathology’

    MattJH: Yes, I’m definitely interested, and it’s definitely a good (great/amazing) movement. I’ll have a look at some of these open notebooks.

    Open data would not prevent the data before hypothesis problem, but open notebook science definitely would.

    I’d like to propose another advantage of a science registry- it would allow scientists to have an idea of who else was doing the same research as them.

    This would allow researchers to collaborate from the outset, allowing, among other things: mega-analyses of matched datasets rather than less powerful meta-analyses of disparate data. In my field there are many many studies that are nearly, but not quite the same, which means that they are difficult to compare and difficult to combine.

  19. diohdan said,

    May 27, 2008 at 2:45 am

    Recently published on

    The Human Injury of Lost Objectivity

    If I were to rate the corruptive tactics performed by big pharmaceutical companies, the intentional corruption of implementing fabricated and unreliable results of clinical trials by pharmaceutical companies by hired third parties who manipulate these trials they sponsor because of their power to control others involved in such trials that is largely absent of regulation would be at the top of the list, and likely the most damaging to the requirement of authenticity and, more importantly, assuring the safety of the public health, as I understand that this does in fact occur.

    Decades ago, clinical trials were conducted at academic settings that focused on the acquisition of knowledge and the completely objective discoveries of meds. Then, in 1980, the Bayh-Dole Act was created, which allowed for such places to profit off of their discoveries that were performed for pharmaceutical companies in the past. This resulted in the creation of for-profit research trial sites, called Contract Research Organizations, which is often composed of community research sites with questionable investigators possibly void of necessary research experience or quality regarding their research purpose and ability. Since they are for- profit, with some CROs making billions of dollars a year. The trials conducted at such places are sponsored by pharmaceutical companies that control and manipulate all aspects of the trial being conducted involving their med being studied in the trial. This coercion is done by various methods of deception in subtle and tacit methods. As a result, research in this manner has been transformed into a method of marketing, which includes altered results of the trial to favor the sponsor’s med. Their activities are absent of true or applied regulation, and therefore have the autonomy to create whatever they want to benefit what may be a collusive relationship between the site and the sponsor.

    Further disturbing is that once the creation of the trials is completed, they are then written by ghostwriters often, although no one seems to know how often. These people are not identified and acknowledged by the sponsor, and may not be trained in clinical research overall, as they are simply freelance writers, as one does not need research training or certification in order to perform this function. Rarely do trial ghostwriters question their instructions about their assignment, which is clearly deceptive and undocumented by the sponsor. Also, these hired mystery writers are known to make about 100 grand a year. This activity removes accountability and authenticity of the possibly fabricated clinical trial even further. The corruptive act is finally completed by the sponsor hiring an author to be placed on the trial that likely had no involvement with the trial, and, along with others, was paid by the sponsor for doing this deceptive act.

    To have the trial published, the sponsor pays a journal to do this in various ways, I understand, such as purchasing thousands of reprints of their study from the journal. Again, how often this process is performed is unknown, yet frequent enough to create hundreds of such false writers and research sites to support the pharmaceutical industry. So benefits of meds studied in such a malicious way potentially can harm patients and their treatment options and safety risks. The purchased reprints are distributed to the sponsor’s sales force to share the content with prescribers which may lack validity.
    Such misconduct discussed so far impedes research and the scientific method with frightening ethical and harmful concerns, if in fact true based on reports by others. If so, our health care treatment with meds is now undetermined in large part with such corruptive situations, as well as the possible absence of objectivity that has been intentionally eliminated. Trust in the scientific method in this type of activity illustrated in this article is absent. More now than ever, meds are removed from the market are given black box warnings. Now I understand why this may be occurring.
    Transparency and disclosure needs to happen with the pharmaceutical industry for reasons such as this as well as many others, in order to correct what we have historically relied upon for conclusive proof, which is the scientific method. More importantly, research should not be conducted in a way that the sponsor can interfere in such ways described in this article, requiring independent sites with no involvement with the drug maker. And clearly, regulation has to be enforced not selectively, but in a complete fashion regarding such matters. Public awareness would be a catalyst for this to occur, after initially experiencing a state of total disbelief that such operations actually are conducted by such people, of course. We can no longer be dependent on others for our optimal health. Knowledge is power, and is also possibly a lifesaver.

    “Ethics and Science need to shake hands.” ……. Richard Cabot

    Dan Abshear

  20. diohdan said,

    June 4, 2008 at 2:05 am

    Current Depression Medications: Do The Benefits Outweigh the Harm?

    Presently, for the treatment of depression and other what some claim are mental disorders, as they are questionable, selective serotonin reuptake inhibitors are the drugs of choice by most prescribers. Such meds, meds that affect the mind, are called psychotropic medications. SSRIs also include a few meds in this class with the addition of a norepinephrine uptake inhibitor added to the SSRI, and these are referred to SNRI medications. Examples of SNRIs are Cymbalta and Effexor. Some consider these classes of meds a next generation after benzodiazepines, as there are similarities regarding their intake by others, yet the mechanisms of action are clearly different, but not their continued use and popularity by others.
    Some Definitions:
    Serotonin is a neurotransmitter thought to be associated with mood. The hypothesis was first suggested in the mid 1960s that this neurotransmitter may play a role in moods and emotions in humans. Yet to this day, the serotonin correlation with such behavioral and mental conditions is only theoretical. In fact, the psychiatrist’s bible, which is the DSM, states that the definite etiology of depression remains a mystery and is unknown. So a chemical imbalance in the brain is not proven to be the cause of mood disorders, it is only suspected with limited scientific evidence. In fact, diagnosing diseases such as depression is based on subjective assessment only, as interpreted by the prescriber, so one could question the accuracy of such diagnoses.
    Norepinephrine is a stress hormone, which many believe help those who have such mood disorders as depression. Basically, with the theory that by adding this hormone, the SSRI will be more efficacious for a patient prescribed such a med.
    And depression is only one of those mood disorders that may exist, yet possibly the most devastating one. An accurate diagnosis of these mood conditions lack complete accuracy, as they can only be defined conceptually, so the diagnosis is dependent on subjective criteria, such as questionnaires. There is no objective diagnostic testing for depression. Yet the diagnosis of depression in patients has increased quite a bit over the decades. Also, few would argue that depression does not exist in other people. Yet, one may contemplate, actually how many other people are really depressed?
    Several decades ago, less than 1 percent of the U.S. populations were thought to have depression. Today, it is believed that about 10 percent of the populations have depression at some time in their lives. Why this great increase in the growth of this condition remains unknown and is subject to speculation. What is known is that the psychiatry specialty is the one specialty most paid to by certain pharmaceutical companies for ultimately and eventual support of their psychotropic meds, as this industry clearly desires market growth of these products. Regardless, SSRIs and SRNIs are the preferred treatment methods if depression or other mood disorders are suspected by a health care provider. Yet these meds discussed clearly are not the only treatments, medicinally or otherwise, for depression and other related disease states.
    Over 30 million scripts of these types of meds are written annually, and the franchise is around 20 billion dollars a year, with some of the meds costing over 3 dollars per tablet. There are about ten different SSRI/SRNI meds available, many of which are now generic, yet essentially, they appear to be similar in regards to their efficacy and adverse events. The newest one, a SNRI called Pristiq, was approved in 2008, and is believed to being promoted for treatment for menopause. The first one of these SSRI meds was Prozac, which was available in 1988, and the drug was greatly praised for its ability to transform the lives of those who consumed this medication in the years that followed. Some termed Prozac, ‘the happy pill’. In addition, as the years went by and more drugs in this class became available, Prozac was the one of preference for many doctors for children. A favorable book was published specifically regarding this medication soon after it became so popular with others.
    Furthermore, these meds have received additional indications besides depression for some really questionable conditions, such as social phobia and premenstrual syndrome. With the latter, I find it hard to believe that a natural female experience can be considered a treatable disease. Social phobia is a personality trait, in my opinion, which has been called shyness or perhaps a term coined by Dr. Carl Jung, which is introversion, so this probably should not be labeled a treatable disease as well. There are other indications for certain behavioral manifestations as well with the different SSRIs or SRNIs. So the market continues to grow with these meds. Yet, it is believed that these meds are effective in only about half of those who take them, so they are not going to be beneficial for those suspected of having certain medical illnesses treated by such meds. The makers of such meds seemed to have created such conditions besides depression for additional utilization of these types of medications, and are active and have been active in forming symbiotic relationships with related disease- specific support groups, such as providing financial support for screenings for the indicated conditions of their meds- screening of children and adolescents in particular, I understand, and as a layperson, I consider such activities dangerous and inappropriate for several reasons.
    Danger and concerns by others primarily involves the adverse effects associated with these types of meds, which include suicidal thoughts and actions, violence, including acts of homicide, and aggression, among others, and the makers of such drugs are suspected to have known about these effects and did not share them with the public in a timely and critical manner. While most SSRIs and SNRIs are approved for use in adults only, prescribing these meds to children and adolescents has drawn the most attention and debate with others, such as those in the medical profession as well as citizen watchdog groups. The reasons for this attention are due to the potential off-label use of these meds in this population, yet what may be most shocking is the fact that some of the makers of these meds did not release clinical study information about the risks of suicide as well as the other adverse events related to such populations, including the decreased efficacy of SSRIs in general, which is believed to be less than 10 percent more effective than a placebo. Paxil caught the attention of the government regarding this issue of data suppression some time ago, this hiding such important information- Elliot Spitzer specifically, as I recall.
    And there are very serious questions about the use of SSRIs in children and adolescents regarding the effects of these meds on them. For example, do the SSRIs correct or create brain states considered not within normal limits, which in effect could cause harm rather than benefit? Are adolescents really depressed, or just experiencing what was once considered normal teenage angst? Do SSRIs have an effect on the brain development and their identity of such young people? Do adolescents in particular become dangerous or bizarre due to SSRIs interfering with the myelination occurring in their still developing brains? No one seems to know the correct answer to such questions, yet the danger associated with the use of SSRIs does in fact exist. It is observed in some who take such meds, but not all who take these meds. Yet health care providers possibly should be much more aware of these possibilities
    Finally, if SSRIs are discontinued, immediately in particular instead of a gradual discontinuation, withdrawals are believed to be quite brutal, and may be a catalyst for suicide in itself, as not only are these meds habit forming, but discontinuing these meds, I understand, leaves the brain in a state of neurochemical instability, as the neurons are recalibrating upon discontinuation of the SSRI that altered the brain of the consumer of this type of med. This occurs to some degree with any psychotropic med, yet the withdrawals can reach a state of danger for the victim in some classes of meds such as SSRIs, it is believed.
    SSRIs and SRNIs have been claimed by doctors and patients to be extremely beneficial for the patient’s well -being regarding the patient’s mental issues where these types of meds are used, yet the risk factors associated with this class of medications may outweigh any perceived benefit for the patient taking such a drug. Considering the lack of efficacy that has been demonstrated objectively, along with the deadly adverse events with these meds only recently brought to the attention of others, other treatment options should probably be considered, but that is up to the discretion of the prescriber.
    “I use to care, but now I take a pill for that.” — Author unknown
    Dan Abshear

  21. laptopbatteriesshop said,

    April 3, 2010 at 4:27 am

    We don’t just offer the typical cheap laptop batteries that you may find from other e-retailers, we pride ourselves with providing our customers with the most cost effective solution towards laptop battery replacement without sacrificing quality. All laptop batteries and AC adapters will meet or exceed OEM (Original Equipment Manufacturer) specifications. Every that is available on our website is also guaranteed to look, fit, and perform just like your original laptop battery (and usually better). The only real difference is the price. You don’t have to pay a ridiculously high price just for a laptop battery. We produce over 95% of the laptop batteries, laptop chargers, laptop chargers and other products that we sell. There’s also no middle man for us to pay. Lower cost for us translates into lower prices for you.