++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

31 Responses

1. BobP said,

   June 10, 2006 at 10:21 am

   Ben, a huge crystal energy generator has been aimed at your heart for several weeks now but you didn’t pick up the vibes.

2. Ben Goldacre said,

   June 10, 2006 at 10:23 am

   That’s strange, something is penetrating my unconscious, suffusing me with a rich golden light, telling me to write more silly articles.

3. Ben Goldacre said,

   June 10, 2006 at 10:36 am

   Sadly, it hasn’t.

   www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=15877763&query_hl=3&itool=pubmed_docsum

4. profnick said,

   June 10, 2006 at 10:43 am

   Ben,
   This is a very interesting point which I guess I hadn’t really thought about before. In 25 years of research I accumulated hundreds of files with experimental data in. The vast majority were negative results and, given that they weren’t the results of drug trials, it’s not surprising to me that no journal would consider publishing my “failed” experiments, so it never crossed my mind to even try. Thus all my published papers were “interesting” (at least to someone one hopes) results which disproved a null hypothesis. So to get back to your point, in every research field there must be a vast proportion of unpublished to published data. Maybe quite a lot of interesting stuff is hidden in all that “noise”?

5. Ben Goldacre said,

   June 10, 2006 at 10:46 am

   profnick: i’m sure that’s true, and online low threshold publication routes are useful, although they still demand that the authors write up the methodology and put a fair bit of work in.

6. Bob O'H said,

   June 10, 2006 at 10:46 am

   Some of us are trying to do something about publication bias. There are now several journals dedicated to providing an outliet for negative results:

   i) Journal of Negative Results in Biomedicine (www.jnrbm.com/)
   ii) Journal of Negative Results in speech and Audio Sciences (journal.speech.cs.cmu.edu/)
   iii) Forum for Negative Results [computer science] (page.mi.fu-berlin.de/~prechelt/fnr/)
   iv) Journal of Negative Observations in Genetic Oncology (www.path.jhu.edu/NOGO/)
   v) Journal of Articles in Support of the Null Hypothesis (www.jasnh.com/)
   vi) The Journal of Spurious Correlations (www.jspurc.org/main2.htm)
   vii) Journal of Negative Results – EEB (www.jnr-eeb.org)
   (Statement of conflicting interests: I’m involved with no. vii)

   I think from the academic side, things will only change if the value of publishing negative results is raised: an obvious way to do this is to submit to these journals, and help them raise their profiles.

   This won’t solve the problem of companies not publishing, of course, but a change in atmosphere towards negative results might help.

   Bob

7. BobP said,

   June 10, 2006 at 11:21 am

   Ben –
   The Department of National Statistics disagrees with your figure of 96 cases in 1996.
   See –
   www.statistics.gov.uk/STATBASE/Expodata/Spreadsheets/D3983.xls
   I guess you are referring to the Health Protection Agency database here?
   www.hpa.org.uk/infections/topics_az/measles/data_mmr_confirmed.htm

8. Ben Goldacre said,

   June 10, 2006 at 11:40 am

   Bob: no.

   The top of your links is for doctors’ notifications where they suspect someone might have mumps, that’s a completely different bit of data.

   The bottom link is for confirmed cases of mumps, and that is the appropriate figure to quote as “mumps cases”.

9. Robert Carnegie said,

   June 10, 2006 at 11:52 am

   On trials, simply registering trials may not be sufficient. Let’s assume for the sake of argument that all drugs companies will do their level best to make their trial results look favourable, and running multiple trials on the same drug is both an effective way to do this, and, in fact, necessary good practice. Well, certainly there are well conducted trials and poorly conducted trials; that’s inevitable, too, apparently. So if a trial is not producing the result that the paymasters want, then they can probably arrange to shift it into the “poorly conducted” category, introduce obvious mistakes, get the guy running the experiment hooked on drink or drugs…

   On the other hand… obviously if it’s possible to estimate beforehand whether a trial will produce a favourable result or not, then drugs companies will prefer to fund the favourable ones, and that isn’t particularly sinister, only commercial – they are in the business of taking our money, principally, it’s what they get up in the morning for. Keeping us alive is incidental.

   Anyway… given that sponsored trials evidently are only part of the problem, how to solve the rest of it? Well, every university scientist wants to be published, whether the results are positive or negative. If you are published then you get tangible benefits. I worked as computer technician in the Faculty of Law at Glasgow University and they asked me if I could see my way to publishing anything.

   So it’s the journals that you need to work on – to establish a protocol so that they review and publish the less interesting work. Perhaps when a journal publishes something exciting, they should always take on responsibility for publishing the less exciting followups? That possibly would just create another bias, but maybe a less harmful one?

10. Bob O'H said,

    June 10, 2006 at 11:59 am

    John A:

    The problem with self-publication is that it is not permenent, and is also not peer reviewed. In other words, you could put any old rubbish up. Whilst peer review is far from perfect, it does provide some quality control.

    On the free release of data, as a statistician I think that’s a great idea in principle: it means I could use other peoples’ data to find out some really boring stuff. There are going to be problems with people being possessive: I’ve been looking at data sets that have taken a couple of decades to collect, and I appreciate that it would be difficult for my collaborators to put these up on the web, and see them used to do what you’ve been wanting to do, but have never had the time. It’s difficulot, and I’m not sure what the solution is. Perhaps if enough people put their data up on the web, it’ll become the norm.

    In summary: bloody social realities gets in the way again.

    Bob

11. Ben Goldacre said,

    June 10, 2006 at 12:00 pm

    Other useful interventions would be to have negative result publications:

    # listed on CV’s,

    # considered in academic job applications, and

    # counted, perhaps specially and separately, in RAE’s.

12. John A said,

    June 10, 2006 at 4:41 pm

    Bob: In other words, you could put any old rubbish up.

    True. But speaking specifically about negative results, I’m not sure how much of a problem a lack of peer review is.

    More generally, one way around the permanency/peer review is to have a central site where all researchers submit stuff and to allow comments to be sent. This would allow researchers to discuss issues about the research in the same place it is published. Some sort of Wiki set up may be useful. In my mind this would be a good place for poeple to make points reviewers may have missed – I’m sure we’ve all read papers with glaring holes reviewers have missed…

13. Delster said,

    June 10, 2006 at 5:18 pm

    one additional advantage to publishing negative results would be a reduction in time spent re-doing that which has already been done. After all if somebody has an idea for a research project it would be nice to be able to find out if it’s already been tried.

    Obviously if a new technique / drug / approach is devised then re trials could be worth it.

14. Teek said,

    June 11, 2006 at 11:00 am

    Delster: “one additional advantage to publishing negative results would be a reduction in time spent re-doing that which has already been done. After all if somebody has an idea for a research project it would be nice to be able to find out if it’s already been tried.”

    very, very good point. from my own (perhaps limited) experience ‘negative results’ often arise because of some biological reason, i.e. compound/protein/molecule X doesn’t have a chance in hell of treating condition Y because substance Z stops it from doing so, something that maybe laboratory A has already found out but lab B spends 10k and six months finding out for itself. if lab A publishes it’s findings, even tho it’s kinda negative (i.e. X does NOT treat Y), B wouldn’t have gone to the effort.

    this brings us onto whether there are two types of negative results – Type I, where for instance a drug/treatment/process has no therapeutic effect where it was expected to do so (like my convoluted example,above ), and Type II, where instead of/as well as a therapeutic effect, the drug/treatment/whatever produces negative consequences and/or deleterious side effects. currently journals do publish Type II negative results, because they are useful in warning others of unwanted consequences to potential drugs etc.

    Type I needs more attention as most of us here seem to be saying, perhaps in the kind of journals that specifically set out to pubilsh negative (or rather ‘not positive) results – like those in Bob O’H’s list)

15. b said,

    June 11, 2006 at 4:59 pm

    One a positive result is printed, null results about the same hypothesis are no longer uninteresting. At the least, they are sensitivity tests that indicate that minor changes in test design will reverse the results, and in some cases they can catch dumb one-in-a-hundred luck with the p-values.

    In a healthy system, twenty people do a study, and only one of them finds that the variable in question is significant with p>0.05. That one publishes, then the other nineteen publish articles building upon the pioneering work of the guy who got lucky, then a metastudy is written finding that the preponderance of evidence indicates against rejecting the null. Since you’re no stranger to the metastudies, I imagine you could give a few examples of a story like this yourself.

    It’s a slow and potentially inefficient system, and the positive result always gets to go first, and people who don’t know how to work the “show works that cited this article” button on their lit database think that the positive result is unrefuted, but in the long run it seems to generally work OK.

16. FlammableFlower said,

    June 11, 2006 at 9:20 pm

    Negative results would be bloody helpful to researchers. As a synthetic organic chemist you spend a lot of time trying to make X from Y, and even though it looks easy on paper, it goes and sticks two figners up at you. Occasionally you see an angry footnote or offhand comment in an article. To see a paper on the lines of “we tried this, it didn’t work so if you want to do the same, think of another way..” is very rare. I did like one blokes take on this problem:

    jun.lemonie.net/JUnChem2001.html

    Articles such as “Great Results: How to Achieve Them With Statistics.”

    I even managed to get something published….

17. stephenh said,

    June 12, 2006 at 8:36 am

    There’s an interesting article on this topic at:

    Public Library of Science

18. Bookfeller said,

    June 12, 2006 at 1:00 pm

    Please don’t deride groups for failing to get their act together. To say that if we can’t set up a single register of clinical trials then “we truly are lame”, is to imply that social and political problems are “easy-peasy”, so failure in these fields should be a matter for shame.

    But in the real world, politics is not easy. It is actually much more difficult than rocket science. Scientists often make the mistake of thinking that because science is technically difficult, all other problems are relatively easy. With a doctorate in physics I used to make that mistake myself. It made me feel ever so clever to assume that people who couldn’t solve political problems were stupid jerks. Then I started trying to work on such problems and found they really aren’t at all easy.

    I enjoy the “bad science” column, and I think some corrective to the sloppy use of science in advertising and journalism is very much needed. However, I have a sneaking worry that the impact on non-scientists may be negative. It probably makes them feel they truly are lame, and that’s not a nice feeling.

    Can we support people who want to think straight without making them feel lame?

19. Agema said,

    June 12, 2006 at 1:17 pm

    I’d expect a homeopathist would need an axe. It’s not like they’d be much use as a poisoner with their dilutions.

    The current system of science needs publications, and publications in good papers, to justify research grants, with no small pressure involved. Nor is it a trivial job to write a paper. To produce one that had minimal impact in the Journal Of Near-Irrelevance is not a good use of the effort that paper writing can involve. There’s also a certain amount of ego: no one likes to admit they went through 20 different things to find something which didn’t work. It’s close to saying: “Sorry, I’m s**t, and here’s a record of my failure.”

    It’s not like the empirical truth is not emerging. It’s more like what you see is the refined truth rather than the crude mass of data. And I agree science would be better if negative results could be readily presented; it would probably save lots of waste, and give you far more to show that you have actually spent the last 3 years working. But under the current system, don’t hold your breath.

20. Bender Blog » Blog Archive » Quem distorce suas pesquisas médicas? said,

    June 12, 2006 at 3:03 pm

    […] A história é repetitiva: "uma pesquisa sobre [qualquer coisa] revela que [substância produzida por um grande laboratório] pode curar [doença crônica séria]" e toda a imprensa internacional (insclusive o Fantástico) noticia. Semanas depois uma outra pesquisa desmente a primeira e a academia segue em frente, mas os pontos negativos nunca recebem o mesmo destaque. Então, de quem é a culpa, pergunta Ben Goldacre? Posted by Bender Filed in Mundo real, Comentários […]

21. Raging Potato said,

    June 13, 2006 at 5:59 am

    With regard to constructing online databases of negative experimental data, Cornell University has established a browseable, searchable database of real experimental data. Users can search, analyse, download and submit data from neurophysiological experiments. The idea is that somebody else might be able to use your data to examine a hypothesis, saving them the expense and time of actually doing the experiment themselves, and thus conforming to 2 of the three R’s of ethical experimentation with animals (reduction, refinement, replacement).

    It’s still at the developmental stage, and I haven’t got it to work for me, but the principle is sound; often (particularly in neurophysiology) the difficult part is actually getting the data. There are many times in which it would be relatively simple to extract useful information from experiments that examined something else.

    If you’re interested, it’s at neurodatabase.org (I read about it in Nature Neuroscience).

    Pip pip

22. Delster said,

    June 13, 2006 at 11:26 am

    I think what would be better than a database of negative results would be a database of all results.

    ie Say i have the idea of checking if controlled ragweed pollen extract exposure would actually reduce allergy symptoms. I can then go away and check the database using those as keywords.

    If this is done i would find that someone has actually done this but using grass pollen as opposed to ragweed. (i know… i was one of the guinea pigs 🙂 )

    I could then study their results to see how their study was designed and hence design mine to cover any gaps i might feel warrented attention or persue their results in another direction.

    This way research would actually complement prior studies and build upon results rather than being another stand alone project.

    Think how this would apply in the world of cancer research where you have lord knows how many individual charities supporting large numbers of studies into various cancer types. I’d be willing to bet that 50% of the money that actually gets into research is wasted on double efforts.

23. Raging Potato said,

    June 14, 2006 at 3:29 am

    Delster

    There is a database of experimental results; it’s called pubmed.com

    The kind of information you desire is contained within the normal scientific literature; you go to pubmed, type in your keywords (ragweed AND allergy) and you get links to the relevant publications. Release of the ‘raw data’ from this kind of study would not be terribly beneficial to the scientific community because the interpretation of such work is very straightforward; you expose the patient, measure the response (i.e. bronchial constriction/wheal and flare/subjective itchyness scores or whatever). There’s not much new data that can be gleened from such a database (as oppossed to reanalysis of old data).

    On the other hand, more technical studies (i.e. recording the behaviour of parts of the brain in response to a stimulus or whatever) contain vast amounts of information, normally as a result of simultaneous recordings of various parameters (e.g. heart rate + blood pressure + nerve activity). The results from a particular study may focus on one subset of the data (i.e. what happens to nerve activity in the 30s after you do proceedure A), when such a subset only represents perhaps 10% of the whole dataset. In these instances, there may be other researchers interested in what the relationship between one parameter and another at basal levels (e.g. is nerve activity modulated by heart rate?). Public access to such results should be encouraged and could be greatly beneficial.

    I’d be willing to bet that much less than 50% of the money that gets into research is wasted on double efforts; I know this because I’m a researcher and I’ve had to run the grant gauntlet. Research that tries to reproduce previous results is simply not funded. There are more important things to fund, and not enough money. The people that decide who the cash goes are experts in the field, recruited by the grant-funding bodies for said expertise. Only about 10-15% of applications are successful, and the people that write the successful grants are the people with the track record of high impact novel findings (with the odd exception). That’s not to say that nobody tries to replicate previous findings – existing models are frequently used to develop new techniques, and sometimes contentious findings need to be reviewed by other laboratories before they’re accepted as fact. This is definitely a minority however.

24. Delster said,

    June 14, 2006 at 10:01 am

    I know that pubmed contains this kind of information on trials. how ever it does not contain information on all trials conducted. Only ones that have been published, and probably not all of those.

    What we’re talking about is combining both the positive result trials and the negative result trials into a database that would provide a better way of cross checking for previous investigations into what ever subject.

    even if only the authors, paper or trial name and key words were entered into this database it would be an excellent research tool.

    To give an example. Say i’ve done a trial, written it up etc but that the trial was a negative result. I may well chose not to publish to the general science community but just pass my results along to those paying or who had to be reported to. What i could then do in this instance is to enter the result into the hypothetical database.

    If somebody then requested the results based upon their search i could direct them to the study results or send along the abstract etc.

    As for your statement that the results of a study of this kind not being advantageous, i’d have to disagree.

    The study in question was not simple exposure to pollen. What was done was an extract of the pollen was made and administered to the patients on a daily basis over a period of 18 months. This was done through a solution held under the tongue for 60 seconds then swallowed. unfortunatly i was only able to do 6 months of this trial due to sever injuries and i don;t know what the end results were.

    Personal experience seems to indicate a good reduction in symptoms for around 4-5 years from my partial treatment (anecdotal i know). However this study would have produced data showing either a positive result (which may indicate a good delivery method to further researchers and encourage further trials with other pollen types) or a negative sugesting the opposite.

    Either way, more knowledge = good, less = bad

25. Robert Carnegie said,

    June 17, 2006 at 11:30 pm

    Agema: I think the point not that research may be done badly and still should be published, nor that a scientist could lose face by publishing or releasing a series of scientifically valid but commercially unrewarding results. If you’re responsible for picking the experiments that you do, and all that you add to scientific knowledge is “That doesn’t work, and that doesn’t work”, it isn’t much. But you may be employed to carry out exactly those experiments. Perhaps that in itself is not a position of prestige…

26. ikhovablovrmeenkaku said,

    July 6, 2006 at 12:53 am

    3397
    The very best portal for all your porn links is Porn Plaza

27. Questions so far « Ethics of Medical Journalism: Tom’s PhD diary said,

    October 18, 2006 at 3:25 pm

    […] 10. Does freely available, peer-reviewed scientific literature – which should in theory remove concerns about conflicts of interest on the part of the writers – confuse journalists more used to hunting out hidden agendas? A common theme appears to be “this researcher has in the past received funding from Drug Company A; therefore we should distrust his research purporting to show the efficacy of Drug Company A’s new product”. Is this unfair or is there reason to doubt it? Goldacre: www.badscience.net/?p=251 ”…over the past few years there have been numerous systematic reviews showing that studies funded by the pharmaceutical industry are several times more likely to show favourable results than studies funded by independent sources”. Sinister? […]

28. Blog Atopowy » Blog Archive » Koniec homeopatii? said,

    November 23, 2007 at 5:13 am

    […] masę pozytywnych, choć nieprawdziwych wyników? Ponieważ jest coś takiego jak „skrzywienie publikacyjne”. We wszystkich dziedzinach nauki, rezultaty pozytywne mają o wiele większą szansę bycia […]

29. Pushing Pixels » Blog Archive » re: depression said,

    January 28, 2008 at 6:22 pm

    […] a cracking new analysis of the “publication bias” in the literature, a group of academics this week published a paper in the New England […]

30. jiangjiang said,

    December 8, 2009 at 2:16 am

    ed hardy ed hardy
    ed hardy clothing ed hardy clothing
    ed hardy shop ed hardy shop
    christian audigier christian audigier
    ed hardy cheap ed hardy cheap
    ed hardy outlet ed hardy outlet
    ed hardy sale ed hardy sale
    ed hardy store ed hardy store
    ed hardy mens ed hardy mens
    ed hardy womens ed hardy womens
    ed hardy kids ed hardy kids ed hardy kids

31. Pulling the Curtain Back from Scientific Publishing - said,

    June 6, 2013 at 1:44 am

    […] you probably won’t be bothered with reading these boring results. Yes, we joyfully embrace publication bias. But in science, knowing something isn’t, or that it is routinely, are really valuable things to […]

Name (required)

Mail (will not be published) (required)

Website

 

Δ

Stuff

• “Where did you go?” My Inaugural Professorial Lecture at Oxford!
• About Dr Ben Goldacre
  • … and upcoming public talks
  • … book me to speak
  • — Twitter
• Books
  • … Bad Pharma
  • … Bad Science
• What You Can Find Here
  • … Audio and Video
  • … The Bad Science Shop

I love them

Archives

• December 2017
• July 2017
• February 2017
• October 2016
• September 2016
• March 2016
• August 2015
• July 2015
• June 2015
• April 2015
• February 2015
• December 2014
• November 2014
• October 2014
• July 2014
• June 2014
• March 2014
• January 2014
• December 2013
• October 2013
• July 2013
• June 2013
• May 2013
• April 2013
• March 2013
• February 2013
• January 2013
• December 2012
• October 2012
• September 2012
• June 2012
• April 2012
• March 2012
• December 2011
• November 2011
• October 2011
• September 2011
• August 2011
• July 2011
• June 2011
• May 2011
• April 2011
• March 2011
• February 2011
• January 2011
• December 2010
• November 2010
• October 2010
• September 2010
• August 2010
• July 2010
• June 2010
• May 2010
• April 2010
• March 2010
• February 2010
• January 2010
• December 2009
• November 2009
• October 2009
• September 2009
• August 2009
• July 2009
• June 2009
• May 2009
• April 2009
• March 2009
• February 2009
• January 2009
• December 2008
• November 2008
• October 2008
• September 2008
• August 2008
• July 2008
• June 2008
• May 2008
• April 2008
• March 2008
• February 2008
• January 2008
• December 2007
• November 2007
• October 2007
• September 2007
• August 2007
• July 2007
• June 2007
• May 2007
• April 2007
• March 2007
• February 2007
• January 2007
• December 2006
• November 2006
• October 2006
• September 2006
• August 2006
• July 2006
• June 2006
• May 2006
• April 2006
• March 2006
• February 2006
• January 2006
• December 2005
• November 2005
• October 2005
• September 2005
• August 2005
• July 2005
• June 2005
• May 2005
• April 2005
• March 2005
• February 2005
• January 2005
• December 2004
• November 2004
• October 2004
• September 2004
• August 2004
• July 2004
• June 2004
• May 2004
• April 2004
• March 2004
• February 2004
• January 2004
• December 2003
• November 2003
• October 2003
• September 2003
• August 2003
• July 2003
• June 2003
• May 2003
• April 2003
• February 2003
• July 2002
• January 2000
• January 1007

Meta

• Register
• Log in
• RSS
• Comments RSS
• Valid XHTML
• XFN