Ben Goldacre, 12 September 2009, The Guardian
This week the peer review system has been in the newspapers, after a survey of scientists suggested it had some problems. This is barely news. Peer review – where articles submitted to an academic journal are reviewed by other scientists from the same field for an opinion on their quality – has always been recognised as problematic. It is timeconsuming, it could be open to corruption, and it cannot prevent fraud, plagiarism, or duplicate publication, although in a more obvious case it might. The problem with peer review is, it’s hard to find anything better.
Here is one example of a failing alternative. This month, after a concerted campaign by academics aggregating around websites such as Aidstruth.org, academic publishers Elsevier have withdrawn two papers from a journal called Medical Hypotheses. This academic journal is a rarity: it does not have peer review, and instead, submissions are approved for publication by its one editor.
Articles from Medical Hypotheses have appeared in this column quite a lot. They carried one almost surreally crass paper in which two italian doctors argued that “mongoloid” really was an appropriate term for people with Down syndrome after all, because they share many characteristics with oriental populations (including: sitting cross legged; eating small amounts of lots of different types of food with MSG in it; and an enjoyment of handicrafts). You might also remember two pieces discussing the benefits and side effects of masturbation as a treatment for nasal congestion.
The papers withdrawn this month step into a new domain of foolishness. Both were from the community who characterise themselves as “Aids dissidents”, and one was co-authored by their figureheads, Peter Duesberg and David Rasnick.
To say that a peer reviewer might have spotted the flaws in their paper – which had already been rejected by the Journal of Aids – is an understatement. My favourite part is the whole page they devote to arguing that there cannot be lots of people dying of Aids in South Africa, because the population of that country has grown over the past few years.
We might expect anyone to spot such poor reasoning – and only two days passed between this paper’s submission and acceptance – but they also misrepresent landmark papers from the literature on Aids research. Rasnick and Duesberg discuss antiretroviral medications, which have side effects, but which have stopped Aids being a death sentence, and attack the notion that their benefits outweigh the toxicity: “contrary to these claims”, they say, “hundreds of American and British researchers jointly published a collaborative analysis in The Lancet in 2006, concluding that treatment of AIDS patients with anti-viral drugs has ‘‘not translated into a decrease in mortality” [30].”
This is a simple, flat, unambiguous misrepresentation of the Lancet paper to which they refer. Antiretroviral medications have repeatedly been shown to save lives in systematic reviews of large numbers of well-conducted randomised controlled trials. The Lancet paper they reference simply surveys the first decade of patients who received HAART – modern combinations of multiple antiretroviral medications – to see if things have improved, and they have not. Patients receiving HAART in 2003 did no better than patients receiving HAART in 1995. This doesn’t mean that HAART is no better than placebo. It means outcomes for people on HAART didn’t improve over an 8 year period of their use. This would be obvious to anyone familiar with the papers, but also to anyone who thought to spend the time checking the evidence for an obviously improbable assertion.
What does all this tell us about peer review? The editor of Medical Hypotheses, Bruce Charlton, has repeatedly argued – very reasonably – that the academic world benefits from having journals with different editorial models, that peer review can censor provocative ideas, and that scientists should be free to pontificate in their internal professional literature. But there are blogs where Aids dissidents, or anyone, can pontificate wildly and to their colleagues: from journals we expect a little more.
20 academics and others have now written to Medline, requesting that Medical Hypotheses should be removed from their index. Aids denialism in South Africa has been responsible for the unnecessary deaths of an estimated 330,000 people. You could do peer review well, or badly. You could follow the single editor model well, or foolishly. This article was plainly foolish.
TeeD said,
September 16, 2009 at 3:29 pm
In the comments on The Guardian version of this post (which unfortunately are now closed), the current president of AIDS denialist organization “rethinking AIDS,” David Crowe, has posted two selections of quotes critiquing of peer review, with no commentary. Without a trace of irony, he quotes extensively from this Duesberg paper:
Duesberg P et al. The chemical bases of the various AIDS epidemics: recreational drugs, anti-viral chemotherapy and malnutrition. J Biosci. 2003 Jun; 28(4): 383–412.
e.g. “Even the professional journals and the science writers of the public media comply with the interests of government- funded majorities because they depend on their monthly scientific breakthroughs, the lucrative advertisements from their companies, and the opinion of their subscribers.”
The paper also complains about how other journals refused to publish it. But this paper actually goes one better than the Medical Hypotheses paper and doesn’t just misrepresent a study on the benefits of antiretroviral therapy, it flat-out misquotes it.
The authors – Peter Duesberg, Claus Koehnlein & David Rasnick – change a sentence from this paper:
Declining Morbidity and Mortality among Patients with Advanced Human Immunodeficiency Virus Infection
content.nejm.org/cgi/content/full/338/13/853
that reads: “Patients with a diagnosis of cytomegalovirus retinitis or M. avium complex disease before study entry or during the first 30 days of follow-up and patients with active P. carinii pneumonia at the beginning of follow-up were excluded from the analyses of the incidence of that opportunistic infection.”
To: “Patients with a diagnosis of cytomegalovirus retinitis or M. aviarum complex disease before study entry or during the first 30 days of follow-up and patients with active P. carinii pneumonia at the beginning of follow-up were excluded.”
www.virusmyth.com/aids/hiv/pddrchemical.pdf – page 399
In order to pretend that the people with AIDS described in the Palella paper weren’t sick before starting antiretroviral therapy, because Duesberg’s theory says antiretrovirals cause AIDS. The paper actually shows that mortality was highest in people who received no treatment, and lowest in people on combinations including a protease inhibitor.
Lloyd said,
September 16, 2009 at 8:23 pm
Dr Goldacre may find that he is in a difficult position. He has used terms such as “almost surreally crass” and “a new domain of foolishness” to ridicule Bruce Charlton’s journal, and having done so will now risk making himself look rather foolish if he publishes a retraction. I respect the notion of a website that illuminates cases of bad science in the media. However there is a danger that the single voice of Dr Goldacre becomes too powerful, and I certainly do not see it as his place to decide what can or cannot be said. Medical Hypotheses is not a typical science journal, but makes no claim to be. One of the worst things about bad science is when something that is bad science, or even not science, puts itself across as good science, and is therefore given credence it does not deserve. Medical Hypotheses does not pretend to be something that it is not. Neither is it widely read by
the lay public. It is intended as a means of stimulating new ideas, and perhaps enabling scientists to go a bit further off the beaten track than they are normally allowed to go. If articles in MH are taken out of context by other media writers, and given greater weight
that they deserve, then this is a mistake of the lazy journalists who do this.
Ben Goldacre does not have the right to decide what is and what isn’t acceptable to theorise. If ever he has the effect of stagnating debate and restricting new ideas, then he has become the enemy of science. He may write what he wishes, but should not encourage a campaign to shut down a journal. Science is not an opinion, nor a
consensus. When a journalist writes “scientists believe” he is usually being lazy and misleading. Science is simply a method for finding things out. One of the things required for good science to thrive is a world in which people can come up with new ideas and try
them out. At any given moment, the scientific mainstream is in a particular state. The only way this can change is if people are allowed to spread new ideas. Without new unaccepted ideas, science halts. If an idea is published in a journal, and that same journal publishes several other ideas that are bad, then the first idea is
scientifically no worse by association with those other bad ones. An idea is either good or not good, and the person who came up with it is an irrelevance, as is its proximity in a journal to other ideas.
Dr Goldacre runs a website which in some ways resembles MH. He is its one editor. He writes what he pleases to write. I agree with much of what he writes. I disagree with much of what he writes. It would not occur to me to mount a campaign to ban his site because I disagreed with one of the items on it. I do not see in Dr Goldacre’s writing any plainly-expressed call for an end to MH, but he is now a man of influence, and may even spark a witch-hunt when he did not mean to. If Dr Goldacre finds a bad idea in MH, he should attack the idea because it is bad. Peer review is flawed, as he admits in his article. If everything has to be peer reviewed to be accepted as “science” then one of the main flaws of peer review (that ideas not in keeping with the current fashion become suppressed) becomes massively magnified. A partial remedy for this is to have outlets that are not
peer-reviewed for new scientific ideas. So long as those outlets are clearly labelled as what they are, this is fine.
John Moore said,
September 16, 2009 at 9:53 pm
I cannot agree with Lloyd, particular over the comment “Medical Hypotheses does not pretend to be something that it is not. Neither is it widely read by the lay public” These statements might be true from a purely technical perspective, but they miss an important point that I wrote about in post 45 yesterday. Apologies for any repetition of the same theme here, but it seems that at some people at least have rather missed the point.
The AIDS denialist groups desperately seek to get their pseudoscience and anti-science views into any journal whatsoever that has some form of credibility, and being listed on PubMed does provide a journal with perceptional credibility. The lay public indeed rarely reads the journals, and members of the public probably don’t have the general knowledge to tell the difference between Med Hyp and Nature or the NEJM. The AIDS denialists ruthlessly exploit that lack of knowledge. They do so by posting “their” papers on websites, and by referring to them as examples of how “their” work has passed peer review (sic), so MUST be quality science. They do this to confuse the public, and the tactic works, at least to an extent. Readers of the denialist web sites who lack a scientific background cannot and do not discriminate between what’s in the denialist papers, and what’s in the vast body of scientific literature on HIV/AIDS. Med Hyp itself might well not “pretend to be something that it is not”, but for sure the AIDS denialists pretend that it’s a quality, peer-reviewed journal. And by doing so they fool, and harm, the lay public.
I don’t follow the vaccine-autism websites, but I think the same scenario likely applies there: “If it’s in Med Hyp, it must be right, right?”.
The way to stop these abuses of the scientific literature is to ensure that only sound, solid science with a rational basis is published in peer-reviewed, PubMed-listed journals. There are websites dedicated to the advancement of crazy ideas that are contrary to the best interests of public health. Their existence is a great shame, but there’s nothing that scientists can do about that. What we can do is insist on high standards for the scientific literature. Med Hyp has not met those standards, and in not doing so it has betrayed our profession.
DavidN said,
September 17, 2009 at 11:37 am
Point taken.
I was shocked last year to hear of AIDS denialist tropes being passed around a dinner table with a very well known British-American businesswoman last year, herself well acquainted with scientific knowledge and her husband being a major educational benefactor.
I think for some people they have multiple and conflicting worldviews and it only seems to bother me/us. Even bringing up peer review makes people yawn. Anti-science thinking has encroached in many other ways, sbversively ‘management’ theory is one that prports to be scientific in a way bt is certainly not.
Most people that earn a decent wage and live in the nice areas of this city, seem to have a major affectation for alternative medicine. So many doctors are no utilising ‘integrative’ many more than when I left a few years ago.
I’m tired and rambling and am going to get flamed for this probably.
TeeD said,
September 17, 2009 at 5:30 pm
“I’m tired and rambling and am going to get flamed for this probably.”
Not by me, I felt bad for taking a dig at disgruntled humanities grads before and I think you make a good point about the encroachment of anti-science thinking. I just noticed that the latest attempt at propaganda for AIDS denial, a seemingly well-funded “documentary” entitled House of Numbers, lists as an Executive Producer Martin Penny, who I think is also a major educational benefactor in the UK (he’s CEO of Good Hair Day). Penny is also cited as supporting an upcoming AIDS denial conference in the US in November:
ra2009.org/index.php?option=com_content&view=article&id=52&Itemid=57
According to the internet archive, Penny’s name was added to the list of people “who doubt the HIV-AIDS theory” between June 29 and July 2 of 2007:
web.archive.org/web/20070629184931/http://www.rethinkingaids.com/quotes/rethinkers.htm
web.archive.org/web/20070702055910/http://www.rethinkingaids.com/quotes/rethinkers.htm
Bizarre.
Nasimf said,
September 17, 2009 at 7:10 pm
My 2p worth. I don’t think we want censorship.
Peer review has its pitfalls (particularly in the exposure of most scientific fraud, where journals do a piss-poor job except in headline cases). That is not to say that MH does a good job, or serves a wortyhwhile purpose, but it could do.
After all they publish anything written in (reasonably comprehensible) English and it would seem an ideal forum for a robust response.
I would personally favour the existence of many more journals like MH, but with some gatekeeping (over factual content as opposed to interpretation).
Journals have totally useless correspondence sections, which is a problem.
DavidN said,
September 17, 2009 at 8:26 pm
@TeeD I find Aids denialism extremely disturbing.
When Jeanette Winterson advocates Homeopathic remedies for AIDS in Africa etc it is tantamount to saying “These people do not deserve real tested treatment for their illness”. That’s only a few steps from ideas advocated by American Celebs that one’s life is wholly predicated on their thought patterns.
I am going to have a good read of this (probably while sitting through this boring conference I’m being sent to today).
I think AIDS denialism is illegal in NZ where I live at the moment. There was a major campaign about the science of AIDS after a young girl Eve van Grafhorst died in the 90’s. I should check it out.
jtuomist said,
September 18, 2009 at 2:21 pm
If we think of a typical article, it consists of two parts: first, a description of the study design and the data obtained; second, interpretations of the results and discussion in a wider context. The first part is permanent in time, as the observed data does not change. In contrast, making interpretations is a social activity (involving also other researchers than the original authors), and it will change in time (sometimes dramatically).
Only the first part should be archived in a frozen, article-type form. These could well be published first and peer reviewed only later, as is the current practice in some fields of science (see arxiv.org/).
The second part, interpretations, should be dealt with in open workspaces designed for mass collaboration. There are methods to effectively organise these discussions to achieve convergence. To give one example, see pragma-dialectics in Wikipedia. These methods are still under-utilised (this blog is a typical example).
As an example, let’s look at the current discussion raised by Peter Duesberg and David Rasnick. If their interpretations are on shaky grounds, they could have easily been shot down in an open workspace, without any need of editorial decisions. In an open workspace, anyone could publish their statements (idea promoted by Bruce G Charlton) and all statements incoherent with facts would be invalidated (idea promoted by Ben Goldacre) by peers. There is no need to remove invalid statements, because they are shown to be invalid. On the contrary, it prevents from repeating invalid statements.
A paradigm shift to open scientific workspaces has at least two major problems. It is not clear how scientists could get merit from participating in mass collaboration instead of writing articles. In addition, this might cause problems to the business logic of scientific journals.
We have drafted a procedure description for a journal that is based on the ideas briefly described above: Opasnet Journal (en.opasnet.org/w/Opasnet_Journal).
Mikko Pohjola
Jouni Tuomisto
environmental health scientists
THL, Finland
heavens said,
September 22, 2009 at 12:28 am
I have a question for Bruce Charlton:
Is there any particular reason that you couldn’t add the fairly basic level of fact-checking that a daily newspaper does to MedHyp? That is, if you see some _fact_ being wildly misrepresented, you call the author and say, “There seems to be an error here, and I hope you can fix it and re-sumbit,” instead of saying “Eh, it’s wrong, but who cares what lies are in it, as long as I sell papers?”
I’ve gotten simple facts wrong before (e.g., by overlooking a single word in a paragraph), and I’ve always appreciated people pointing it out to me. Surely this would be better than having the entire world ridicule the author, the journal, and yourself.
heavens said,
September 22, 2009 at 12:31 am
Also, to other readers:
I don’t think that MedHyp’s dubious reputation is known outside of the medical scholars. I remember a well-respected and very senior chemistry professor at a large American research university gloating about a graduate student publishing something in this journal. He thought that the quick response and lack of changes meant that the idea was absolutely perfect and predicted great things for the student’s career.
wayscj said,
November 21, 2009 at 6:40 am
ed hardy ed hardy
ed hardy clothing ed hardy clothing
ed hardy shop ed hardy shop
christian audigier christian audigier
ed hardy cheap ed hardy cheap
ed hardy outlet ed hardy outlet
ed hardy sale ed hardy sale
ed hardy store ed hardy store
ed hardy mens ed hardy mens
ed hardy womens ed hardy womens
ed hardy kids ed hardy kids ed hardy kids