Ben Goldacre, Saturday 8 August 2009, The Guardian.
Much of what we cover in this column revolves around the idea of a “systematic review”, where the literature is surveyed methodically, following a predetermined protocol, to find all the evidence on a given question. As we saw last week, for example, the Soil Association would rather have the freedom to selectively reference only research which supports their case, rather than the totality of the evidence.
Two disturbing news stories demonstrate how this rejection of best practice can also cut to the core of academia.
Firstly, the Public Library of Science in the US this week successfully used a court order to obtain a full trail of evidence showing how pharmaceutical company Wyeth employed commercial “ghost writers” to produce what were apparently academic review articles, published in academic journals, under the names of academic authors. These articles, published between 1998 and 2005, stressed the benefits of taking hormones to protect against problems like heart disease, dementia, and ageing skin, while playing down the risks. Stories like this, sadly, are commonplace: but to understand the full damage that these distorted reviews can do, we need to understand a little about the structure of academic knowledge.
In a formal academic paper, every claim is referenced to another academic paper: either an original research paper, describing a piece of primary research in a laboratory or on patients; or a review paper which summarises an area. This convention gives us an opportunity to study how ideas spread, and myths grow, because in theory you could trace who references what, and how, to see an entire belief system evolve from the original data. That analysis was published this month in the British Medical Journal, and it is quietly seminal.
Steven Greenberg from Harvard medical school focused on an arbitrary hypothesis: the specifics are irrelevant to us, but his case study was the idea that a protein called β amyloid is produced in the skeletal muscle of patients who have a condition called “inclusion body myositis”. Hundreds of papers have been written on this, with thousands of citations between them, and using network theory Greenberg produced a map of interlocking relationships, demonstrating who cited what.
By looking at this network he could the identify intersections with the most incoming and outgoing traffic. These are the papers with the greatest “authority” (and Google uses the same principle to rank webpages in its search results). All of the ten most influential papers expressed the view that β amyloid is produced in the muscle of patients with IBM: in reality, this is not supported by the totality of the evidence. So how did this situation arise?
Firstly, we can trace how basic laboratory work was referenced. 4 lab papers did find β amyloid in IBM patients’ muscle tissue, and these were among the top ten most influential papers. But looking at the whole network, there were also 6 very similar primary research papers, describing similar lab experiments, which are isolated from the interlocking web of citation traffic, but which received no or few citations: these papers, unsurprisingly, contained data that contradicted the popular hypothesis. Crucially, no papers refuted or critiqued this contradictory data. Instead, they were just ignored.
Using the interlocking web of citations you can see how this happened. A small number of review papers funneled large amounts of traffic through the network, with 63% of all citation paths flowing through one review paper, and 95% of all citation paths flowed through just 4 review papers by the same research group. These papers acted like a lens, collecting and focusing citations on the papers supporting the hypothesis, in testament to the power of a well received review paper.
But Greenberg goes beyond just documenting bias in what research was referenced in each review paper. By studying the network, in which review papers are themselves cited by future research papers, he showed how these reviews exerted influence beyond their own individual readerships, and distorted the subsequent discourse, by setting a frame around only some papers.
And by studying the citations in detail, he went further again. Some papers did cite research that contradicted the popular hypothesis, for example, but distorted it. One laboratory paper reported no β amyloid in three of five patients with IBM, and its presence in only a “few fibres” in the remaining two patients: but 3 subsequent papers cited these data saying that they “confirmed” the hypothesis. This is an exaggeration at best, but the power of the social network theory approach is to show what happened next: over the following 10 years, these 3 supportive citations were the root of 7848 supportive citation paths, producing chains of false claim in the network, amplifying the distortion.
Similarly, many papers presented aspects of the β amyloid hypothesis as a theory, but gradually, through incremental mis-statement, in a chain of references, these papers came to be cited as if they proved the hypothesis as a fact, with experimental evidence, which they did not.
This is the story of how myths and misapprehensions arise. Greenberg might have found a mess, but instead he found a web of systematic and self-reinforcing distortion, resulting in the creation of a myth, ultimately retarding our understanding of a disease, and so harming patients. That’s why systematic reviews are important, and that’s why ghost writing should be stopped.