Ben Goldacre, Saturday 8 August 2009, The Guardian.
Much of what we cover in this column revolves around the idea of a “systematic review”, where the literature is surveyed methodically, following a predetermined protocol, to find all the evidence on a given question. As we saw last week, for example, the Soil Association would rather have the freedom to selectively reference only research which supports their case, rather than the totality of the evidence.
Two disturbing news stories demonstrate how this rejection of best practice can also cut to the core of academia.
Firstly, the Public Library of Science in the US this week successfully used a court order to obtain a full trail of evidence showing how pharmaceutical company Wyeth employed commercial “ghost writers” to produce what were apparently academic review articles, published in academic journals, under the names of academic authors. These articles, published between 1998 and 2005, stressed the benefits of taking hormones to protect against problems like heart disease, dementia, and ageing skin, while playing down the risks. Stories like this, sadly, are commonplace: but to understand the full damage that these distorted reviews can do, we need to understand a little about the structure of academic knowledge.
In a formal academic paper, every claim is referenced to another academic paper: either an original research paper, describing a piece of primary research in a laboratory or on patients; or a review paper which summarises an area. This convention gives us an opportunity to study how ideas spread, and myths grow, because in theory you could trace who references what, and how, to see an entire belief system evolve from the original data. That analysis was published this month in the British Medical Journal, and it is quietly seminal.
Steven Greenberg from Harvard medical school focused on an arbitrary hypothesis: the specifics are irrelevant to us, but his case study was the idea that a protein called β amyloid is produced in the skeletal muscle of patients who have a condition called “inclusion body myositis”. Hundreds of papers have been written on this, with thousands of citations between them, and using network theory Greenberg produced a map of interlocking relationships, demonstrating who cited what.
By looking at this network he could the identify intersections with the most incoming and outgoing traffic. These are the papers with the greatest “authority” (and Google uses the same principle to rank webpages in its search results). All of the ten most influential papers expressed the view that β amyloid is produced in the muscle of patients with IBM: in reality, this is not supported by the totality of the evidence. So how did this situation arise?
Firstly, we can trace how basic laboratory work was referenced. 4 lab papers did find β amyloid in IBM patients’ muscle tissue, and these were among the top ten most influential papers. But looking at the whole network, there were also 6 very similar primary research papers, describing similar lab experiments, which are isolated from the interlocking web of citation traffic, but which received no or few citations: these papers, unsurprisingly, contained data that contradicted the popular hypothesis. Crucially, no papers refuted or critiqued this contradictory data. Instead, they were just ignored.
Using the interlocking web of citations you can see how this happened. A small number of review papers funneled large amounts of traffic through the network, with 63% of all citation paths flowing through one review paper, and 95% of all citation paths flowed through just 4 review papers by the same research group. These papers acted like a lens, collecting and focusing citations on the papers supporting the hypothesis, in testament to the power of a well received review paper.
But Greenberg goes beyond just documenting bias in what research was referenced in each review paper. By studying the network, in which review papers are themselves cited by future research papers, he showed how these reviews exerted influence beyond their own individual readerships, and distorted the subsequent discourse, by setting a frame around only some papers.
And by studying the citations in detail, he went further again. Some papers did cite research that contradicted the popular hypothesis, for example, but distorted it. One laboratory paper reported no β amyloid in three of five patients with IBM, and its presence in only a “few fibres” in the remaining two patients: but 3 subsequent papers cited these data saying that they “confirmed” the hypothesis. This is an exaggeration at best, but the power of the social network theory approach is to show what happened next: over the following 10 years, these 3 supportive citations were the root of 7848 supportive citation paths, producing chains of false claim in the network, amplifying the distortion.
Similarly, many papers presented aspects of the β amyloid hypothesis as a theory, but gradually, through incremental mis-statement, in a chain of references, these papers came to be cited as if they proved the hypothesis as a fact, with experimental evidence, which they did not.
This is the story of how myths and misapprehensions arise. Greenberg might have found a mess, but instead he found a web of systematic and self-reinforcing distortion, resulting in the creation of a myth, ultimately retarding our understanding of a disease, and so harming patients. That’s why systematic reviews are important, and that’s why ghost writing should be stopped.
bravante said,
August 8, 2009 at 2:46 am
Ben,
I completely agree with the need of systematic reviews. The problem is that many reviewers tend to over-represent their own research and push their own agenda. And this gets even worse when they start referencing previous reviews, which makes very difficult to trace the original data.
But things have been getting even worse lately, with the advent of the so-called minireviews. These are basically a summary of a small subset of articles and very prone to bias. And then we have those that reference the minireviews…… No wonder the literature can be manipulated!
RogerWilco said,
August 8, 2009 at 6:01 am
Ben,
This is great article and very thought-provoking. However, I was very disappointed by the sentence: “Similarly, many papers presented … the… hypothesis as a theory, but gradually … these papers came to be cited as if they proved the hypothesis as a fact.”
You appear to be blurring scientific term “hypothesis” and the informal definition of the word “theory”. This is more fodder for the “evolution is only a theory” crowd. It’s best to avoid using “theory” at all in this way.
SteveGJ said,
August 8, 2009 at 6:38 am
It’s another use of the fascinating network theory that has so many implications for us all. At least in this sort of case, it’s possible to formally trace this network due to scientific papers having to cite their sources. It looks to me here as if we have something rather analogous to academic paper inbreeding. Too many papers being produced from far too narrow a meme pool, and the mechanism of environmental selection using systematic reviews is too weak to do the job of weeding out the bad (or at least, unsupported) ideas. Indeed it might be that the bad meme has evolved some immunity to such processes (yes, the analogy is getting more than a trifle fanciful at this stage).
What would be nice is if somebody could come up with a means of analysing related documents and ideas such that the relationships could be determined from content in the way that mitochondrial DNA is used in population studies. Alas, I suspect that such a thing is almost impossible. However, the reason why it would be so interesting is that network theory tells us that there are particular nodes that are immensely influential. What holds for academic papers is also true for all those other rather important things in life: politics, your place in society, religious groups, journalism, art, culture how companies or organisations operate and innumerable other things.
Of course the Internet provides a wonderful testbed for such things. Patterns of communication, social networking sites, hyperlinks and so on all provide for some fascinating and, perhaps worrying, insight into how society really works.
Back to academic and research papers. It would be interesting to make comparisons between different academic disciplines. For instance, are the references used in the humanities different in nature to those of some of the “hard” natural sciences.
rhebus said,
August 8, 2009 at 10:05 am
bravante: The important word here is “systematic”. In a high-quality systematic review, such as those produced by the Cochrane collaboration, you show not just your review but also your methods for finding important papers, your criteria for inclusion, even the terms you entered into search engines. People can then see if you are applying your criteria fairly or are biasing in favour of either your own research or your own theories.
Granted, not every review is as squeaky clean as the Cochrane collaboration, but they demonstrate that a good, reliable, trustworthy review is an achievable goal.
Michael Gray said,
August 8, 2009 at 10:37 am
But the woo pushers will still claim that you do not criticize mainstream medical research…
Synchronium said,
August 8, 2009 at 10:45 am
Presumably Steven Greenberg has a graphical representation of this work, link a link graph of the web?
Would be cool to see it.
scotslawstudent said,
August 8, 2009 at 11:46 am
RodgerWilco:
I do see your point but I think you have to admit that evolution -is- only a hypothesis that has stood up beautifully to empirical examination. Until we hear the definitive answer from some divine creator being it remains a hypothesis that is open to rebuttal if someone collects enough evidence which can’t be explained by the current consensus. It’s looking like no one ever will but it remains, hypothetically (ha 😉 ) possible.
I think “irony” is another word which has suffered greatly from the curse of the informal meaning, where it has drifted from being a very specific literary device into meaning basically “unfortunate.” If someone comes at that sentence thinking “well, theory just means hunch” then I do see your point.
Remembering that I read the column as a Guardian reader and not as a scientist I understood that sentence to be that a hypothesis that you’d only get away with because you’re about to follow it with some research has then been stretched into a shape that the original evidence can’t reasonably support and made much more certain, in future papers. I’m trying to decide if, for me, Ben’s parenthesis “…as a fact, with experimental evidence…” suggests that theories don’t have experimental evidence behind them or if it merely distinguishes hunch from truth. Maybe Dr Ben should do a basic primer on the scientific method every so often ;).
scotslawstudent said,
August 8, 2009 at 11:50 am
Ah, I have have just mentally read back my previous post, for the avoidance of doubt I mean to say- Dr Ben should do a basic primer in his column every so often. That means that people won’t come to the column thinking that theory is the same as hunch when used in a scientific context. I think the above could be read as Dr Ben should study a basic primer himself and that’s not what I meant.
njdowrick said,
August 8, 2009 at 1:32 pm
Scotslawstudent: the theory of evolution through natural selection *is* a theory, not just a hypothesis. A hypothesis is an isolated idea: for example, the idea about the protein that is discussed in Ben’s article above. A theory is an idea or set of ideas that between them correlate a vast range of observations. Other theories in science include: the theory of plate tectonics, the theories of relativity, the theory of quantum mechanics, the wave theory of light, atomic theory, the kinetic theory of gases, the germ theory of disease, the Big Bang and Steady State theories of cosmology …
So a theory is a much grander thing than a mere hypothesis! Calling an idea a theory, far from casting any doubt on it, is to praise the idea for its power and the wide range of situations it can be applied to. I have to agree with RogerWilco (comment 2) that Ben is using the term “theory” in its colloquial sense rather than in its scientific sense. It is this colloquial sense that leads to confusion when people hear scientists talking about the “theory” of evolution.
BenHemmens said,
August 8, 2009 at 3:44 pm
Oh yes, everyone who’s spent time in research knows this problem. Once, we found a PNAS paper a bit odd, so we went off and had a look at the effects in our own lab. Result, total debunk of the original paper. Where did we publish it? FEBS Letters.
As a rule of thumb, you will get a negative or critical report into something with an impact factor of 1/10 of where the original claim appeared. If you can publish it at all. This is a pretty bloody major weakness of the system.
hozepipe said,
August 8, 2009 at 8:36 pm
I agree with both njdowrick and RogerWilco on this point and want to add that a theory can also have predictive power and a mechanism by which we can understand why and how something is the way it is. Theories are grander than facts and hypotheses!
philbo said,
August 9, 2009 at 5:31 pm
Bertrand Russell once said:
“What a man believes upon grossly insufficient evidence is an index into his desires — desires of which he himself is often unconscious. If a man is offered a fact which goes against his instincts, he will scrutinize it closely, and unless the evidence is overwhelming, he will refuse to believe it. If, on the other hand, he is offered something which affords a reason for acting in accordance to his instincts, he will accept it even on the slightest evidence. The origin of myths is explained in this way.”
Even in areas like science where we try try to develop methods which avoid human bias, it seems this still holds true.
Staphylococcus said,
August 10, 2009 at 2:20 am
@ scotslawstudent
“you have to admit that evolution -is- only a hypothesis that has stood up beautifully to empirical examination.”
That is, essentially, the definition of a scientific theory. The theory of evolution has more evidence to support it than the theory of gravity, or atomic theory – we know the models for the last two are wrong, but we continue to use them because we have no real alternative.
smart said,
August 10, 2009 at 10:37 am
Mr Goldacre is heavily into selective omission of evidence himself.
For instance in one of his 2006 articles Mr Goldacre would have you believe that “scare stories” about mercury fillings “come round every few years”. And yet this is an obscene falsehood, presented in an offensively arrogant manner. The reality of dental mercury amalgam is that there has long been decisively strong evidence of harmfulness, and the defenders such as Mr Goldacre merely pretend that that evidence does not exist.
For instance I wrote to the Chief Dental Officer pointing out the 26 studies of 6000 patients, showing exceptional benefits from amalgam removal. These cannot be merely dismissed as placebo effects. I sent him these papers here: www.iaomt.org/articles/category_view.asp?intReleaseID=214&month=10&year=2006&catid=35
The CDO wrote back asserting that all the studies I had cited had been “superceded” by the SCENIHR report. And yet when you search through the 72 pages of that report those 26 studies are
NOT EVEN MENTIONED. And it turns out that a whole lot more is NOT EVEN MENTIONED in the 72 pages of scenihr, as indicated in Mats Hanson’s critique obtainable at www.mercurymadness.org/viewdata.aspx?sectionid=1&dataid=124.
And that lot is NOT EVEN MENTIONED by the supposedly evidence-based expert Ben Goldacre too. And the above evidence probably accounts for the fact that amalgam has this year been banned in Sweden, Norway and Denmark, and that last year the US FDA was forced by legal challenge into flipping from its asserting safety of amalgam to instead issuing warnings of its neurotoxicity instead.
It’s notable that Ben Goldacre hasn’t done a “bad science” thing on any of that yet. Mr Goldacre appears to be not really in the business of debunking pseudoscience but only in debunking anything that challenges big-billions corporate profits even when it is necessary to side with the pseuds against the truth to do so.
smart said,
August 10, 2009 at 10:46 am
PS- the iaomt link above now lacks the 26th study, (Wojcik Godfrey Christie Haley 2006) which you can find as third link down on www.drfarid.com/mercury.html
gemmad said,
August 10, 2009 at 12:43 pm
This reminds me of a number that is used widely in the field that I work in for the number of hepatocytes per gram of human liver. Everyone in the field was (and some annoyingly still are) using a value that was thought to be correct because it was cited in a number of papers and review articles. However, when the number was traced back to the source (via numerous papers citing review articles and review articles citing papers that cited other papers and so-on) it was discovered that the number came from a book written decades ago. The guy who wrote it had decided on the number because it sounded about right and people had been faithfully using it ever since. It was very popular, perhaps because it led to good predictions of human drug characteristics from in vitro (lab) data. All of the other numbers in the literature that may have come from more reliable sources didn’t give quite as good predictions!
MedsVsTherapy said,
August 10, 2009 at 3:08 pm
The BMJ Moreno analysis looks cool – I have not read it yet, but the pictures are nice! 😉 They fit with what I know from the Turner NEJM study, plus what I know about depression and its various treatments, psychosocial and pharmacological.
My comment is this: I hope that the Moreno analysis, and the Turner analysis, have not dramatically raised the standard for the level of proof to demonstrate that someone is trying to lie to you and mislead you. These emerging analytic methods brings with them the down-side of the “p level:” the possibility that we will abandon common sense, and a decent review of the actual data, the possibility that we will abandon Othello’s “ocular proof,” and instead wait for a “contour enhanced funnel plot” before being able to point out subterfuge, to “call BS.”
What if yet another pharmaceutical company gets taken to court, and a professional-witness statistician for Pharma demonstrates that the accepted criteria for deception includes a “contour enhanced funnel plot”?
I knew, long before the Turner meta-analysis and the PLoS ghost-writer analysis, that there was pharma-funded funny business going on. I knew drugs were being over-promoted. I knew they were going well beyond their data.
I hope the “contour funnel plot,” and other such analyses, do not claim precedence over the “intra-ocular significance test” — when the undeniable significance hits you between the eyes. While these analyses are kind of putting the nail in the coffin on scams such as those that continue to be carried out by Big Pharma, it will be terrible if the only way to fight back is to use some awesome, heady statistical analysis. Reminds me of how we all came to believe we could develop no knowledge without a “LISREL” model, until those techniques fell out of the vanguard and into their proper places.
I have read studies that have surmised some theoretical relation among variables because the SEM indicated that the “fit” improved when some path was added or withdrawn — yet the raw material, the zero-order correlations among the measures involved, were never anything to write home about themselves, much less build up a complex network of effects. Yet the “p” level for the model could be quite impressive.
The hallmark of science is skepticism. Descartes doubted. But it is possible that Descartes should have also stressed honesty. Science, including in the name of health care, requires honesty. A p-level does not substitute for honesty, and neither does FDA approval, or expert opinion, or impact factor. Just as we should hold decent skepticism of any assertion and any theory, we should also be skeptical of any p level, and any funnel plot.
When we get distracted away from plain old skepticism, the scallawags will find a way to exploit whatever else we might substitute in the place of plain old skepticism.
You say that thimerosal casuses autism? Prove it. You say that homeopsychovibrational water cures arthritis? Prove it. And, ultimately, the strength of your proof cannot depend a “p level.” Or a RMSEA, or a CFI, or an I-squared. Or publication in a highly cited journal.
SteveGJ said,
August 10, 2009 at 4:02 pm
On the subject of “myths” or rather, spurious precision, then it is quite stunning how such things as the recommendced maximum guideline amounts of sugar and alcohol intake go from being a compromise recommendation made by a commitee somewhere based on answering an impossible question at government behest (nobody really knows what the “safe” level of many of these components of the diet are, if only because individual variation in tolerance is huge) to something viewed as scientifically proven. In fact they are guidelines, and is ome cases aren’t much better than informed guesses in many cases. Somehow many pressure groups and media outlets don’t seem inclined to trust the public with this bit of information.
We then up with the position that Chris Hoy is vilified by pressure groups as promoting junk food because he is advertising bran flakes and that the (dry eight) sugar content exceeds some arbitrary percentage figure that the FSA has come up with. In fact it was a complete nonsense. Firstly nobody that I know of actually eats dry Bran Flakes (it’s an ingredient of a meal, not the whole meal). Secondly the absolute amounts eaten in a typical portion are a very small proportion of the daily guideline amount and, thirdly, there seems to be absolutely no evidence at all that eating the damn things is bad for you. In fact if you read the (much more informative) US sites on bran flakes, they positively recommend the things. So who is right? The FSA or the US authorities? it doesn’t stop the media and many authorities treating government numbers as somehow definitive.
A similar thing happens with units of alcohol – there is prescious little agreement between different countries over what a unit of alcohol is (try and find out what it is in this country is bad enough – from what I can work out it is 10gms), let alone how many is a reasonable allowance. Then there is the government definition of “binge” drinking, which the media adopts and which I have failed to find any real rationale for.
Note, none of this is to deny that excess indulgence is not an issue, just that there are mythical levels of precision assigned to some elements of diet (some to the point of being downright misleading).
ex89158 said,
August 10, 2009 at 9:58 pm
Smart wrote, in part, “And the above evidence probably accounts for the fact that amalgam has this year been banned in Sweden, Norway and Denmark, and that last year the US FDA was forced by legal challenge into flipping from its asserting safety of amalgam to instead issuing warnings of its neurotoxicity instead.”
No, it probably does not account for anything; in fact, it certainly does not. Sweden, Norway, and Denmark banned dental amalgam not because of any risk to patients, but as an occupational safety measure to protect workers in dental offices. These workers are exposed to large amounts of mercury, much more than any patient could absorb from fillings. These countries banned mercury amalgam to protect dental office workers. Notice that in none of the above mentioned countries has any government agency or medical, or dental society advocated removing amalgam fillings. They would have done this if patient safety had been the concern, but it is not.
The U.S. FDA acknowledging the neurotoxicity of mercury proves nothing, because their statement is merely a repetition of what we have known for over a century: that mercury is a neurotoxin IN HIGH CONCENTRATIONS. The FDA has the authority to regulate, and even ban, dental products, and yet it still licenses mercury amalgam for dental reconstruction, because the science shows that the small concentrations of mercury people absorb from fillings is below the threshold of neurotoxicity, and that people with many fillings are no more prone to any illness than people with few or none.
smart said,
August 11, 2009 at 12:14 am
I notice that ex89158 manages to write so many words in supposed reply to my above, and yet completely fails to even mention (let alone engage with) the key points about the way that Ben Goldacre, the Chief Dental Officer, and the SCENIHR report all fail to mention any of the huge evidence that runs counter to their falsehood. One has to wonder whether these people are consciously deceiving themselves or instead just have some peculiar mental block which prevents them seeing any fact which fails to confirm their prior assumptions. (There is a third possibility that they are engaged in straightforward deceit.) Either way it looks like a waste of time trying to engage with them as they just persist in pretending (or whatever is the correct word for their process).
smart said,
August 11, 2009 at 12:23 am
ex89158 asserts: “the science shows that the small concentrations of mercury people absorb from fillings is below the threshold of neurotoxicity, and that people with many fillings are no more prone to any illness than people with few or none.”
Well, if you deliberately ignore those 26 studies of 6000 patients, along with all the other substantial evidence of harm cited by Hanson, then, pseudo-yes. Proof by pretending. Furthermore it’s been shown that most of the body burden of merc comes from dental fillings (and it is not cancelled out by selenium as is that from fish).
The reason no government is recommending removal of fillings is because (1) it would be a huge task, (2) any such announcement would result in a huge panic, and (3) if removal is not done carefully enough it would cause more harm than good.
ptermx said,
August 11, 2009 at 12:43 pm
Ben concludes his article by suggesting that as a remedy for the “systematic and self-reinforcing distortion” of biased and misleading citation, “ghost writing should be stopped”.
I don’t see that the outlawing of ghost authorship will in itself achieve much. Would a systematic and objective review be any less valuable if it turned out to have been ghosted? Are the effects of bias mitigated if the attribution of authorship is genuine and the author gives every appearance of sincerely believing that what s/he wrote is accurate and objective?
The market for ghost authorship depends on two things: the perception (by those who sponsor the articles) that influence depends on the name of the author as much as what the article says; and the perception (by those who allow their names to appear on articles they didn’t write) that number of publications is more important than quality of research.
These factors relate to the ways in which academic authors generate, justify and exploit the trust that is placed in them. Ghost authorship is a symptom, not the cause, of whatever malady is to be found there.
foofdawg said,
August 12, 2009 at 8:28 pm
Straight from the USofA FDA:
Even in adults and children ages 6 and above who have fifteen or more amalgam surfaces, mercury exposure due to dental amalgam fillings has been found to be far below the lowest levels associated with harm. Clinical studies in adults and children ages 6 and above have also found no link between dental amalgam fillings and health problems.
www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/DentalProducts/DentalAmalgam/default.htm
Now, I’m not a scientist and have not read the articles mentioned by Smart above, however I did take a look at the website provided, and it appears that the original sole function of the IAOMT was to fight against dental amalgam fillings, due to their supposed danger, which makes them a bit biased on this topic in my opinion.
The good news is that nobody is forced to get dental amalgam fillings (that I am aware of), as there are now several alternatives available, and I myself discussed the pros and cons of each with my dentist before getting a tooth filled.
PsyPro said,
August 13, 2009 at 12:14 am
Among the worst in biased summaries and reviews were those supporting the tight linkage between diet and heart disease. This article of some 15 years ago documents the bias very nicely, including referencing articles to asupport the opposite of what they actually found:
RAVNSKOV, U. (1994). QUOTATION BIAS IN REVIEWS OF THE DIET-HEART IDEA
J Clin Epidemiol Vol. 48, No. 5, 713-719, 1995 pp.
smart said,
August 13, 2009 at 1:04 am
foofdawg wrote: Now, I’m not a scientist and have not read the articles mentioned by Smart above, however
“however…”? However what? All that matters in your “reply” is that you happily declare that you wilfully ignore the 26 contrary evidence studies of 6000 patients even when they are pointed out to you. I can think of plenty of unflattering adjectives to follow up here with, but I don’t really think they should be necessary except in the case of the other True Believers of The Cult of The Great Ben for whom it appears there are no words that would make any difference anyway.
smart said,
August 13, 2009 at 1:18 am
Three days ago I pointed out above that Mr Goldacre has himself been guilty of a very serious instance of this very same fault of pretending away all the evidence that runs counter to his thesis of “scare stories” about dental mercury. Three days on, Mr Goldacre has had nothing to say about the matter. And several of his true believers have replied to my posts but have conspicuously consistently failed to express any disapproval of that serious defectiveness of the supposedly great multi-award-winning Mr Goldacre’s output. Indeed we have one, foofdawg who seems actually quite complacent that he himself takes a view without bothering to check out the studies that Mr Goldacre ignored. Personally I’d say the only award that Mr Goldacre really merits is the wooden skull-implant for Extreme Etc Nerve. And as for his favourite notion that the nutri industry is much the same as big pharma, oh, let’s ignore the little detail that many thousands are killed by pharma whereas none have ever been killed by vitamins, even self-dosed.
PsyPro said,
August 13, 2009 at 2:53 am
Smart writes:
“For instance I wrote to the Chief Dental Officer pointing out the 26 studies of 6000 patients, showing exceptional benefits from amalgam removal. These cannot be merely dismissed as placebo effects.”
Really? “exceptional benefits”? and “cannot be merely dismissed a placebo effects”? Virtually all of the alleged benefits can be dismissed as placebo effects, or, even more likely, illusory placebo effects (i.e., not even placebo benefits happened), and they are hardly exceptional. Virtually everyone of these studies consists of little more than testimonials following elective treatment. The few that have any kind of control group (which consists of no removal, but also no treatment of any kind—hardly a placebo control) showed almost the same collective effects. In fact, there is nothing in this highly selective group of studies worth referencing or commenting on. Sorry, Smart, but there is nothing here.
Staphylococcus said,
August 13, 2009 at 3:13 am
@ Smart.
Firstly, I will declare upfront, that I have no feelings on the subject of amalgam fillings one way or the other. I have a couple myself and have no issues with them, but if shown to be bad and a safer alternative presented, I see no reason not to discontinue their use.
As for the meta review you cited, I see several issues. Firstly the author is a member of an advocacy group campaigning against amalgam fillings, which could lead to unintentional bias. A (quick) search of the scientific literature on PubMed revealed that this review has not been published in a peer-reviewed journal.
The author also makes some interesting claims in the introductory section of the paper but fails to cite any of the literature in support of them. For example:
“There is no relation between symptom severity and levels of Hg in blood or urine. Individual sensitivity is more important.”
And:
“Our experience during 20 years has shown that dentists often cheat; all amalgam is not removed but only part of it and the rest is covered with composite plastics. The patients will often first become better but gradually the symptoms recur. Amalgam is regularly present beneath gold-crowns and – bridges and gold-amalgam contacts are often the worst causes of ill health. There are also brass screw posts with a thin and porous gold plating, often directly cemented into amalgam fillings. Such constructions often result in periapical inflammations.”
The second quote is, in my opinion, somewhat disturbing as this is supposed to be a meta review and he is invoking unsubstantiated anecdotal evidence to explain the persistence of symptoms.
Before I move on I must declare that I have not had time to read all the studies present. However, I wonder how you can so easily sweep aside the placebo effect from these studies as you did in you initial post. It’s obvious that no ethics committee on the planet would allow you to do a properly controlled trial with a placebo group that were told that their amalgam was to be removed and were given a sham act of removal without it actually taking place.
This leaves these studies wide open to recall bias, regression to the mean, the placebo effect of dramatic intervention, etc. Additionally the author of the meta review states himself that “The reason these patients visit the dentist is to have amalgam removed. A minority of patients remove amalgam to reduce the risk of future problems, for the unpleasant feeling of having a load of poison in the mouth, for aesthetic or environmental reasons.”
From what I have read, I would suggest that this review has been ignored for good scientific reasons.
Staphylococcus said,
August 13, 2009 at 3:20 am
P.S. I am not a member of the “Super fantastic fanclub of Dr Ben Goldacre” or whatever they call themselves at the moment.
I am a scientist and I’ll criticise whether it confers with Goldacre’s opinion or not.
PsyPro said,
August 13, 2009 at 3:32 am
Staphylococcus said,
“August 13, 2009 at 3:20 am
P.S. I am not a member of the “Super fantastic fanclub of Dr Ben Goldacre” or whatever they call themselves at the moment.
I am a scientist and I’ll criticise whether it confers with Goldacre’s opinion or not.”
I am also a scientist, and have no idea of any such club, but am at any rate not a “member” except in so far as Dr. Goldacre and I share a sceptical bent, and I appreciate the hard work he often puts into his attempts to debunk and question. At the same time, it would be great fun to catch the skeptics’ skeptic in some blindness or bias—indeed, I suspect Dr. Goldacre would appreciate and, indeed, highlight such lapses (as only proving his point that we are all susceptible to such blinders) and the irony. There is no conspiracy here.
Staphylococcus said,
August 13, 2009 at 11:01 am
PsyPro, it would seem we agree on all points. It was a tongue in cheek comment ment to undermine any sillyness aimed at who’s “side” I’m on (the side of reason, if anyone’s confused) 🙂
Text really is a sucky communication medium.
Wyatt Earp said,
August 13, 2009 at 6:39 pm
Oooh, if you google “Ben Goldacre fan club” you get some real treats. In particular, there’s a page on BG on a site called “Whale” that’s not to be missed.
Chirimolla said,
August 15, 2009 at 2:50 pm
Us philosophers are way ahead of you. Philosophers of science have been talking about social networks influencing scientific research and consensus since Paul Feyerabend published his seminal “Against Method” in 1970.
It’s a shame that the only philosopher of science that scientists ever seem to have heard of is Karl Popper. His ‘falsification’ theory of scientific method provides a great ego-boost to science, but has not been highly regarded by philosophers for a very long time.
If scientists cast their net a little more widely, who knows, they may actually learn something.
Ben Goldacre said,
August 15, 2009 at 5:26 pm
chirimolla:
“Us philosophers are way ahead of you. Philosophers of science have been talking about social networks influencing scientific research and consensus since Paul Feyerabend published his seminal “Against Method” in 1970… If scientists cast their net a little more widely, who knows, they may actually learn something.”
social scientists have been studying social networks and the diffusion of ideas since the 1940s, the classic papers on this are:
Ryan, B. and N. C. Gross (1943), “The Diffusion
of Hybrid Seed Corn in Two Iowa Communities.”
Rural Sociology, 8(1): 15-24.
and this one on the spread of medical innovations, which is very commonly cited:
Coleman, J. S., E. Katz and H. Menzel (1966),
Medical Innovation: A Diffusion Study,
Inidianapolis, The Bobbs-Merril Company, Inc.
if philosophers of science such as yourself were slightly less pompous, spent less time trying to take credit for other peoples ideas, and actually bothered to read the literature from the fields they claim to critique, they would look less like dicks.
frisbee said,
August 22, 2009 at 12:39 am
Yes I’m sure there are “myths and misapprehensions” in science and in science reporting. Ghost writing should be illegal. Selecting reference papers simply because they support your data is wrong.
But you are assuming in this article that all papers are equal and this is not the case. As a reseacher you will come across papers that are rubbish. The methodology may be flawed, you may have discussed in person with the authors and know that their study had overlooked something crucial, then see their work published a year later with no improvement. What do you do? You have a couple of choices. Refer to the work and add a lengthy critique of why it is wrong: spend 500 words on something that is irrelevant and add 10 references to your reference list.
Or ignore it.
We have word maximum lengths and a restriction on the number of citations when publishing papers in journals. There are probably more important things for us to discuss than a rubbish paper. Many of the people in the field will know that the paper is rubbish anyway and eventually (hopefully) future studies will provide new evidence fr what we believe to be true. Its a kind of pragmatism and there are problems in doing this. Particularly if somebody with no critical understanding of the field uses network theory (which is indiscriminate) to make a judgment of data based based on citations.
frisbee said,
August 22, 2009 at 12:44 am
clarification: When I said “eventually (hopefully) future studies will provide new evidence f[o]r what we believe to be true” what I meant was that without having to redo old experiments, future experiment will move forward on an idea in whatever direction the data take you. I did not mean to imply a hope that future studies would always support my hypotheses.
smart said,
August 28, 2009 at 2:14 am
I’m tempted to ignore the further reactions to my earlier comment due to the pathetically low intellectual level. But anyway.
Staphylococcus said: Firstly the author is a member of an advocacy group campaigning against amalgam fillings, which could lead to unintentional bias. A (quick) search of the scientific literature on PubMed revealed that this review has not been published in a peer-reviewed journal.
But so what? Most medical studies are authored by members of the (pseudo-)scientific corporate medical establishment and the peer-reviewed journals largely block the publication of anything that challenges their profit-making dogmas. So if we apply your reasoning on an equal basis then there’s no data left to discuss. Are you really an idiot? No, just severely, hopelessly baised in favour of corporate med, like most of Mr Goldacre’s uncritical acolytes (plus himself) I suggest.
Furthermore it matters not where it is published. The main function of the review is to draw together the results of 25 studies for the reader to judge for themself. He does add some sensible commentary which can be appreciated on its own merits if you are not already a corporatised bonehead.
The author also makes some interesting claims in the introductory section of the paper but fails to cite any of the literature in support of them. For example:
“There is no relation between symptom severity and levels of Hg in blood or urine. Individual sensitivity is more important.”
But that is a well-established elementary factoid of mercury toxicity and it is not critical to the hazard evidence anyway. Hanson has better things to get on with than spoon-feed the scientific infants here. Why waste everyone’s time with this pseudo-critique when you yourself have not bothered to check for evidence that he was wrong there?
“Our experience during 20 years has shown that dentists often cheat;
[………….]
The second quote is, in my opinion, somewhat disturbing as this is supposed to be a meta review and he is invoking unsubstantiated anecdotal evidence to explain the persistence of symptoms.
I trust you won’t expect the sun to rise next morning or bother to eat food again in the next few years, as these too are matters of mere unsubstantiated anecdotal evidence never confirmed by any proper published scientific study. Truly infantile reasoning there. I suggest participating in some randomised trials of parachute effectiveness: www.bmj.com/cgi/content/full/327/7429/1459
I wonder how you can so easily sweep aside the placebo effect from these studies as you did in you initial post.
From what I have read, I would suggest that this review has been ignored for good scientific reasons.
and PsyPro likewise said:
Virtually all of the alleged benefits can be dismissed as placebo effects, or, even more likely, illusory placebo effects [….] In fact, there is nothing in this highly selective group of studies worth referencing or commenting on. Sorry, Smart, but there is nothing here.
Dear outstanding geniuses, Hanson’s document itself discusses this matter quite adequately. It is pointed out that many patients came to the bothersome task of amalgam removal as the endpoint of a great many years of illness unhelped by any other interventions; and that improvements were maintained over many years after. Some such as Hesham el-Essawy have recovered from seriously non-placebic illnesses such as heart problems. Only a seriously sloppy reader can conclude that all this can be dismissed as placebo.
I myself have been mentally and physically trashed for 40 years since this curse was launched on me. Age 14, Exams FIRST in Maths, FIRST in Physics, FIRST in Chemistry, FIRST in Geography, third in French, fifth in Latin. Age 17, not even graded in any of the three A-levels enrolled to study.
And since then a lifetime of benefits-dependency.
In this sort of context, your cheap casual presumptions are very offensive.
Futhermore, and this is a key point in all this, the whole point was not the merits of the Hanson etc evidence, but the fact that all the pro-amalgam-quacks (including so-so-so-superior Mr Goldacre) consistently LIE these studies into non-existence. For instance in 72 pages of SCENIHR not the slightest mention of the many studies which just happen to challenge the safety of amalgam. These facts brook no discussion. Which is probably why Mr Goldacre has only commented here on other trivia in the meantime and seems content to rely on those half-baked responses to permit his continued silence on this serious challenge to his own outrageously offensive slurs about “scare stories” that “come round from time to time”. What a stinker of a rubber duck.
Hiranyaka said,
September 18, 2009 at 2:35 pm
Dear Smart
The dental amalgam controversy is certainly an attention-grabbing medical debate. There is evidence of sickness induced by mercury toxicity, and in some cases it may be related to dental fillings. Thus for some patients amalgam removal accompanied by proper treatments should alleviate their symptoms. However, not all studies are conclusive in this regard, including some of those your opinion is based on. Indeed, proper controls for the effect of amalgam removal are often wanting. As it follows, public health policy varies a lot between countries. Nevertheless, one could argue that as precautionary principle, use of mercury should be banned, although for various reason alternatives are not ideal either.
I notice however that you seem to be under the assumption that using capital letters and insulting people will convince readers of the sincerity of your claims. Rather the contrary. If other advocates of mercury ban display the same attitude to address relevant public health authorities, small wonder that their campaign has met such a moderate success so far.
Staphylococcus said,
September 29, 2009 at 10:37 am
Smart said:
“But so what? Most medical studies are authored by members of the (pseudo-)scientific corporate medical establishment and the peer-reviewed journals largely block the publication of anything that challenges their profit-making dogmas. So if we apply your reasoning on an equal basis then there’s no data left to discuss. Are you really an idiot? No, just severely, hopelessly baised in favour of corporate med, like most of Mr Goldacre’s uncritical acolytes (plus himself) I suggest.”
The difference is that most studies actually cite the evidence and don’t just pull a bunch of “facts” out of their arseholes. It does matter where you are published (to a certain degree) because it means that your paper has at the very least been through peer review. Whilst not perfect, the peer review system would certainly rip this pathetic attempt at a meta review to shreds for the author’s continual insertion of personal bias, failing to cite sources, use of anecdotal evidence and generally poor reasoning skills.
Do you have any evidence to support these claims of a great conspiracy theory amongst journals? I highly doubt it. It’s besides the point, anyway as there are now plenty of independent, online publications to publish with that aren’t connected to the dreaded “BigPharma” boogie man.
Incidentally, I’m not biased towards corporate med. Why should I be? They don’t pay me anything.
“But that is a well-established elementary factoid of mercury toxicity and it is not critical to the hazard evidence anyway. Hanson has better things to get on with than spoon-feed the scientific infants here. Why waste everyone’s time with this pseudo-critique when you yourself have not bothered to check for evidence that he was wrong there?”
The point is, I shouldn’t have to. He should cite his sources so that I can go and read the original study and see if the conclusions they have drawn are reasonable. This is basic stuff.
“I trust you won’t expect the sun to rise next morning or bother to eat food again in the next few years, as these too are matters of mere unsubstantiated anecdotal evidence never confirmed by any proper published scientific study.”
Are you serious? Oh dear. I mean obviously the turning of the earth and the need for sustenance (two phenomena which have been shown through the scientific method, I should add) are on the same level as manufacturing “evidence” for a puerile attempt at a meta review. I think your analogy generator is broken.
“Only a seriously sloppy reader can conclude that all this can be dismissed as placebo.”
Except, that, I, um, didn’t. But that’s ok, maybe you can turn your irrelevant English test scores toward some reading comprehension instead of consistently insulting anyone who dares to disagree with you. What I did say was the the placebo effect cannot be dismissed in its entirety. The only way to show this would be to conduct a properly controlled trial and actually demonstrate the problems. Until then, you cannot dismiss it (well you can, but your reasoning would be flawed) which means your claims amount to very little.
“And since then a lifetime of benefits-dependency.
In this sort of context, your cheap casual presumptions are very offensive.”
Ah, so we finally get to the stick up your arse. Before you accuse someone else of being biased toward “BigPharma” (what a lark! How are those presumptions working out for you?), you should take a good look at your own rose-tinted glasses. Perhaps you really were affected by the mercury in you fillings. Who can say for sure? What we do know is that really, really wanting it to be true doesn’t make it so.
So I’ll reiterate what I was trying to say before (and which you seem to take as a personal insult for some reason…):
Your meta review was rubbish for good scientific reasons. Quite apart from the aforementioned use of anecdotes and other poor “evidence”, the biggest problem with the whole thing was the complete lack of controls. If you cannot control for variables, then you cannot draw solid conclusions and your data are meaningless. That’s it. You can whine about whatever you like. Insult me some more (as I’m sure you will). But that’s the rub and it’s what real scientists have to deal with every day. Do my data ACTUALLY support my conclusions or could there be some other explanation? These studies have not met this burden and so they have been loudly ignored by everyone. And so they should have.
smart said,
November 10, 2009 at 4:30 am
These longwinded further replies are entirely besides the point which was that Ben Goldacre (and all the other amalgam quacks) are guilty of the very same offence of failing to even mention any evidence that might go against them (whether soundly or not). And I and others have better things to do with my time than answer the rest of the half-baked falsehoods contained in these additional irrelevances. Mr Goldacre’s failure to respond to my point surely proves him to be the nasty hypocritical charlatan that he is.
wayscj said,
November 21, 2009 at 6:36 am
ed hardy ed hardy
ed hardy clothing ed hardy clothing
ed hardy shop ed hardy shop
christian audigier christian audigier
ed hardy cheap ed hardy cheap
ed hardy outlet ed hardy outlet
ed hardy sale ed hardy sale
ed hardy store ed hardy store
ed hardy mens ed hardy mens
ed hardy womens ed hardy womens
ed hardy kids ed hardy kids ed hardy kids