Note: The Guardian accidentally edited this column such that the last paragraph contained an untrue statement. I have emailed the readers editor for a correction.
Ben Goldacre
The Guardian,
Saturday May 10 2008
So basically I sit here with a big bag of standard tools from the world of evidence, and wait for stories to come along which allow me to deliver a 600 word lecture on them. Sit tight, this one’s slightly complicated. In America last week the papers went crazy: artificial blood products cause a 30% increase in deaths, and a 2.7-fold increase in heart attacks, according to a new meta-analysis in the Journal of the American Medical Association. There is, incidentally, a trial of these products still ongoing in the UK.
In many respects the first part of this story was similar to the antidepressants scandal: a large number of trials had been done, over a decade, but the results had not been published, languishing unseen in the FDA’s files. Many of the companies involved even declined to hand over data to the National Institutes for Health researchers doing the meta-analysis. When Biopure declined, the researchers were forced to rely on a pooled analysis of their data from the FDA with inadequate information. The data from 2 trials of PolyHeme were only available from Northfield Laboratories’ press releases, and a request for more detailed information was again declined. It’s quite possible there are also trials which have never been made public.
But the bad behaviour of firms not being open is never as interesting as the science. A meta-analysis is a study where you take lots of individual trials, some of which may be quite small, and effectively put all the figures into one big spreadsheet. This allows you to get a more accurate answer about an outcome, like death, because the numbers of patients involved are then much larger. They joy of meta-analyses is that they can help to avoid what is called a “type II error”: missing a genuine finding.
This meta-analysis was a perfect example of how useful the tool can be. Individually, none of these artificial blood trials produced a damning result, largely because they were too small to do so: they were “underpowered” for that purpose, with small numbers of patients, and even smaller numbers of deaths. They sometimes showed an excess of deaths in the artifical blood group, and sometimes in the normal donor blood group, but these differences were never “statistically significant”. Only when the numbers were pooled was the dramatic risk revealed.
![clip_image002[5]](http://www.badscience.net/wp-content/uploads/clip-image0025.jpg)
But that’s not the interesting bit. Academic researchers have been talking about something called “cumulative meta-analysis” for 25 years: essentially, you run a rolling meta-analysis on a given intervention, and each time a new trial is completed, you plug the figures in to get your updated pooled result, to get a feel for the trend of where the results are headed, and most usefully, have a good chance of spotting a statistically significant answer – for good or bad – as soon as it becomes apparent, without risking lives on further unnecessary research. The NIH researchers did a cumulative meta-analysis, adding in each study year by year, and found that if this had been done all along, the answer – that artificial blood products increase heart attacks and death – would have been apparent in 2000. The subsequent studies were arguably unethical, and ethics committees, if they had been given access to this information, might not have given permission to throw more good lives after bad.
![clip_image002[16]](http://www.badscience.net/wp-content/uploads/clip-image00216.gif)
But for all the venality of hidden data, this issue transcends good and evil: because important answers can also be missed simply because people don’t look. The earliest example of a cumulative meta-analysis is from 1981, in a paper which looked at the routine use of antibiotics during surgery to prevent infection. They showed, by doing a cumulative meta-analysis, that research had continued to be done for years after antibiotics had been shown to be effective, not only at preventing infections during operations, but also in reducing the death rate afterwards. Because this was missed, thousands of patients were randomly allocated to receive placebo control pills in studies, denied acccess to a treatment which was known to be effective, and many of them will have died unnecessarily, simply for the lack of a bit of clever number crunching.
Ideas like cumulative meta-analysis from the world of evidence have saved countless lives, and they could save many more. They are clever and they are fascinating. They are the same tools you hear rubbished by big pharma, by homeopaths, and by lobbyists from the $56bn food supplement industry. But you will never find them celebrated, anywhere, in popular culture.
References:
In the links above as ever. Reproduced below for those who like it old skool.
This is an excellent review on cumulative meta-analyses:
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care
Joseph Lau, Christopher H. Schmid and Thomas C. Chalmers
Journal of Clinical Epidemiology
Volume 48, Issue 1, January 1995, Pages 45-57
dx.doi.org/10.1016/0895-4356(94)00106-Z
Here’s the paper:
Cell-Free Hemoglobin-Based Blood Substitutes and Risk of Myocardial Infarction and Death. A Meta-analysis
Charles Natanson, MD; Steven J. Kern, BS; Peter Lurie, MD, MPH; Steven M. Banks, PhD{dagger}; Sidney M. Wolfe, MD
JAMA. 2008;299(19):(doi:10.1001/jama.299.19.jrv80007).
jama.ama-assn.org/cgi/content/full/299.19.jrv80007
Here’s the caustic editorial:
The Future of Clinical Trials Evaluating Blood Substitutes
Dean A. Fergusson, MHA, PhD; Lauralyn McIntyre, MD, MSc
JAMA. 2008;299(19):(doi:10.1001/jama.299.19.jed80027). —
jama.ama-assn.org/cgi/content/full/299.19.jed80027
Here’s the ancient antibiotics paper if you enjoy academic archaeology:
A survey of clinical trials of antibiotic prophylaxis in colon surgery: evidence against further use of no-treatment controls.
ML Baum, DS Anish, TC Chalmers, HS Sacks, H Smith and RM Fagerstrom,
New Engl J Med 305 (1981), pp. 795–799.
And here’s the best book ever written on evidence based medicine for a general audience:
Testing treatments. better research for better healthcare.
Evans I, Thornton H, Chalmers I.
London: The British Library, 2006. ISBN 0 712 3 4909 X.
You can read the full text for free online here.
I should say that there is a bit of a problem with cumulative meta-analyses which I thought of on the loo yesterday. We laugh at people who stop a 6 month trial at 4 and a half months because they got a good result, like we laugh at people who extend a 6 month trial to 12 months because the results aren’t positive. Used foolishly and robotically a cumulative meta-analysis could be subject to the same criticism. That’s why evidence based medicine isn’t about using data robotically.
Munin said,
May 10, 2008 at 8:56 am
In case anyone else was wondering, the offending paragraph in the Guardian reads:
“Cumulative meta-analyses have saved countless lives, and they could save many more. They are clever and they are fascinating. They are the same tools you hear rubbished by big pharma, by homeopaths and by lobbyists from the $56bn food supplement industry. And you will never find them celebrated, anywhere, in popular culture.”
I assume the error concerns the first sentence, where they replaced “Ideas like cumulative meta-analysis from the world of evidence” with “Cumulative meta-analyses”.
They also replaced “But” with “And” in the final sentence. Somehow this scans better but I don’t know why.
Munin – visiting the Guardian website so you don’t have to. Oh, on you go then.
www.guardian.co.uk/science/2008/may/10/medicalresearch.health
Robert Carnegie said,
May 10, 2008 at 10:41 am
I may have forgotten why it is that “We laugh at people who stop a 6 month trial at 4 and a half months because they got a good result, like we laugh at people who extend a 6 month trial to 12 months because the results aren’t positive.” What’s funny?
Can’t it be said that a trial is looking for a signal, positive or negative, strong or weak – and statistical detection of the signal may come sooner or later?
Or is it that the 4 and a half month trial will for instance miss subjects who suddenly die after 5-6 months, or that changing the rules during a trial is cheating and is not statistically valid, or that one trial isn’t important enough to justify such a decision anyway, it’s the meta-analysis of different experimenters’ trials that matters?
I want to mention some other problems that I perceive: meta-analysis may be lumping together different data, as in this case where any or all artificial blood products or blood substitutes may be covered, regardless of their differences (I presume there are differences) – I suppose there is an argument for carefully designing trials to be compatible in meta-analysis; and when successive runs of the same cumulative meta-analysis are published to the general public, it may be perceived as new evidence each time, huge trials of hundreds of thousands of subjects. But meta-analysis is not new data at all, that’s the point, and this year’s cumulative meta-analysis is not itself to be added to last year’s cumulative meta-analysis, last year’s goes in the bin and is replaced by the new one. There also is a danger that either a trend will be perceived in cumulative meta-analysis, so that a future value is expected to lie beyond the range of reported meaurements – say you report 1.4, 1.5, 1.55, people expect you to keep going up to 1.7 or 1.8 but the true prediction is the latest actual value, the 1.55. Conversely, if you say that more research is needed, people may expect your finding to be confounded.
One other thing – if trials separated by long times are compared, what about variables that you haven’t accounted for? We live different lives to previous generations – passive smoking and lead in petrol have come and gone, we’re more obese, we have different vaccinations, we’re exposed to ozone and fine dust from computer printers and copiers, the planet’s magnetic field is weaker. Of course trials include a control group that is subject to all the same factors except for what the trial is for, but is that enough? Should older trials be weighted lower, treated as stale?
Ben Goldacre said,
May 10, 2008 at 12:51 pm
i’m surprised you think these trials weren’t comparable for the purpose of a meta-analysis looking at death. they were all trials comparing similar products against control (eg donor blood), and they pooled the deaths and heart attacks.
here is the blobbogram which you may not have looked at.
i think it’s really rather persuasive that there was a hint of danger in almost all the trials which only became statistically significant in a meta-analysis.
the cumulative meta-analysis moreso:
these are potentially useful products in cases where no donor blood is available, eg a battlefield, but i certainly wouldn’t choose them over donor blood for myself.
you’re welcome to, and i wish you the very best of luck!
sakent said,
May 10, 2008 at 10:43 pm
What are the risks of using donor blood (eg infection) and do the risks of using artificial substitutes outweight this?
nick_127 said,
May 11, 2008 at 3:54 pm
“Real” blood products may not be that safe either, and I don’t just mean because of the risks of BBIs. New Scientist ran an article in their 26th of April edition (UK)about the increased risk of death following blood transfusions. The article mentions several studies but only references 3 properly. I’m an amateur science geek so I haven’t got an ATHENS or MEDLINE account, but the references are below for anyone who has.
Journal of the American Medical Association, vol 292, p1555.
Circulation, vol 116, p2544.
The New England Journal of Medicine, vol 358, p1229.
Ben Goldacre said,
May 11, 2008 at 5:00 pm
mm
i think what you might be missing is that the majority of patients pooled in this meta-analysis were from trials comparing artificial blood with normal transfusion blood products (6/16 RCTs but the two biggest trials by far were using blood as a control)
jama.ama-assn.org/cgi/content/full/299.19.jrv80007v1/JRV80007T2
muscleman said,
May 11, 2008 at 7:20 pm
The New Scientist article was more about the risks of routine blood transfusions to boost haematocrit than emergency room transfusions after massive injury. So a different question is being asked in that work vs the cumulative analysis your article is about. The suggestion is that maybe transfusions are not as risk free as we thought and gives some possible reasons like stiffness of stored erythrocites, cytokines and low NO2 levels.
A sidebar makes the point that the JW’s insistence on bloodless surgery has shown that many transfusions during or post surgery are avoidable. The religious can have their uses 😉
Delster said,
May 12, 2008 at 12:20 pm
@nick_127
There is of course a risk of infections etc from donated blood products but there are processes in places to ensure the risk is as low as possible.
Where the meta-analysis compares artificial vs real blood product this risk is already incorporated into the trial as the real product comes with this risk built in as it were.
I think one idea that should be used is for patients with planned surgery should make donations in advance so they get their own blood back. No risk of rejection / incompatability there but it’s not something i’ve heard being offered.
Obviously ER type trauma will not have this option which is why i’ll keep giving blood 🙂
Nick Bland said,
May 12, 2008 at 4:50 pm
The risk of infection from donated blood is small (until something new comes along that we don’t screen for e.g. HIV) but that risk is not “built into” this study as data was only collected for up to 30 days after infection. Any infection that takes longer to kill you or is debilitating to one extent or another will not be included in this data.
I don’t think this really detracts from the point of the paper but as soon as a paper becomes a news story its important to point out the restrictions of the studies. these restrictions are usually freely acknowledge by the authors but rarely by journalists (no offence to any journalists inteneded)
Robert Carnegie said,
May 13, 2008 at 1:36 am
There were two topics in the article; the dangers of artificial blood products, and the wonderfulness of meta-analysis. I think meta-analysis has potential weaknesses as I suggested, not necessarily so much in this particular case. But in the depression and SSRI meta-analysis, I as a sometime patient am dissatisfied with meta-analysis that treats the pill I’m taking and the other pills that I am not taking but other people are, as the same. It occurs to me sceptically that it allows the meta-analysing scientist to have a bigger sample and superficially better statistical precision and apparent scientific validity, whereas if they only counted say fluoxetine then it would be a smaller meta-experiment but excluding data arguably irrelevant to fluoxetine users.
defectivebrain said,
June 23, 2008 at 5:06 pm
I blogged about fake blood in January, and described some of the science behind the various designs of faker blood.
defectivebrain.blogspot.com/2008/01/fake-blood-this-is-hospital-not-horror.html
I tried to be as positive as possible about it, but the quest to make fake blood is fraught with difficulty.
Real blood is cheaper, proven to work, and is commonly available.