Bad science
Ben Goldacre
Wednesday July 18, 2007
The Guardian
Whatever you think about Andrew Wakefield, the real villains of the MMR scandal are the media. Just one week before his GMC hearing, yet another factless “MMR causes autism” news story appeared: and even though it ran on the front page of our very own Observer, I am dismantling it on this page. We’re all grown-ups around here.
The story made three key points: that new research has found an increase in the prevalence of autism to one in 58; that the lead academic on this study was so concerned he suggested raising the finding with public heath officials; and that two “leading researchers” on the team believe that the rise was due to MMR. Within a week the story had been recycled in several national newspapers, and the news pages of at least one academic journal.
But where did the facts come from? I contacted the Autism Research Centre in Cambridge: the study the Observer reported is not finished, and not published. The data has been collected, but it has not been analysed. Unpublished data is the antithesis of what science is about: transparency, where anyone can appraise the methods, and the results, and draw their own conclusions.
This study is the perfect example of why this is important: it was specifically designed to look at how different methods of assessing prevalence affected the final figure. So it is no surprise that one of the results from an early analysis is high, “one in 58”, using techniques which deliberately cast the widest net. But even other figures in the initial analysis were less dramatic, and similar to current estimates, and the Observer admits it was aware of them. It seems it simply cherry picked the single most extreme number and made it a front page splash story.
The Observer is unrepentant: it says it has the “final report”, from 2005. I can’t get it to show it to me but the Cambridge team suspect the paper has seen the last of the quarterly progress reports to the funders. So how did the Observer manage to crowbar MMR into this story?
First, it claimed that the lead researcher, Professor Simon Baron Cohen, “was so concerned by the one in 58 figure that last year he proposed informing public health officials in the county.” Prof Cohen is clear: this is inaccurate and scaremongering.
And the meat? The Observer claims that “two of the academics, leaders in their field, privately believe that the surprisingly high figure [one in 58] may be linked to the use of the controversial MMR vaccine.” This point is repeatedly reiterated, with a couple of other scientists disagreeing to create that familiar, illusory equipoise of scientific opinion which has fuelled the MMR scare in the media for almost a decade now.
But in fact, the two “leading experts” who were concerned about MMR, the “experts”, the “leaders in their field”, were not professors, or fellows, or lecturers: they were research associates. I rang both, and both were very clear that they wouldn’t describe themselves as “leading experts”. One is Fiona Scott, a psychologist and very competent researcher at Cambridge. She said to me: “I absolutely do not think that the rise in autism is related to MMR.” And: “My own daughter is getting vaccinated with the MMR jab on July 17.”
She also said, astonishingly, that the Observer never even spoke to her. And in the Observer’s “readers’ editor” column one whole week later, where the Observer half heartedly addressed some of the criticisms of its piece, the Observer persisted in claiming she believes MMR causes autism: it believes it knows the opinions of this woman better than she knows her own mind. Despite her public protestations. The only voice that Dr Scott could find – bizarrely – was in the online comments underneath the readers’ editor piece, where the Observer continued to call her an MMR “dissenter”, and where she posted an impassioned and slightly desperate message, protesting her support of MMR, and threatening legal action.
That’s one of the leading experts. The other is Carol Stott. She does believe that MMR causes autism (at last). However, she is no longer even a “research associate” at the Autism Research Centre.
Carol Stott works in Dr Andrew Wakefield’s private autism clinic in America, which the Observer failed to mention, and she was also an adviser to the legal team which failed in seeking compensation for parents who believed that MMR caused their child’s autism, which the Observer failed to mention. She was paid £100,000 of public money for her services. She says her objectivity was not affected by the sum, but even so this seems an astonishing pair of facts for the Observer to leave out.
And were Stott’s views private, or secret, or new? Hardly. Stott is so committed to the cause against MMR that when the investigative journalist Brian Deer exposed the legal payouts in 2004, although she had no prior contact with him, she spontaneously fired off a long series of sweary emails titled “game on”: “Try me, shit head … Believe me, you will lose … so go fuck yourself. Got it yet shit head. Try me … Twathead … waiting … oh yes … Stick that where it feels good. Shit head … well, ur a bit slow on the uptake … Give it time I s’pose. Twat.” And so on.
On the phone I genuinely warmed to her, and she regrets that many people have fallen into entrenched positions on MMR on both sides. But she’s not a leading expert (as she herself agrees); she’s not a sombre Cambridge academic suddenly expressing a fresh concern (her views are very public); and in any case, even she is very clear that this new research reported in the Observer would tell us nothing whatsoever about MMR causing autism.
Nothing has changed, and this scare will never be allowed to die. If we had the right regulatory structures, almost every section of the media would be in the dock, alongside Wakefield.
Please send your bad science to bad.science@guardian.co.uk
Oh ye of little faith.
I think we should recognise that criticising a paper within the same stable is a very unusual thing, and I think it shows phenomenal integrity that the Guardian was willing to give me the opportunity to write about this as I would about any other newspaper story that was so wrong, even if it took some time. I honestly don’t think it could happen anywhere else, especially as I was criticising the central facts of a news story, not a comment or opinion piece.
On the other hand I might be completely wide of the mark and this is a non-event. I don’t know, I don’t know much about media politics generally, and I don’t go into the building much (I’d be scared of getting a wedgie in the back staircase off the Observer people now).
Either way the most important thing is that the Observer’s was a very prominent story, it was picked up all over the place (can you believe the BMJ quoted that one in 58 figure as fact in a news story?), and now there is something in the public domain that puts the facts a little straighter. I hope it doesn’t trigger, er, “reprisals” and I would be very surprised if there’s anything in there that doesn’t stand up.
Building a story out of scary unpublished data has been a recurring feature of the anti-MMR coverage (eg here, while newspapers ignore any reassuring negative risk findings like here and here) and to me that is really the central science- and risk-communication issue arising from this Observer piece. If there hadn’t been a need to get bogged down in the dreary details I would much have preferred to just write about that, but I was on a juggernaut of pulling the piece apart by then.
Mainly of course my indignation was simply a contrivance to get the words “shithead” “fuck” “twat” and “twathead” onto the pages of a national newspaper. The especially good news is that my very lengthy analysis of the Observer piece is being published in the British Medical Journal on Friday, which I am assured after the cut still features twat, fuck, and shithead action. After this has been published, I believe it will be the sweariest thing the BMJ has printed to date. The gauntlet is down.
Analogue said,
July 18, 2007 at 1:04 am
Bravo Ben, bite the hand that feeds you and all that!
Seriously though, keep up the good work. Nice to see that at least some of these irresponsible idiots do get their comeuppance, although as you say there’s a way to go yet…
Andrew Clegg said,
July 18, 2007 at 1:48 am
w00t!
Props to the Guardian for seeing sense, even after a bit of deliberation. I hope the powers that be at the BMJ realise they’ve messed up.
Andrew.
Gimpy said,
July 18, 2007 at 2:25 am
Ben your my hero…………I’m impressed with the Guardian but I’m more impressed with the swearage in the BMJ. That truly is awesome. Of course the MMR stuff is wonderful and all that but gratuitous swearing in serious pubilications is class.
I spent rather too much time today arguing with some JABS member on CiF. Your swearing is a vicarious pleasure.
Filias Cupio said,
July 18, 2007 at 6:06 am
Ben, here is a request for an article:
Some time ago, when BSE was ravaging the UK’s cattle herds, the “establishment” said that it was safe to eat beef from BSE cattle, but a few dissenters said it was not safe.
Currently, the scientists assure us MMR is safe, but a few dissenters say it is not.
In the first case, the dissenters were right, and the claims that it was safe was Bad Science (or at least Wrong Science). In the second case, the claims of danger are Bad Science.
From the public’s point of view, assuming they have access to competent news reporting, how could they tell the difference between these two cases? What signs were there, during the BSE scare, that the assurances of safety were poorly founded?
Bob O'H said,
July 18, 2007 at 7:33 am
I’m all mixed up. I could have sworn it was Wednesday today, not Saturday. Does this mean Harry Potter 7 is for sale at last?
Bob
woodchopper said,
July 18, 2007 at 8:45 am
Bravo to the Gruniad, bravo Ben, and bravo to al who posted online comments and wrote letters.
Filias, you make a good point. While there were was clearly very a big difference between the quality of research on BSE and MMR it would be difficult for a member of the public to tell.
I think that the Governments misreprisentation of the research re. BSE has to take a lot of blame for the public’s mistrust of scientists.
manigen said,
July 18, 2007 at 9:46 am
Congratulations Ben. For a short while there I was concerned about the Guardian’s behaviour, but on balance this looks to have been a good outcome.
stomec said,
July 18, 2007 at 10:09 am
Just to say congratulations, Ben. It was of course my indignant letters to the Observer + Guardian (plus threat to defect to the Indie) that did the trick…
But as you say, it is unusual and heartening to see the Grauniad admitting shome mishtake.
Could we now see more swearing in, ooh, the Lancet? Nature? Personally, I have been desperately looking for a new regulatory gene for years now, just so I could name it twat…
igb said,
July 18, 2007 at 10:33 am
“Some time ago, when BSE was ravaging the UK’s cattle herds, the “establishment” said that it was safe to eat beef from BSE cattle, but a few dissenters said it was not safe.
Currently, the scientists assure us MMR is safe, but a few dissenters say it is not.
In the first case, the dissenters were right,”
Were they? An awful lot of people ate beef from infected herds: in essence, anyone eating beef during the 1980s was at risk, for sure. But the cases of CJD tied to BSE look fairly flakely: often young, sometimes vegetarian. Isn’t the alleged effect being independent of the strength of the stimulus one of Langmuir’s signs of pathological science? It was routinely claimed that a million victims of CJD within fifteen years was a conservative estimate: we’re now in a position where the rate of CJD might be slightly higher than it was, but that in the face of far greater awareness and better diagnosis, and the differential diagnosis for vCJD relative to CJD makes polywater look well-defined.
BSE was a clear and present danger to cows. The evidence for an epidemic, or indeed any cases at all, of BSE being caused by it looks pretty thin, and involves dredging around in the statistical noise floor while people with a financial interest in the problem existing make strident claims. Sound familiar?
igb said,
July 18, 2007 at 10:36 am
“The evidence for an epidemic, or indeed any cases at all, of BSE being caused by it”
CJD, I meant. Of course. Must be the prions getting to me.
stever said,
July 18, 2007 at 10:43 am
excellent. respect to Ben and good on the guardian for doing the right thing.
Munin said,
July 18, 2007 at 11:17 am
Nice work – it will be interesting to see the response from the Observer.
Off topic, but also in today’s Guardian: the SAGE report on the effect of low frequency electric fields on human health hits the shelves today (in case you’re experiencing deja vu, this follows an interim report published in April).
www.guardian.co.uk/uk_news/story/0,,2128647,00.html
“New homes and schools should not be built within 60 metres of high voltage power lines until the link with childhood cancers is better understood by scientists, according to a committee of MPs. They also recommend that home buyers should be provided with information on the level of electromagnetic fields within homes before they buy.”
Woo be, or not woo be?
used to be jdc said,
July 18, 2007 at 1:19 pm
superburger: “If he is found innocent of a few of the alleged examples of misconduct, then he is partially exonerated and can claim to have been probed right.”
I read that as “proved right”, but it’s actually funnier the way you posted it.
superburger said,
July 18, 2007 at 2:30 pm
“He is Prof Baron-Cohen ”
doesn’t he play Ali G?
le canard noir said,
July 18, 2007 at 3:24 pm
Respek!
BSM said,
July 18, 2007 at 4:15 pm
““Some time ago, when BSE was ravaging the UK’s cattle herds, the “establishment” said that it was safe to eat beef from BSE cattle, but a few dissenters said it was not safe.
Currently, the scientists assure us MMR is safe, but a few dissenters say it is not.
In the first case, the dissenters were right,”
Were they?”
I have to say that I’m a bit with the “were they” crowd on this one.
Bear in mind that I did not beef from the mid-eighties until well after I was sure the rules were working properly, so I’m up there with the paranoid self-preservers on this on. But it is now really hard to know whether this was all a storm in a teacup. Was the regulation a sledgehammer to crack a (relative) nut? It certainly had a massive detrimental effect on farming and the UK food chain.
I think the problem with BSE was a bit more subtle than with MMR. The Gubmint was caught out claiming something was perfectly safe, when any fool could have told them, and many non-fools did tell them, that there is no such thing as zero risk and perfect safety. So, to reacquire some semblance of public trust the action they took was then very severe.
I blame Selwyn bloody Gummer for force-feeding his child with a brains and nose burger on national television.
So, were the final actions proportionate to the real risks? I can’t see any way of finally knowing. Were the actions taken appropriate even if the risks were tiny just because it’s a really horrid disease whatever would have been the numbers affected? Dunno.
p.s. ‘m also fairly sure I saw the first ever case of BSE when I was a student. It had all the classic clinical signs that we came to recognise, but post mortem examination of the brain was hampered somewhat by the cow been shot and disposed of following a communication cock-up. Oops.
Despard said,
July 18, 2007 at 4:19 pm
#24: I know it’s pretty toothless, but do you think the Press Complaints Commission would be any use in this situation?
Teek said,
July 18, 2007 at 4:36 pm
has anyone read the comments under Ben’s guardian article on comment is free…?! i dont think i’ve evenr seen such a large number of positive posts, all of them saying well done, great stuff, bravo etc. much more of this and BG won’t be able to fit his head thru the door at Grauniad HQ…!!
just joshing of course, my point is that there are many out there who support the rational and scientific approach to journalism – so why aren’t any of them journalists…?!
pv said,
July 18, 2007 at 6:07 pm
As far as I know the PCC are highly unlikely to be interested in this unless someone like Fiona Scott or Simon Baron-Cohen complain. And even then the complaint would only be about how those individuals are affected. You have to be an affected party and, apparently, the public as a group cannot be included. Even if this whole affair has cost the taxpayer millions and people have died as a result of it the press cannot be held responsible for that, and no-one can seek redress. Which is why the cunts of the press can carry on pretty much regardless. There is precious little regulatory framework to cover this sort of abuse by the press.
It’s all “self-regulation” anyway and it would be like expecting burglars to hand themselves in to a group of senior burglars, after every robbery they commit, for slap on the wrist.
doctormonkey said,
July 18, 2007 at 7:43 pm
both articles are v good, pat on back to BG!
re BSE/new variant CJD (nCJD) – as I understand it we were due to have had an explosion in the numbers of cases of nCJD over recent years after all of the BSE scare a while back, instead I understand that the numbers have fallen. I also believe that our understanding of prion disease is not that great and particularly the transmission of BSE to nCJD – a friend of mine was disecting human brains for a neuro-anatomy BSc and was told that there is no certainty that anything stops prions – gloves, hand washing or even sterilisation! the nCJD thing is still at best an unknown and from my point of view seems to have fallen off the radar but similarities to the MMR/autism thing are probably valid
Robert Carnegie said,
July 18, 2007 at 8:35 pm
Prions have always been with us. Of course there are good prions and bad prions, and the good ones can be turned bad – that’s the trouble, in fact. Until quite recently it wasn’t clear why we have them, but there is new evidence that they protect you against Alzheimer’s disease.
Sporadic, spontaneous Creutzfeld-Jakob disease also seems to have been always with us, usually unnoticed because there have been so many other mysterious ways to sicken and die. Now most of the other mystery illnesses have been uncovered. Creutzfeld-Jakob disease can occur without a cause – so to speak.
But there seems now to be no significant doubt that a minor outbreak of CJD cases with a different profile has occurred recently, now seems to be declining, and science does not strongly differ from common sense in associating a rare but temporarily increasing human brain disease with a widespread animal brain disease when humans are eating the brain tissue of the animals in question.
The precise point at which the nuumber of apparent confirmed cases of a new disease justify a public health programme against it is difficult to choose, particularly if the public health programme interferes with someone’s business. The matter of electricity supplies has just come up again. I think that the matter of BSE and CJD was treated correctly by scientists and also by legislators, although BSE itself should have been taken more seriously.
I also think that the overall risk to health and life of a diet including more than a sparing amount of red meat still outweighs the risk of CJD at its highest – and animal nerve tissue is now mostly excluded from human diet. From animal diet – I don’t know, but I hope so.
simongates said,
July 18, 2007 at 9:16 pm
Something I don’t think has been said is that even if the unpublished Cambridge paper did find a prevalence of 1 in 58 that may very well not provide any strong evidence that the prevalence was higher then expected because the confidence interval around a low prevalence like this will probably be quite wide. I think I read somewhere that the sample size was actually 116 i.e. there were 2 autistic spectrum children. If this is right the 95% confidence interval around the estimate of 1.7% would be 0.5% to 6.0%, which includes pretty much every value that has ever been claimed. So claiming an autism epidemic (let alone one caused by MMR) from these sorts of figures would be misleading even if the numbers were correct.
IanWac said,
July 18, 2007 at 9:33 pm
Nice article Ben.
Seems that the only way that you get any sort of a proper apology from the media in this country is if you’re in your 80s and you wear a crown. Oh, and you have to be Queen too.
RS said,
July 18, 2007 at 10:24 pm
Not sure how gloves can fail to protect against vCJD prion proteins! Although you want to ensure you don’t inhale the stuff.
There is an interesting hypothesis regarding vCJD, prion protein homo/heterozygosity derived from what happened with kuru and suggesting we might be in for some more cases due to a long incubation period.
BSM said,
July 18, 2007 at 10:28 pm
“Seems that the only way that you get any sort of a proper apology from the media in this country is if you’re in your 80s and you wear a crown. Oh, and you have to be Queen too.”
And that’s another bloody funny thing.
I suppose we must accept that the trailer for that BBC programme was misleadingly edited, but she was in a snot about something and that seems not to have attracted any comment.
p.s. Forgot to add earlier: bloody excellent job Ben. The swearing is a triumph
Gimpy said,
July 18, 2007 at 11:46 pm
igb the difference between nvCJD and ‘MMR causes autism’ is that animal models were developed that clearly showed neural degeneration as a result of cow prions in diet.
www.springerlink.com/content/l74x00q787472164/
The link between prions and nvCJS is fairly indisputable whereas the link between MMR and autism is non-existent.
Having said all that nvCJD seems to be a disease where only a tiny amount of genetically susceptible individuals are at risk of acquiring it. Maybe the story was overblown but initially there really wasn’t much research on the issue. The end result of the scare was better animal welfare, better quality processed meat and a better awareness of what makes good quality meat in the consumer. It is hard to identify any benefit from the MMR scare.
DaveF said,
July 19, 2007 at 2:27 am
Well done Ben – I don’t suppose we’ll know what negotiations you had to go thru with the Grauniad, but well done them in the end too. I look forward to seeing your column as the front page lead in Sunday’s Observer – that was what you were holding out for I trust.
nekomatic said,
July 19, 2007 at 10:23 am
“New homes and schools should not be built within 60 metres of high voltage power lines until the link with childhood cancers is better understood by scientists, according to a committee of MPs.”
Concern over health effects of high voltage power lines is not the same thing as concern over low frequency electromagnetic fields. There’s a hypothesis that ionisation of the air by the high voltage can charge microscopic particles and make them more likely to enter people’s lungs, for example. I have no idea what the current (sorry) consensus on that hypothesis is, but it doesn’t involve a direct biological effect of the EM field.
stvb2170 said,
July 19, 2007 at 11:49 am
Is this the scientific equivalent of the sports pages ‘bubble double’. In the sport section a newspaper reports one day that a player is leaving his current team, leading with this story (often as an EXCLUSIVE). The next day they lead with the story that said player is staying.
Did the Observer produce this Bad Science just so that you would be able to write about how bad it was the next week?
Gimpy said,
July 19, 2007 at 12:05 pm
muscleman, I know that it was very difficult to infect the animal models. This could be because the animal models were a flawed representation of the disease, interspecies transfer is easier regarding prions between cows and humans than it is between mice and humans or that because transmission is so rare it is difficult to induce through feeding in a lab sized animal population, or any combination of the above.
This is all besides the main point though that vCJD seems to be real (if thankfully incredibly rare) and the most likely explanation is prions in nervous tissue from BSE infected cows. The link between human prions and CJD is indisputable and so is the link between bovine prions and BSE. This is a wholly different situation from MMR/autism where there is no supporting evidence.
used to be jdc said,
July 19, 2007 at 12:55 pm
www.cdc.gov/vaccines/pubs/pinkbook/pink-chapters.htm has some lovely facts on vaccination, with free downloadable chapter pdfs (unfortunately you have to download measles, mumps and rubella separately – there is no triple pdf available). I went straight for the bit on orchitis.
gadgeezer said,
July 19, 2007 at 1:33 pm
igb – Dr Michael Fitzpatrick has written a good account of the whooping-cough debacle.
Brian Deer, of course, has an account of the matter that argues that the courts may well have got this one wrong when they awarded compensation for whooping cough vaccine damage.
The Intertubes are playing up so I can’t verify this URL but I think that it is still correct.
briandeer.com/dtp-dpt-vaccine.htm
germslayer said,
July 19, 2007 at 2:21 pm
Neil Desperandum (can’t see comment no, sorry)
For Measles deaths look here:
www.hpa.org.uk/infections/topics_az/measles/nots_and_deaths.htm
To quote:
“In 2006 there was one measles death in a 13 years old male who had an underlying lung condition and was taking immunosuppressive drugs. Prior to 2006, the last death from acute measles was in 1992. Other measles deaths shown above are in older individuals and were caused by the late effects of measles. These infections were acquired during the 1980s or earlier, when epidemics of measles occurred.”
I don’t know when the scare started, but there have been small numbers of deaths throughout.
jackpt said,
July 19, 2007 at 3:47 pm
Well done Guardian.
Deano said,
July 20, 2007 at 9:12 am
igb wrote:
“I’m too young (42) to recall the full details of the whooping cough saga of the 1970s: did that have the same general feel, or was there more basis to it?”
– There was more basis to it.
Whooping Cough vaccine presented a small but real risk of brain injury to some children. However when whooping cough was prevelant, this was far outweighed by the risks of the disease itself.
As a result of this vaccine uptake was high, and Whopping Cough was almost eradicated.
At some point theoretically the risks of the vaccine to an individual then outweigh the risks of the disease.
An parent could choose to ‘cheat’ the system – relying on ‘herd immunity’ for their child’s protection.
Of course if this selfishness spreads – then herd immunity will be compromised and there will be an outbreak amongst unvaccinated individuals.
Which despite all the warnings was exactly what happened – partly as a result of irresponsible media reporting by journalists who supported parents who were doing what was “best for their children”.
Totally in the spirit of those Thatcherite times of course
Newer vaccines have reduced side-effects – although the American Medical system conspires against vaccination there, and there are unneccessary outbreaks – Worldwide about 300,000 kids die from it each year…
en.wikipedia.org/wiki/Pertussis#Whole-cell_pertussis_vaccine_controversy
diudiu said,
December 21, 2009 at 5:48 am
ed hardy ed hardy
ed hardy clothing ed hardy clothing
ed hardy jeans ed hardy jeans
christian audigier christian audigier
ed hardy t shirts ed hardy t shirts
ed hardy uk ed hardy uk
ed hardy bags ed hardy bags
ed hardy hoodies ed hardy hoodies
ed hardy mens ed hardy mens
ed hardy womens ed hardy womens
ed hardy kids ed hardy kids ed hardy kids