Vital statistics

June 23rd, 2005 by Ben Goldacre in bad science, PhDs, doctors, and qualifications, statistics | 12 Comments »

Ben Goldacre
Thursday June 23, 2005
The Guardian

· Of course, the past two years of Bad Science was nothing more than a cover for the “popular statistics” lecture series I really wanted to give, but knew I could never sell to a newspaper. So, on to Professor Roy Meadows and Meadows’ Law, that: “One sudden infant death is a tragedy, two is suspicious and three is murder until proved otherwise.” Pay attention: flaky statistics can get you jailed, or erased from the GMC register if you’re not careful. Meadows said the likelihood of there being two sudden infant deaths in the same family was one in 73 million. People are queueing up to point out that the Royal Statistical Society said this aspect of the case involved flaky reasoning. Nobody in newspapers was geeky enough to explain why.

· Welcome to my world. This is an example of a common error of reasoning known as “prosecutors’ fallacy”. It rests on creating a figure erroneously, and then applying it erroneously. First, let’s take where the 73m came from. Meadows appears to have taken a figure for sudden infant death syndrome (Sids), about one in 8,500, and squared it to find the likelihood of Sids happening twice in the same family. This would only be valid if we could be sure Sids always happened by chance and independently of family factors such as genetics and environment. In fact, there are strong reasons to believe that there are, unknown, genetic and environmental factors which will make Sids more likely in any child, and since genetic factors and environment tend to be shared in families, you may be more likely to find Sids twice in the same family than the initial calculation suggests. The error in the figure 73m is therefore likely to be extremely large, and in one direction only, that is, overestimating. The true figure may be much less incriminating.

· The real action is in how we use this inaccurate figure. Press reports at the time said 73m to one was the chance the two deaths of Clark’s children were “accidental”. The court appeared to concur. But this was an error. A very rare event has already occurred: the unexplained death of two siblings. What the jury ought to have considered is: which is the more likely rare event, double murder or double Sids? If anything we want the relative probabilities. The original case did not consider this; if it had, you would have thought a conviction to be less likely. The appeal court ruling accepted flaws with the figure, but said it established “a very broad point, namely the rarity of double Sids”. Scream now.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

12 Responses



  1. Michael Harman said,

    October 24, 2005 at 11:04 am

    Professor Sir Roy Meadows was apparently supremely confident of two things. One – familial, genetic, and environmental factors had no effect whatever on SID; and two – nobody had any real idea of the causes of SID. How he reconciled those two things is beyond me (unless he secretly believed that SID syndrome did not exist and that all sudden infant deaths were murder).

    Even accepting that SID is purely random, the squaring of the chance was illegitimate. Looking at the entire population of mothers, what is the chance that any one of them will have a SID? 8500 to 1. That population includes some who have already experienced a SID. What is the chance that one of them will experience a second SID? It’s still 8500 to 1, not 73 million to 1.

    The chance of winning the lottery is 14 million to 1, and the chance of someone winning it twice is that squared, 200 trillion to 1. That’s the chance of some specific person, say my wife, winning twice. (I personally don’t try the lottery; it’s a voluntary tax which I choose not to pay.) Yet I think that there has in fact been a double win (possibly syndicates were involved). And if you look at all the winners over the years, the chance of any of them winning again is the standard 14 million to 1; and considering that most of them probably put on several pounds every week, it isn’t that astonishing if one of them wins a second time.

  2. Ben Goldacre said,

    October 24, 2005 at 11:14 am

    Sure, but that figure is not the problem: even if Roy Meadow had given a figure of “1 in 15,000″ the court would still have been wrong to interpret that as the likelihood of the defendant’s innocence.

    Look at it this way. After the extremely rare event of two children dying in the same family has occurred, then the two causes you are choosing between to explain this double death – murder or accident – are both suddenly quite common, and if you’re going to bring stats into it, then you should at least compare the rarity of double homicide with the rarity of double cot-death.

    Comparing the rarity of “double cot-death” with the very common “no deaths at all”, as the court did, is a completely irrelevant comparison.

  3. Michael Harman said,

    October 25, 2005 at 2:38 pm

    Actually it did occur to me that infanticide could be a “family factor”, but I didn’t pursue that point because I couldn’t see where it would lead.

  4. Peter Ellis said,

    October 30, 2005 at 10:44 am

    It’s got nothing to do whetther infanticide is familial or not.

    It’s a question of ignoring how inherently rare murder is.

    It’s like saying “This person has acquired a huge amount of money twice. Since the likelihood of winning the lottery twice is so low, they must have burgled two multi-millionaires”

    It’s completely bogus because it doesn’t take into account whether or not it’s likely (in the absence of any other evidence) that someone has done the burglary.

    To establish whether or not they *did* burgle multimillionaires, you have to look for other evidence of guilt or innocence (whether they left fingerprints at the site, whether they can point to winning lottery tickets, etc.)

    In fact, since since child murder is such a rare event compared to SIDS, it’s more equivalent to saying “This person is extremely flat. Since being run over by a twice is extremely unlikely, they must have had two grand pianos dropped on their head”. It’s the same logical flaw: failing to take into account that being hit by a grand piano is *even more* unlikely than being run over by a steamroller

  5. eugene merrett said,

    February 17, 2006 at 12:01 pm

    If we make the assumption that SIDS are independant (which is what Meadows believes in then it is reasonable to make the 1 in 73 million proposition). But if there are a million families in the UK with 2 children then it is not inconcievable that a family may suffer double SIDS. In fact over the years with new families starting up every day it is very likely that a family will suffer double SIDS within a decade or so . So there is clearly a reasonable doubt.

    However what if the odds were say 1 in a trillion (i.e 4-5 deaths from SID assuming purely independant events)- surely in this case it very likely that the person is guilty of homocide as a double event is so inconcievable given the relative small number of families.

    Am I wrong?

    P.S has there been a case of a single person (not a syndicate) who has won a 14 million lottery twice! I do not think so since the number of players is too small compared to the odds to make such an event remotely possible

  6. Robert Craig said,

    March 4, 2006 at 9:59 pm

    There’s a simpler analogy which may make the silliness of this more evident. In my street 10 women have just had babies. One baby is white: nine are black. So the probability of a white baby in my street appears to be 1/10. Next year they’re all due to give birth again – what’s the probability that the one woman who had a white baby with have anothr white baby? Professor Meadow’s use of statistics would appear to suggest 1/100.

  7. Ben Goldacre said,

    March 4, 2006 at 10:30 pm

    robert: yes, that’s an analogy for the error in his figure, one in 73 million. but the key issue is not that, it’s what you do with that figure, see above.

  8. Robert Craig said,

    March 6, 2006 at 8:38 pm

    Yes – I agree completely, There are two issues – wrong statistics in getting the 1/100 in my analogy, and wrong deductions – that the white woman who now has two white babies must have been at it with bleach and should be locked up.

  9. alangdon said,

    June 13, 2006 at 11:49 am

    It strikes me that some of you are being a bit slow.

    Read Ben Goldacres excellent and increasingly frustrated explanations before posting anything else.

    Does anyone know if the original defense spotted that Meadows was guilty (and so glaringly) of the prosecutor’s fallacy? If not, should they not be struck off as well? Surely no defense lawyer or judge should be ignorant of the prosecutor’s fallacy. That juries will always remain so is a problem with juries (and that Doctor’s are bad at statistics is well known; why call them in as expert witnesses?)

  10. mduk said,

    January 7, 2007 at 1:05 am

    I know it’s a little late to respond here, but I did a quick search for some comparative figures for violent death rates in infants. The best reference I found was www.azdhs.gov/plan/report/im/im/injury/injury.htm prepared by the Arizona Center for Health Statistics, from which I gleaned the following:

    – the US average infant injury mortality was 25.6 per 100,000 (for males 0-4 years, 1987-1993)
    – around 13% of such deaths are attributable to “neglect, maltreatment, or murder”

    Assuming figures for girls are similar, and that these figures are broadly comparable in the UK, this implies that the chance of an infant dying such a death is around 1 in 30,000.

    Making Meadows’ starting assumption here would give us a 1 in 900,000,000 chance that this case involved a double homicide!

  11. wayscj said,

    November 21, 2009 at 7:36 am

    ed hardy ed hardy
    ed hardy clothing ed hardy clothing
    ed hardy shop ed hardy shop
    christian audigier christian audigier
    ed hardy cheap ed hardy cheap
    ed hardy outlet ed hardy outlet
    ed hardy sale ed hardy sale
    ed hardy store ed hardy store
    ed hardy mens ed hardy mens
    ed hardy womens ed hardy womens
    ed hardy kids ed hardy kids ed hardy kids

  12. iphone revolution said,

    December 30, 2009 at 8:36 am

    iphone wireless

    iphone

    Apple iphone