Saturday October 28, 2006
There once was a time when your biggest worry, as a paediatrician, was being lynched by a herd of illiterate tabloid readers; but if you’re Professor Sir Roy Meadow you get scapegoated by the innumerate too. First he was struck off by the GMC for giving flawed evidence as an expert witness in the Sally Clark trial. Then he won an appeal. And now, this week, he’s half lost it again.
But what if the flaws in his evidence weren’t his fault alone? In the Sally Clark case, where two children in the same family had died, Meadow quoted “one in 73 million” as the chance of two children in the same family dying of Sudden Infant Death Syndrome (SIDS). This was a problematic piece of evidence for two very distinct reasons: one is easy to understand, the other is an absolute mindbender, but if you can concentrate, for three short minutes on a Saturday morning, I can prove that you are smarter than Professor Sir Roy, the judge in the Sally Clark case, her defense teams, the appeal court judges, and almost all the journalists and legal commentators reporting on the case.
We’ll do the easy problem first: the figure itself is iffy, as everyone now accepts. It was calculated as 8543 x 8543, as if the chances of two SIDS episodes in this one family were independent. This feels wrong from the outset, and anyone can see why: there might be environmental or genetic factors at play, both of which would be shared in the same family. But forget how pleased you are with yourself for understanding that fact. Even if we accept that two SIDS in one family is much more likely – say, 1 in 10,000 – any such figure is still of dubious relevance.
The action is in what we do with this spurious number. Many press reports at the time stated that one in 73 million was the chance that the deaths of Sally Clark’s two children were accidental: that is, the chance that she was innocent. Many in the court process seemed to share this view. The factoid certainly sticks in the mind. But this is an example of a well known and well documented piece of flawed reasoning known as the “Prosecutor’s Fallacy”.
Two babies in one family have died. This in itself is very rare. Once this rare event has occurred, the jury needs to weigh up two competing explanations for the babies’ deaths: double SIDS or double murder. Under normal circumstances – before any babies have died – double SIDS is very unlikely, and so is double murder. But now that the rare event of two babies dying in one family has occurred, the two explanations – double murder or double SIDS – are suddenly both very likely. If we really wanted to play statistics, we would need to know which is relatively more rare, double SIDS or double murder.
Not only was this crucial nuance missed at the time, it was also clearly missed in the appeal: they suggested that instead of “1 in 73,000,000″ Meadow should have said â€œvery rareâ€. They recognised the flaws in its calculation – the easy problem above – but still accepted it as establishing “a very broad point, namely the rarity of double SIDS.” But that was wrongheaded, and rarity is irrelevant, because double murder is rare too. The appeal court thought Meadow had maybe got the number of millions wrong, when in fact the precise figure was a side issue: and an entire court process failed to spot the nuance of how the figure should be used.
If it is true that Meadow should have spotted and anticipated the problems in the interpretation of his number, then so should the rest of the people involved in the case: a paediatrician has no more unique responsibility to be numerate than a lawyer. The Prosecutor’s Fallacy is also highly relevant in, for example, DNA evidence. Anyone who is going to trade in numbers, and use them, and think with them, and persuade with them, and lock people up with them, also has a responsibility to also understand them, and as you’ve just seen it’s hardly rocket science. But paediatricians aren’t just lynched by the illiterate herd: in this case, one was scapegoated by the innumerate.
Another way of looking at this interesting problem is…
Let’s say you come across two arrows stuck in a wall, a millimetre apart. I am standing next to the wall with a bow and arrow looking well pleased with myself. Am I a good archer? Your answer depends on the context. If there is one target drawn on the wall, and it was drawn before I got there, and I fired my two arrows dead into the centre of the bullseye, then there is little doubt that I am a total gangsta.
If, on the other hand, the wall is massive, and filled with tens of thousands of arrows that I’ve been firing into it over the course of many years, then the fact that there are two adjacent arrows here and there is not headline news. If I came along and drew a target around the two that are close together, you wouldn’t congratulate me on my archery skills.
That, I’ll say it again, is the irrelevance of the one in 73 million figure, even if it were a valid estimate.