Of course there are lots more problems around all this. However, if Henk had done a decent computation the words “1 in 342 million” would never have been spoken in court – it might have been 1 in a 1000 or 1 in 10 000. Not so devastating. If he had checked the data the number would have collapsed and the case would have collapsed. If someone had mentioned that if there were 2 missing incidents in shifts outside Lucia’s, then the number would have collapsed, everything would have been different. Richard de Mulder, Henk’s colleague at the Erasmus law faculty, a lawyer with an MBA who knew how to start up and how to close down Microsoft Word, hence the big man in legal informatics, said that even if the data would have been a bit wrong the conclusion would have been the same. He said that everything that Henk did was state-of-the-art correct. And that it all meant that Lucia had to explain why the incidents happened in her shifts. Henk had provided some reasons “by way of example” and this was all there was on the table, so the court asked Lucia verbally if any of those things was true, and she gave the wrong answers, so she crucified herself on the statistical cross which Henk had put up for her.

Henk never admitted these mistakes in public, only in private. He never read the judge’s verdict and he spread slanderous accusations about Ton Derksen the philosopher of science. Henk had been brain-washed by the lies spread from the children’s hospital, he was deeply moved by the fact that a baby had died when admitted over the weekend merely for social reasons (to give the parents a break). And Lucia was on duty. This child had in fact been incorrectly diagnosed by Arda Derksen and she was the only medical specialist ever who claimed that the death was unnatural. It was her own patient. As the witch-hunt was her own witch-hunt, and she was the director of investigations.

I am so disappointed that Henk never admitted any resonsibility for anything. He became a turn-coat, he lost his scientist’s humility and took on the lawyer’s arrogance. I am so sad about that.

In those old days the idea was that you did some research, played around with various statistical consulation jobs, slowly came to a PhD thesis topic, wrote some papers yourself, and at some stage found a “supervisor” who became the responsible “promotor” when you defended your later completed thesis.

So we had some fun together and wrote some papers together, and did other things we liked and didn’t like … in about 1976 Henk’s wife was pregnant with their first child and he wanted to have a half-time position. Our boss did not allow that so he quit and went to a Geography department in a different city. At that time he had not chosen a topic for his PhD nor a supervisor. He moved into social-economic geography and from their to economics and finally to law. As our paths diverged we lost contact altogether, but I always retained a soft spot for him and a great deal of respect.

The statistics he had learnt previously in his last (master’s) year at university was basicly “almost nothing”, even according to the standards of the day, both from the applied and from the theoretical point of view.

When the Lucia case started up, in 2001, one read in the newspapers that “statisticians” Henk Elffers and Richard de Mulder were giving evidence in the court. No-one in the statistical world had ever heard of either, except for the one or two colleagues of ours back in ’74, ’75 who were still in the business. Later the defence got the services of a rather pure and philosophically inclined probabilist and an even more theoretical and philosophically inclined computer scientist and logician. The new defence lawyer of Lucia had met one of them at a cocktail party. The first defence lawyer of Lucia had actually done some research and found that a statistician in Norway, Odd Aalen, had worked on a similar case. He asked my old friend Odd for advice and Odd suggested he contact myself. However the new lawyer preferred to stick with his new friends Ronald Meester and Michiel van Lambalgen.

Later there were often talks and symposiums about how one should statistically analyse such data. Henk always gave very clear, and very clearly motivated, talks, in which he spelt out a kind of idealised version of what he had done for the courts. The Bayesian’s tended to discredit themselves with stupid and over-ambitious claims. The probabilists tended to miss the point.

It was quite a shock when we later found out, say in 2007, that Henk’s sanitized version of his work was rather different from what he had actually done. The multiplication of three p-values for no reason at all adjacent to the statement that the conclusion from combining the data from the three wards was a resounding “this was not chance”, was an apalling “mistake”. The number 1 in 342 million lived a life of its own, most journalists, many lawyers, all the men and women in the street, interpreted this as “the probability that Lucia is innocent is…”.

Of course the product of three p-values can be used as a test-statistic and a p-value can be computed from it, under the assumption of independence (under the null). p-values are essentially uniform random numbers between 0 and 1, under the null, so their product has the distribution of a product of uniform random numbers between 0 and 1. Twice the negative logarithm of this has a chi-squared distribution and Bob’s your uncle (Fisher’s combination method). This is however only a “last resort

Dear board of Haga

I was yesterday invited to give a major lecture to KNAW (the Royal Dutch Academy of Sciences) on the case of Lucia de Berk in all its ramifications and societal aspects. I would so like to be able to report that a mutually respectful and beneficial collaboration between scientists and Haga Hospitals is now helping to clarify what really happened, and to uncover the lessons that should be learnt for the future.

Now that the Lucia case is completely closed – in particular, there were no murders or otherwise unnatural deaths at all – I suppose there can no longer be any objection to a thorough multidisciinary and in particular statistical / epidemiological analysis of medical incidents at JKZ, say between 1995-2005. This would be so valuable for the future, and moreover, in accord with the current insight that sophisticated scientific evidence in the legal context has to be made as publicly available (to scientific inspection) as possible (cf reports of US Academy of Sciences, adopted by many scientific organizations worldwide).

Thanks to the investigations of Meulenbelt, Tytgat and Aderjan we now know that the nurses at JKZ worked in emergency situations with exemplary professionality, in contrast to the more mundane level of diagnosis and treatment. Moreover their insights into the medical state of the babies in their care was often better than that of specialists or their assistants, though usually not acknowledged as such.

Unfortunately, 30% of the Dutch population still believe that your former employee Lucia de Berk is a serial killer, and influential circles at the top of your hospital still continue to broadcast the slanderous accusations that “there is so much more against her” and “the whole case is nothing but an out-of-control family feud driven by the jealousy of some family members for the much more succesful careers of others”, and “it became an awful media hype, what could 100 professors of statistics or a second rate novelist know about the case?”.

Despite repeated attempts from 2004 onwards to warn the concerned individuals and organizations that something was terribly wrong with the whole case, your hospital and its senior personnel took no notice whatsoever, but instead intensified attempts to discredit those who had uncovered this particularly inconvient truth.

Doctors and nurses in confidence told us of many persons’ deep concern about the case, but no-one dared to speak out. Even a retired medical specialist wouldn’t say anything in public, since that would damage the prospects of his children in medical school. The few medical experts who dared to contradict or criticise findings of some colleagues during the trial were later ostracized by other colleagues for the breach in collegiality.

Highly confidential inside information about the case was repeatedly leaked from the Public Ministry and from the judiciary (even from the supreme court) to senior employees of your hospital. Those “outsiders” unfortunate enough to be driven by a dedication to justice and truth were subject to vile personal attacks in the media, accused of undermining the foundations of the state, and subjected to phone-taps and ostracism.

Vile disinformation about your former employee Mrs de Berk was leaked from your hospital to the press. Critical police investigators were taken off the case and critical hospital employees were silenced.

But that is all in the past now.

As always, despite this past, I remain hopeful of a mutually fruitful, mutually respectful, and civilized (gentlemanly) collaboration in the future.

Yours sincerely
Richard Gill

Distinguished Lorentz Fellow 2010-2011
President of Dutch Statistical Society
Member of KNAW

It’s my opinion that understanding the personalities of the key persons around which this whole case revolved, is the key to understanding why there was a case at all. Moreover: “tout comprendre c’est tout pardonner” – this understanding exhonerates those persons (chef-de-clinique/chief paediatrician at JKZ; director of JKZ) from any blame, of course they could not do otherwise and would do the same again today, if the case happened again; they have both said so, in public! Allegedly (ie, according to the reports of the Committee for Reconsideration of Closed Cases to the Dutch Public Ministry, of Advcate-General Mr. Knigge to the Dutch Supreme Court, and of prof Meulenbelt to the high court at Arnhem) they committed gross errors of professional judgement (both medical and managerial) which caused the whole catastrophe to explode out of control, devasting lives, almost killing Haga’s employee Mrs Lucia de Berk (she lay paralyzed by a stroke on the floor of her cell for 10 hours and was later refused decent medical treatment), ruining Netherlands international reputation for justice and humanity, and costing the Dutch taxpayer millions of Euros.

Once we know that those couple of people are not to blame, it follows that blame falls on the organisation around them (and above them). It follows that a great deal needs to be learnt about the root causes of the case, so as to prevent some bad luck and some unlucky personality interactions from unleashing a new social nuclear bomb and nuclear winter yet again.

I think that Goldacre made a necessary sacrifice by being vague about what p-values are, other than ‘things that do not directly represent probabilities’ and ‘things that cannot be simply multiplied together.’ His explanation is a superset of yours, which is more accurate. *30% of readers will walk away from the original post with a misinterpretation of the statistical argument. You suggest that the misuse of p-values is such an important issue that it would be better to be more specific – maybe bringing that number down to 20%, but maybe sacrificing a few readers altogether.

Your comment was aggressive, but only towards p-values and their misuse, and that’s a pretty righteous thing to be aggressive about.

[*] statistics drawn from my ass.

@quasilobachevski

Referring to someone respectfully while criticising them isn’t by definition sarcastic. Some people don’t feel comfortable referring to people by their first names that they don’t personally know, especially while criticising them.

Also: ad populum, ad verecundiam (the only reason gf defended his/her qualification is because it was attacked), bald-faced assertion (we don’t all know anything.)

I’m not sure what argument from “unfortunate tone” falls under, but you know that it’s not right. I thought the tone was fun. I’ve never seen somebody so pissed off by the mere existence of p-values:)

Sorry, I just couldn’t let that attack stand as the last comment on the issue.

To me, your original comment was the opposite of a ‘gentle’ correction. Your repeatedly referring to Ben as ‘Mr. Goldacre’ came across (again, to me) as sarcastic and aggressive. (It’s notoriously easy to strike an unintended tone in blog comments threads, of course.)

Your comment’s unfortunate tone combined with the fact that no one else here thinks that Ben’s statistics were incorrect! We all know (you’re not the only one with a stats qualification here) that strictly speaking one can’t assign p-values without making a pair of hypotheses. But this is a newspaper article, and Ben’s elision of this point seems reasonable, appropriate and not even slightly misleading.

In short, your original comment came across as aggressive and wrong. You ask JayScottGreenspan what you missed – the answer is context.

However, I did notice some confusion by both him and some commenters which I tried to (gently?) correct. I believe I was very specific about what the mistakes were. For the record, I am a qualified statistician. Not that that will help in a fire…

P-values *are* pretty confusing things when you look closely, actually, and as KMBayes correctly surmised, I’m uncomfortable with their popularity and wish Bayesian analyses were more mainstream (although I happily acknowledge this is happening, slowly). I have seen them being abused too many times even in the supposedly hallowed “peer-reviewed literature” and just used to opportunity so fulminate against them a little, hopefully educationally.

JayScottGreenspan, I don’t understand your need to advise me to ‘Try reading more, and typing less’. What did I miss, exactly? You claim that ‘“let’s say p=0.5″ basically means p could be anything, we don’t know what p is, but let’s just suppose p is 0.5 so we can see how ludicrous multiplying them can be’. Fair enough; I was more concerned with Ben’s previous sentence, which was ‘a pattern of incidents that is purely random noise’, which as I tried to explain with my coin-tossing example is nothing to do with p-values. Nothing to get your panties in a twist about, but I felt given my area of competence (which, in the spirit of your valuable if patronising advice that we are poor judges of our own, I refer to my degrees in) it would be remiss not to say something.

Keep up the good work Ben.

There’s no problem with preferring Bayesian statistics – no one denies that the Bayesian point of view has a lot to offer. There is a problem with saying ‘You’re wrong’, as gf basically did to Ben, when s/he really meant ‘I prefer a different point of view.’

As Ben points out in his book, we are all very poor judges of our own areas of competence.

O/T

Poor example – Speer pulled a very artful con job at nuremburg claiming he didn’t know about the death camps, his balancing act helped him escape the noose.

On the morning of 6 October 1943 Speer gave a speech at Posen to the Gauleiter. In the afternoon Himmler gave a speech in which he casually mentioned the extermination of europe’s Jews:

The brief sentence “The Jews must be exterminated” is easy to pronounce, but the demands on those who have had to put it into practice are the hardest and most difficult in the world … We, you see, were faced with the hard question , “What about the women and children?” … The hard decision had to be taken to have this people disappear from the face of the earth. … By the end of this year, the matter of the Jews will have been dealt with …

www.nizkor.org/hweb/people/h/himmler-heinrich/posen/oct-04-43/index.html

Clearly anyone at Posen listening could hardly claim ignorance about the plight of the Jews. Speer maintained that he left the conference after his morning speech. However there is no evidence of this – moreover Himmler addressed Speer directly during his speech, clearly implying Speer was still there with him.

Of course none of this has anything to do with party comrade Speer: it wasn’t your doing.

Speer’s rather lame defence was that Himmler must have forgotten his glasses and mistaken someone else as him!

Two of Speer’s close friends (two of the Gau) were definitely at the speech making it even more implausible that Speer did not know the content even if his excuses are true and he wasn’t there. Moreover as armaments minister his job would have brought him into contact with all the information he needed. He would have known about the forced labour used in Germany’s factories. And his visits to occupied Poland and Ukraine (especially the factories) should have left him in little doubt as to the nature of the regime he served.

Your answer is a little unkind. Based upon gf’s answer it’s fairly clear to me that he’s much more comfortable with the Bayesian approach to statistics and I’m in full agreement. For those of you with an interest in stats and who are unfamiliar with Bayesian the Bayesian approach I suggest you get aquainted. It’s the way forward.

I will risk people invoking Godwin’s Law, but claim the justification of listening to a radio play this afternoon about Nazi architect Albert Speer’s incarceration in Spandau. I looked him up on Wikipedia and found he was about the only one to plead guilty at
Nuremberg
“In political life, there is a responsibility for a man’s own sector. For that he is of course fully responsible. But beyond that there is a collective responsibility when he has been one of the leaders. Who else is to be held responsible for the course of events, if not the closest associates around the Chief of State?”

Although he denied knowledge of the death camps, he did not regard that as exonerating him:

“For from that moment on I was inescapably contaminated morally; from fear of discovering something which might have made me turn from my course, I had closed my eyes … Because I failed at that time, I still feel, to this day, responsible for Auschwitz in a wholly personal sense.”

Henk Ellfers, on the other hand, having been a major prosecution witness in a gross miscarriage of justice, blusters, retrospectively self-justifies, and tries to pass the buck to others.

I can understand a bit of ad hominem on gill1109′s part…

Other medical doctors swore that the event about which they were asked an opinion was an unnatural event. In their written evidence to the court it says that the reason they say this is because Lucia was present, because otherwise they would have considered the event unsurprising.

wtf? that’s a logical fallacy if I ever saw one, and one the judges should’ve thrown out.

You seem very angry about Ben’s treatment of p-values. We all have our favourite ways of explaining complicated scientific or mathematical phenomena, and often popular science writers come up short. But given the space restrictions, I think Ben dealt with the issue at hand pretty well.

Describing specific null and alternative hypotheses might have made pedants happy, but it would have wasted space and I doubt it would have been of much help to anyone who doesn’t already know what a p-value is.

Also, Ben never asserted that p-values are ‘probabilities of the data’. I can see why you think p-values are evil. They certainly seem to have confused you!

Yes, I just wasn’t/amn’t sure what two hypotheses @gf is referring to…

Is my understanding/lack of understanding of how the Fisher method workds/doesn’t work totally off?

Of course we should not blame you for this. It was your colleague and collaborator law professor Richard de Mulder who explained your advanced maths to the court and told them that what it meant was that *Lucia* had to explain her presence at the events.

You did not mention half a dozen other confounders, and for that I must blame the so-called experts for the defense, who certainly did even more damage than you had already done. They agreed that the numbers were amazing but argued that there were different models which led to different probabilities and therefore that no single probability could be given as “the probability that it was not chance”.

The judges obliged by writing on page 1 of their 80 odd summary of the verdict, that “a statistical probability calculation plays no part in the conviction”. Indeed, no calculation, and they scrupulously avoided words like “statistics” or “chance”. Instead they got medical doctors to swear that so many events in such a short time is totally impossible in normal circumstances. Other medical doctors swore that the event about which they were asked an opinion was an unnatural event. In their written evidence to the court it says that the reason they say this is because Lucia was present, because otherwise they would have considered the event unsurprising.

The court calls this “incontrovertible medical-scientific evidence that all the deaths were unnatural”. Yes it is incontrovertible because the court has decided it is true. Scientific because a prof. dr. in medicine signed his name to it.

In one case all but one of six experts thought the event was natural. One thought it was unnatural. The court writes that this is the good expert because he alone could see that the event was unnatural, and writes that medical science has advanced so far that a good expert could see even on the basis of a single A4 of rough notes whether a death was natural or unnatural.

My friend, you were screwed, but I forgive you, because if it hadn’t been for you, the statisticians would never have got involved in the case (*they* have nothing to lose), and because the statisticians got involved in the case, it got reopened. So if it wasn’t for you, Lucia would have perished in jail long ago and nobody would have cared or noticed.

I was disappointed that you chose to side with the conservative lawyers rather than the liberal scientifists (where you were born and raised), but I guess he who pays the piper calls the tune, and that you value a chair in a law department more than the respect of mathematicians. Who after all have no power whatsoever.

Dear dr. Goldacre,

It is a pity you did not reach me as I was absent, (though otherwise the time you allowed me for reaction was extremely short as well), so that I could not correct your complete misunderstanding of both my report to the court in the De Berk case and the role it played in evidence. Bad journalism, I say.
It is sad to observe that your comment was entirely based on hearsay, from a source that should have known better.

I felt obliged to send a reaction to the editor of the Guardian.

I enclose the text of it.

Sincerely

Henk Elffers, senior researcher Netherlands Institute for the Study of Crime and Law Enforcement NSCR
Amsterdam

—-

Dr Ben Goldacre (Guardian of April 10, 2010) gets hold of the wrong end of the stick in his comments on the role of statistics in the murder conviction of the Dutch nurse Lucia de Berk, whose case is now retried. His treatment completely misapprehends the statistical report prepared by me for the court in first instance, and he misunderstands the role of the report in evidence.

No, my statistical report did not neglect the Texas sharpshooter problem. For that reason, it explicitly investigated the question whether, given the number of nurses and their roster of duty, and given the number of incidents, it is compatible with chance that someone, whichever member of the staff, would be present at all incidents.

No, for the analysis used, it is not relevant what the rate of occurrence of incidents was in previous periods, as the analysis was conditional on the period under consideration.

No, the analysis was not done after arbitrarily dividing the data in three portions. I investigated the question whether someone (whichever member of the nursing staff) could have met all incidents in the first hospital by chance (result: very unlikely). Subsequently, I investigated the question whether the person thus identified could have met by chance the incidents as observed (given rosters etc.) in two previous appointments (results, respectively: unlikely and slightly unlikely).

No, one should not combine these results by means of Fisher’s method, as the first test does not address the same hypothesis as the other two. It is better to state the individual test results as such. As a matter of fact, I did not present the multiplied tail probabilities as a combined significance test, though I have been criticised for not being crystal clear in my way of reporting this part of my results.

No, the demonstrated incompatibility with chance is not a proof that the accused, though present at all incidents, has had a hand in them. I pointed this out to the court, suggesting several possible alternative explanations. The court, therefore, decided that the statistical argument could not be used for a conviction, as indeed Dutch judges had already ruled in a similar case.

No, mrs. De Berk has not been convicted on statistical evidence, but on medical evidence. The revision of her process hinges on the fact that new medical experts, reanalysing toxicological evidence, now testify that no unnatural deaths have taken place. Indeed, no statistical evidence is brought forward in the revision proceedings at all. Of course, the statistical analysis as presented loses all relevance if the occurrence of incidents has been recorded incorrectly.

Henk Elffers (M.Sc. mathematical statistics, Ph.D. psychology of law)
Amsterdam