You’ll remember the Durham fish oil “trial” story, possibly the greatest example of scientific incompetence ever documented from a local authority.
Initially they said – to blanket media coverage – that they were running a trial on fish oils, giving pills to 3,000 children to see if it improved GCSE performance. I pointed out, along with several academics, that their experiment was incompetently designed, for no good reason, and so would only produce false positive results. They responded that this was okay, as they hadn’t called it a “trial”. This was very simply untrue: they had, repeatedly, in press releases, and interviews, and who cared anyway, because whatever you called it, this was still a stupid experiment. Durham’s response was to edit the online version of their press release to remove the word “trial”.
Then I asked what they were going to do in this experiment, how they were going to measure results, and more. They refused to give me this information – for an experiment by a public body performed on thousands of children – so I used the Freedom of Information act. They still refused. Then hundreds of you wrote to their information commissioner, using the FoI, and they refused again, accusing us of running a “vexacious campaign”.
Then the GCSE results for Durham came out: they weren’t too great, so I asked for the results of the “trial”. Durham refused to give me this information. Then they announced, bizarrely, untruthfully, in a formal response to a written question: “It was never intended, and the county council never suggested, that it would use this initiative to draw conclusions about the effectiveness or otherwise of using fish oil to boost exam results.”
This was, once again, very untrue. Durham’s own press release had clearly said they were giving out the pills “to see whether the proven benefits it has already brought children and young people in earlier trials can boost exam performances too”. All the press coverage said the same. The council’s chief schools inspector Dave Ford said “the county-wide trial will continue until the pupils complete their GCSE examinations next June, and the first test of the supplement’s effectiveness will be when they sit their ‘mock’ exams this December.” Suddenly this trial did not exist.
Now, bafflingly, suddenly, in defiance of their previous denials, Durham have released some results. “Detailed analysis of the outcome of the initiative,” they say “shows that pupils who took the Omega-3 supplement did better than those who did not.” Hardly. Let’s try to disentangle what they think they’ve done.
“Initially, just over 3,000 Year 11 pupils began the study, taking the Omega-3 tablets at school and at home. By the time GCSE examinations came around, 832 pupils had 80 per cent or greater compliance.” This is appalling. 2,168 of their subjects dropped out of the trial: they must count these people in the results. They do not. This makes the rest of their claimed results even more meaningless.
“Mr Ford and his colleagues then sought to identify the same number of Year 11 pupils who had not taken the supplement and match them to those who had, according to school, gender, prior attainment and social background.” They originally said they were going to compare childrens’ predicted GCSE performance (whatever on earth that means) against actual performance. Of course, they refused to say how they would analyse this, despite hundreds of requests: and it is vitally important that people performing experiments are clear what they are measuring, and how they will analyse it, before they begin, otherwise they can move the goalposts and get a false positive result afterwards, by slicing the cake a dozen different ways.
“The GCSE results of 629 ‘matched pairs’ – fish oil takers and non-fish oil takers – were then analysed.” Who are these 629? I thought it was 832? But more importantly, by selectively only looking at the results from the pupils who were most highly adherent to the capsules regime, they have skewed their sample, entirely unnecessarily. They have, in fact, simply discovered that school performance is better in children who are more highly adherent to a school regime involving pills, and who are, in all probability, also more adherent to everything at school, harder working, better performing, from completely different families, with higher aspirations, and so on. If you wanted to design an experiment to produce a spurious false positive result, you could not do any better than this.
This result has nothing to do with the pills, it is laughably incompetent science, in an experiment performed on thousands of Durham children. It has been widely reported in the local press, and in the Telegraph (doubtless with more to come) as “perhaps there is something in this preliminary stuff”. Every journalist is baffled by the details. Some report that there is “controversy”, in the classic style of journalists who simply report “two sides”.
But there are not two sides here: there is nothing in this work. It is not “controversial”. A crap unpublished result presented in a press release from an incompetent experiment analysed incompetently in god knows how many different ways by incompetent people who have shrouded themselves in secrecy and demonstrated themselves to be incapable of making reliable true statements about their own research – or even whether it exists – is not a “maybe”.
Fish oil pills are now the biggest selling food supplement product in the UK, with a market size of £110m. Nobody has ever tested them. Most damningly of all, Durham council had the children, and the pills, necessary to perform a decent piece of research. The only thing they were missing was the rigour, and help was offered. The only real question now is this: why has Dave Ford performed a incompetent experiment on thousands of children? And more importantly, why has Durham Council let him?