Ben Goldacre
Saturday October 7, 2006
The Guardian

It is often unfairly assumed that I am a tenacious obsessive who refuses to let go. So at Durham council – as reported all over the newspapers and television – they’ve done loads of research on omega-3 fish oils making kids clever. It’s all very well saying that, but I need to see the data, to be sure there are no flaws.

Science has a certain authority, which makes it attractive to journalists and salesmen alike, but the authority comes from the transparency: it’s not about taking things on faith, or newspaper articles, it’s about openly publishing your data and your methods, so everyone can check your working. That’s why papers are published. That’s why you should not go to the media with unpublished data, unless you are ready and willing to go through it in detail.

So what happens when a bloke like me comes along, who knows how to read a piece of scientific research, and wants to find out some real information about the trials?

Well, your technical questions get directed to Madeleine Portwood, senior educational psychologist. She has been on all the television news and given interviews to all these journalists on the wonders of omega-3 and Durham’s fabulous research: she has a lot of time for the story. But she won’t even return my phone calls, or my emails.

If I want to find out about the data, beyond inadequate answers with a huge delay, the press office tells me, I have to travel 275 miles to Durham, to do it in person. Obstructive, but I will eventually make the time, just to prove a point.

But in the meantime, what can you do? All I want to do is find out the science behind their endless headlines. I’ve been communicating through the Durham press office. A week ago I asked simple, basic questions such as: for all these trials, what’s the story? Who were the kids? How old? How many? What were they given? For how long? What was measured? What were the results? A week later, no joy.

So then what do you do, to try and get some useful data? I go to the website, and the data is a farce. It features testimonials that would not be out of place on bonkers alternative therapy websites (and I am familiar with that genre). One child says: “Now I am not so interested in the TV. I just like reading books. The best place in all the world is the library. I absolutely love it.” It’s a miracle, honey.

But what about all those complicated-looking graphs? This site is the perfect illustration of how important it is that data is properly published in peer-reviewed academic journals before making dramatic claims, so that people can simply see exactly what you did.

It’s hard to tell what’s there. There is some data from a published trial by some Oxford researchers, but apart from that, I can’t find any sign of Durham’s own placebo controlled trials that they keep banging on about.

There are plenty of graphs, but the graphs are just reporting more of those classic Durham “trials”, with no placebo control group, that cannot give useful data (you know, the ones where they change their minds about whether they’re trials or not, depending on circumstances). So they report improvements, for example, with sciencey looking graphics to illustrate them, but there are no statistics to say if the changes were statistically significant.

I’m trying to think of a way to explain to the lay person just how much data is missing from this site, and how useless that renders the information there. How’s this: nowhere on the page – nowhere on the entire site, as far as I can see – does it tell you how many children were in this study. I can’t think of a single more basic piece of information about a study than how many subjects there were, and it’s simply not there.

All in all, it’s incredibly difficult for me to establish what has been done and where. My interpretation of all this running about is very simple: Madeleine Portwood is falling over herself to make time for journalists who know nothing about science, with her “research”, and her “limbic system”, and her graphs; but as soon as someone who knows about things like t-tests and the CONSORT guidelines for presenting clinical trial research data comes along, she shuts up shop.

Journalists of Britain: if a woman from Durham fitting this description approaches you with a “science” story involving children, you know where to come first.

· Links, references, and an explanation of what on earth CONSORT guidelines are, at badscience.net. Please send your bad science to bad.science@guardian.co.uk

Links and References:

The site in question is:

www.durhamtrial.org/

It’s very confusing so I’ll talk you through what seems to be on it. Firstly, and crucially, none of the randomised controlled trials Durham/Portwood claim to have done, or the positive results that are discussed, can be found anywhere there.

There are two of those “trials” which don’t have a placebo group. Durham, Portwood, Ford, and Equazen still can’t make their mind up about these. First they called them trials and talked about the positive results they were hoping for. Then when people like me pointed out that this is a rubbish way of collecting data, they said, oh no, they’re not trials, we never said they were, and rather preoccupied with the use of the word “trial” for some reason. But in the same breath they still go on about collecting data from this kind of useless study design, and in fact, 2 of the 3 things on Durham Trials website are reporting exactly that kind of data. They just can’t make their minds up. It’s a stupid way of collecting data, and we don’t really care what you call it.

Anyway, that’s the “secondary” and “pre-school” parts of the Durham Trials site.

Then there’s all this stuff which looks like a placebo controlled randomised controlled trial. Brilliant, you think: here’s the action, they’re presenting us with some real data. No. In fact, it took a fair while for me to even spot it (and I know this literature backwards now) but these graphs and results are all taken from the “Oxford-Durham Dyspraxia Trial” designed, performed and published by two researchers from Oxford University. This study was certainly done with some research assistance from Durham Council, as the study was done on Durham children (although if you read it eg Portwood’s name is nowhere on the paper), and it’s very good of them to have helped on it: ownership of this paper (on this site and elsewhere) seems to have been rather grandly adopted by the Durham fish oil posse. This is pretty much neither here nor there as far as I’m concerned, but it’s a shame to do it in ways that create further unnecessary confusion in the minds of those you are trying to communicate with.

There is no citation to or acknowledgement of the Oxford researchers’ published paper, so no way of knowing that it is a properly published study, designed, performed and published by other people, where it was published, by who, etc (The Oxford-Durham study: a randomized, controlled trial of dietary supplementation with fatty acids in children with developmental coordination disorder. Pediatrics, 2005. 115(5): p. 1360-6 since you asked).

I would say this confusing lack of clarity is the dominant theme in the presentation by durham of their research.

EDIT: Oh my god! The site is down! If it stays down during today I’ll post an archived copy of it so you can feel the joy, Durham Council are used to their sites being hosted alongside other peoples (long story).

CONSORT:

The CONSORT guidelines are an internationally well-respected set of standards for the information that should be presented for a clinical trial, and they represent the basic things you would need to know about a study to assess its quality and results.

The quick checklist is here:

www.consort-statement.org/Downloads/checklist.pdf

And there is more on the CONSORT project and its history here.

At the best of times, you would not take it at face value that somebody’s trial simply proved that “X improves Y”: you would want to know how they measured Y, whether it was randomised, whether it was blinded (that is, the subjects and the experimenters didn’t know who got the active treatment), and if so how, was the blinding tested to see if it worked, and a million other things.

CONSORT is very useful, and the benchmark, the gold standard of information you’d want, to assess research claims. If someone made claims, I’d expect them to at least be able to answer the kind of questions in CONSORT, you know, on the phone or something.

I had a phone conversation with the director of Equazen a couple of days ago (genuinely pleasant businessman, glad we spoke) and I sent him an email about the kind of information I’d like to be able to have to assess someone’s positive research claims. By the by, I’m not sure I think that Equazen have made such specific claims about their research as the Durham posse have.

Anyway, we’re chatting about me getting to see more details of Equazen’s many unpublished studies and this is a small extract of what I sent to him on CONSORT, which I post here because it’s late, I wrote it today, I can’t imagine he’d mind, and I think it explains the issue fairly well (forgive the “can’t type with capitals in emails” idiosyncracy):

“…as you know, my main interest is that if people make a claim about scientific research, it’s not enough to have their interpretation of the data, one has to be able to see what was done, to who, how many people, what was measured, how the study was conducted, what the results were, what the stats on those results were, whether it was randomisd and blinded, if so how, and so on. that’s because the details are where things that might make you have concerns about a study are likely to arise.

“to take an extremely extreme example: somebody might say to me, “i have research that shows green children are more intelligent than blue children”. so i go to read their full study, and when i do, it turns out that they measured intelligence as IQ, perhaps using some sub-tests that are known to be more culturally determined than others, so there’s a problem already. then i look at who they’ve compared with who, and the children were from completely different backgrounds, and that was not accounted for. and then maybe it turns out the differences in results between the two groups were only very small, albeit that they were statistically highly significant, and so on. i could still see why they summarised the research as they did, but i am not so sure i agree with the interpretation, or the confidence they have in the results, even if they are still an honest account of a positive finding.

“anyway, the full list of what most people would consider to be enough info to make a sound judgement on a clinical trial is enshrined in a very famous and internationally recognised document called the CONSORT statement.

“here is the checklist:

“http://www.consort-statement.org/Downloads/checklist.pdf

“if anyone came along making an assertion about their research, i would go and find the paper in the journal and look at the kind of stuff that’s in the CONSORT statement to judge it. if a research team approached the media in advance of publication of a trial – which many would say should only be done in cases of big public interest, even though it does happen – it would certainly be very reasonable to expect that you would be able to approach them and ask about the kind of stuff in the CONSORT statement, and they’d have it on the tip of their tongues, as it were.”