I’m sure there’s some data in here somewhere…

October 7th, 2006 by Ben Goldacre in adverts, bad science, equazen, fish oil, nutritionists, references, statistics | 67 Comments »

Ben Goldacre
Saturday October 7, 2006
The Guardian

It is often unfairly assumed that I am a tenacious obsessive who refuses to let go. So at Durham council – as reported all over the newspapers and television – they’ve done loads of research on omega-3 fish oils making kids clever. It’s all very well saying that, but I need to see the data, to be sure there are no flaws.

Science has a certain authority, which makes it attractive to journalists and salesmen alike, but the authority comes from the transparency: it’s not about taking things on faith, or newspaper articles, it’s about openly publishing your data and your methods, so everyone can check your working. That’s why papers are published. That’s why you should not go to the media with unpublished data, unless you are ready and willing to go through it in detail.

So what happens when a bloke like me comes along, who knows how to read a piece of scientific research, and wants to find out some real information about the trials?

Well, your technical questions get directed to Madeleine Portwood, senior educational psychologist. She has been on all the television news and given interviews to all these journalists on the wonders of omega-3 and Durham’s fabulous research: she has a lot of time for the story. But she won’t even return my phone calls, or my emails.

If I want to find out about the data, beyond inadequate answers with a huge delay, the press office tells me, I have to travel 275 miles to Durham, to do it in person. Obstructive, but I will eventually make the time, just to prove a point.

But in the meantime, what can you do? All I want to do is find out the science behind their endless headlines. I’ve been communicating through the Durham press office. A week ago I asked simple, basic questions such as: for all these trials, what’s the story? Who were the kids? How old? How many? What were they given? For how long? What was measured? What were the results? A week later, no joy.

So then what do you do, to try and get some useful data? I go to the website, and the data is a farce. It features testimonials that would not be out of place on bonkers alternative therapy websites (and I am familiar with that genre). One child says: “Now I am not so interested in the TV. I just like reading books. The best place in all the world is the library. I absolutely love it.” It’s a miracle, honey.

But what about all those complicated-looking graphs? This site is the perfect illustration of how important it is that data is properly published in peer-reviewed academic journals before making dramatic claims, so that people can simply see exactly what you did.

It’s hard to tell what’s there. There is some data from a published trial by some Oxford researchers, but apart from that, I can’t find any sign of Durham’s own placebo controlled trials that they keep banging on about.

There are plenty of graphs, but the graphs are just reporting more of those classic Durham “trials”, with no placebo control group, that cannot give useful data (you know, the ones where they change their minds about whether they’re trials or not, depending on circumstances). So they report improvements, for example, with sciencey looking graphics to illustrate them, but there are no statistics to say if the changes were statistically significant.

I’m trying to think of a way to explain to the lay person just how much data is missing from this site, and how useless that renders the information there. How’s this: nowhere on the page – nowhere on the entire site, as far as I can see – does it tell you how many children were in this study. I can’t think of a single more basic piece of information about a study than how many subjects there were, and it’s simply not there.

All in all, it’s incredibly difficult for me to establish what has been done and where. My interpretation of all this running about is very simple: Madeleine Portwood is falling over herself to make time for journalists who know nothing about science, with her “research”, and her “limbic system”, and her graphs; but as soon as someone who knows about things like t-tests and the CONSORT guidelines for presenting clinical trial research data comes along, she shuts up shop.

Journalists of Britain: if a woman from Durham fitting this description approaches you with a “science” story involving children, you know where to come first.

· Links, references, and an explanation of what on earth CONSORT guidelines are, at badscience.net. Please send your bad science to bad.science@guardian.co.uk

Links and References:

The site in question is:


It’s very confusing so I’ll talk you through what seems to be on it. Firstly, and crucially, none of the randomised controlled trials Durham/Portwood claim to have done, or the positive results that are discussed, can be found anywhere there.

There are two of those “trials” which don’t have a placebo group. Durham, Portwood, Ford, and Equazen still can’t make their mind up about these. First they called them trials and talked about the positive results they were hoping for. Then when people like me pointed out that this is a rubbish way of collecting data, they said, oh no, they’re not trials, we never said they were, and rather preoccupied with the use of the word “trial” for some reason. But in the same breath they still go on about collecting data from this kind of useless study design, and in fact, 2 of the 3 things on Durham Trials website are reporting exactly that kind of data. They just can’t make their minds up. It’s a stupid way of collecting data, and we don’t really care what you call it.

Anyway, that’s the “secondary” and “pre-school” parts of the Durham Trials site.

Then there’s all this stuff which looks like a placebo controlled randomised controlled trial. Brilliant, you think: here’s the action, they’re presenting us with some real data. No. In fact, it took a fair while for me to even spot it (and I know this literature backwards now) but these graphs and results are all taken from the “Oxford-Durham Dyspraxia Trial” designed, performed and published by two researchers from Oxford University. This study was certainly done with some research assistance from Durham Council, as the study was done on Durham children (although if you read it eg Portwood’s name is nowhere on the paper), and it’s very good of them to have helped on it: ownership of this paper (on this site and elsewhere) seems to have been rather grandly adopted by the Durham fish oil posse. This is pretty much neither here nor there as far as I’m concerned, but it’s a shame to do it in ways that create further unnecessary confusion in the minds of those you are trying to communicate with.

There is no citation to or acknowledgement of the Oxford researchers’ published paper, so no way of knowing that it is a properly published study, designed, performed and published by other people, where it was published, by who, etc (The Oxford-Durham study: a randomized, controlled trial of dietary supplementation with fatty acids in children with developmental coordination disorder. Pediatrics, 2005. 115(5): p. 1360-6 since you asked).

I would say this confusing lack of clarity is the dominant theme in the presentation by durham of their research.

EDIT: Oh my god! The site is down! If it stays down during today I’ll post an archived copy of it so you can feel the joy, Durham Council are used to their sites being hosted alongside other peoples (long story).


The CONSORT guidelines are an internationally well-respected set of standards for the information that should be presented for a clinical trial, and they represent the basic things you would need to know about a study to assess its quality and results.

The quick checklist is here:


And there is more on the CONSORT project and its history here.

At the best of times, you would not take it at face value that somebody’s trial simply proved that “X improves Y”: you would want to know how they measured Y, whether it was randomised, whether it was blinded (that is, the subjects and the experimenters didn’t know who got the active treatment), and if so how, was the blinding tested to see if it worked, and a million other things.

CONSORT is very useful, and the benchmark, the gold standard of information you’d want, to assess research claims. If someone made claims, I’d expect them to at least be able to answer the kind of questions in CONSORT, you know, on the phone or something.

I had a phone conversation with the director of Equazen a couple of days ago (genuinely pleasant businessman, glad we spoke) and I sent him an email about the kind of information I’d like to be able to have to assess someone’s positive research claims. By the by, I’m not sure I think that Equazen have made such specific claims about their research as the Durham posse have.

Anyway, we’re chatting about me getting to see more details of Equazen’s many unpublished studies and this is a small extract of what I sent to him on CONSORT, which I post here because it’s late, I wrote it today, I can’t imagine he’d mind, and I think it explains the issue fairly well (forgive the “can’t type with capitals in emails” idiosyncracy):

“…as you know, my main interest is that if people make a claim about scientific research, it’s not enough to have their interpretation of the data, one has to be able to see what was done, to who, how many people, what was measured, how the study was conducted, what the results were, what the stats on those results were, whether it was randomisd and blinded, if so how, and so on. that’s because the details are where things that might make you have concerns about a study are likely to arise.

“to take an extremely extreme example: somebody might say to me, “i have research that shows green children are more intelligent than blue children”. so i go to read their full study, and when i do, it turns out that they measured intelligence as IQ, perhaps using some sub-tests that are known to be more culturally determined than others, so there’s a problem already. then i look at who they’ve compared with who, and the children were from completely different backgrounds, and that was not accounted for. and then maybe it turns out the differences in results between the two groups were only very small, albeit that they were statistically highly significant, and so on. i could still see why they summarised the research as they did, but i am not so sure i agree with the interpretation, or the confidence they have in the results, even if they are still an honest account of a positive finding.

“anyway, the full list of what most people would consider to be enough info to make a sound judgement on a clinical trial is enshrined in a very famous and internationally recognised document called the CONSORT statement.

“here is the checklist:


“if anyone came along making an assertion about their research, i would go and find the paper in the journal and look at the kind of stuff that’s in the CONSORT statement to judge it. if a research team approached the media in advance of publication of a trial – which many would say should only be done in cases of big public interest, even though it does happen – it would certainly be very reasonable to expect that you would be able to approach them and ask about the kind of stuff in the CONSORT statement, and they’d have it on the tip of their tongues, as it were.”

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

67 Responses

  1. Dr Aust said,

    October 11, 2006 at 2:35 pm

    Glad to see that linseed oil has inflitrated your polymerisations, coracle. My mad chemistry A level teacher (blogs passim) would be pleased.

    I used to eat polyunsaturate-rich marge myself, but have defaulted back to butter since ‘Er Indoors (the medical half of the family but born and raised on a dairy farm) flaty refuses to touch “plastic marge”.

    Like many other diet / “lifestyle” interventions, it isa question of degree. All in all I suspect that:

    (i) not smoking;
    (ii) staying physically active;
    (iii) not being a huge fat bloater and
    (iv) not chowing through shedloads of high saturated fat high-salt processed shite

    …are so dominant in the “cardiovascular disease risk” sweepstakes that butter vs. marge on your morning toast is unlikely to be a biggie, except insofar as it relates to (iii).

  2. wewillfixit said,

    October 11, 2006 at 2:56 pm

    You have to have butter. Marge tastes awful spread really thick….

  3. Aspiring Pedant said,

    October 11, 2006 at 3:39 pm

    Not only does margarine taste better than butter but it spreads more easily & thinly.

  4. wewillfixit said,

    October 11, 2006 at 3:44 pm

    Who wants it spread thinly on a freshly toasted crumpet?


  5. doctormonkey said,

    October 11, 2006 at 4:20 pm

    hmm, we are now debating margerine vs butter, can we get back to basing homeopaths again?

    back on the point, Dr Aust’s link to oceans alive says that the mercury binds to protein and the oil should be fat/lipid so i suppose it should be ok… unless some nasties are lipid soluble (as many things are in pharmacology) such as dioxins, PCBs and pesticides (same source) and so will be in highest concentrations in the oil/lipid/fat of the fish

    still sure they are safe?

  6. pv said,

    October 11, 2006 at 5:32 pm

    This is the point (and it’s not very nice):

  7. Gordon said,

    October 11, 2006 at 6:07 pm

    I was surfing and trawling “Durham fish trial” on google. I found this quote attributed to an Equazen spokesperson that seems relevant to doctormonkey’s concern:

    “We use a highly sophisticated six-stage process to ensure that the oil used in the finished product is the highest quality and free from contaminants. This process starts from the careful sourcing of fish (sardines, pilchards and tuna) from local fisherman fishing in the world’s cleanest seas.

    The oils are then washed, chilled and filtered so that contaminants are removed — well in excess of FSA requirements — whilst maintaining the integrity of the oil itself.”

    It was on this site:

    So it appears that Equazen’s oils are full of natural goodness without the nasty man-made pollutants.

  8. Gordon said,

    October 11, 2006 at 6:07 pm

    On another note, and on the Durham council site there was mention of a pre-trial assessment of some children, 94% of whom “had moderate or severe ratings for Attention Deficit-Hyperactivity Disorder (ADHD), 94 per cent had the same Inattention Scale rating and 89 per cent were rated as having severe impulsivity.” I’ve just listened to a talk by a postgraduate student in the psychology department where I study/work who stated that 1-5% of children would have ADHD. So either Durham has a huge ADHD problem or the children tested were all selected because they showed signs of, or had been diagnosed with, ADHD. The point is, as Ben and others have said above, that this 94%, 89% or whatever percent showed improvement on whatever scale of measuring they used is useless information because it tells us nothing about who the participants (subjects) were.

    In answer to Ben’s post (#26) I trawled Web of Science for any reference to M* Portwood (search term to find all authors surname Portwood who have an initial M). There were 26 hits. All of them, as far as i could determine, were for MM Portwood who specialises in something completely different. The only published work for Madeliene Portwood I could find were the books she has published on dealing with dyspraxia.

  9. Dr Aust said,

    October 11, 2006 at 6:20 pm

    Dr M

    As I said above somewhere, I think they do have some tricks for removing “chemical nasties” like PCBs.

    Apart from anything else, before they were telling you the stuff would make you dead brainy, a lot of the “Fish(Oil)Pitch” – esp. in the US – was that the Fish oils would give you “the goodies of fish without the EVIL Hg / PCBs”

    Although it would take more than that to make me trade my pan-fried salmon steak for a fistful of fish-pills.

    I do wonder how much of their “clean-up” is basically just NOT using the tissues where nasties are most concentrated, which would be guts (esp liver), gills and skin, applying some fairly basic toxicology.

    But once you have a “liquid product” there will always be SOME nifty chemical way of cleaning it up. Just depends how much you are prepared to spend doing it

  10. doctormonkey said,

    October 11, 2006 at 7:17 pm

    the problem seems to be (if the source i mentioned before is accurrate) that many of these nasties (although not mercury) are specifically in the oil from the fish as this is almost pure lipid and these nasties are lipid-soluble and although you may be able to remove them it sounds a little difficult and i would worry about the processes to remove them as well or instead

    btw i tend to use cartoonish descriptions of things, including to patients, such as “nasties” rather than PCBs, dioxins, pesticides… but i can speak “science” when i have to

  11. Dr Aust said,

    October 11, 2006 at 9:18 pm

    Agree lipid-soluble contaminants would be the biggest worry, DrMonkey.

    Equazen also have this line about “using fish from the world’s least polluted oceans” -though not sure which ones those would be these days. I can believe coastal fish from (e.g.) polluted estuaries would have more toxins/pollutants than ocean fish.

    Which reminds one that the people eating the MOST polluted fish are likely to be the 3rd world poor living near industrial zones, who presumably rely disproportionately on local coastal fish pulled from heavily polluted waters. Double whammy.

    As was mentioned about, the FishOilersall mention “molecular distillation” to remove organic impurities like PCBs – for another example see


    My n yrs back Chemistry degree didn’t mention “molecular distillation”, although I remember plain old “distillation” well enough. I don’t know how well this removes lipophilic PCBs, aryl hydrocarbons, pesticide residues etc. etc. but maybe someone else hereabouts does.

    Of course, it’s rather a pity we can’t NOT pollute waters with all this shite in the first place. On that at least I’m with the Greenies.

  12. doctormonkey said,

    October 11, 2006 at 10:19 pm

    equazen have a nice pdf flyer with their own processing, not a formal rundown but quite informative and as they are the focus of the attention and we seem to be managing to damn them and Durham LEA perfectly well with their own words…


    essentially, they seem to remove the protein (where the naughty mercury lives) and remove the “bad” fats from the oils but i’m not sure what they do that removes all of the lipid soluble badness

    they also give their source fish and oceans: sardines, pilchards and tuna from the south pacific and atlantic

  13. jdc325 said,

    October 12, 2006 at 10:29 am

    #61 – From reading the PDF you referenced, it seems they don’t explicitly state that any one particular process will clear dioxins, PCBs and furans from the oil. Apparently, however, the filtration stage uses a natural source of calcium carbonate (from the ‘special clay’) that “marries with any unwanted contaminants”. To me that this implies that this treatment stage will remove ALL contaminants (including the ‘nasties’ such as PCBs etc…). Does anyone know if this would actually work? Or by contaminants do they just mean heavy metals?

  14. doctormonkey said,

    October 12, 2006 at 7:37 pm

    #61 / #63 – but the pdf i referenced in #62 is from equazen’s website about their process and it does not have such a section

    so far as i know, some such clays and things will absorb some amount of some nasties but i am not sure it is a complete panacea, especially as it might also cleanse the product of the good things too!

  15. Michael Harman said,

    October 15, 2006 at 11:18 pm

    The illustration is interesting, but don’t you think the guy is taking a hell of a risk? What if someone crept up behind the fish and shouted “Boo”?

  16. jdc325 said,

    November 14, 2006 at 1:26 pm

    Ridiculously late, I know but I came across a link to this today (refers to safety concerns over fish that several people have raised on this thread).

    JAMA 2006; 296: 1885-1899

    Refers to fish consumption rather than fish oil consumption, but thought it may be of some interest.

  17. FrogJacuzzi said,

    May 25, 2012 at 12:00 pm

    Dr Aust,

    What on Earth is wrong if people DO ‘just say “oh stuff it” and go back to eating the one whose taste they prefer.’?

    I wish they would! I wish people would explain themselves for eating butter and drinking red wine by saying, “Because I like them!”

    Instead they have righteous anger against margarine, because they feel they were mislead for years, they spit the word ‘hydrogenated’, and they claim that magarine actually tastes foul!

    Unobtrusively bland, more like! Talk about trying to win an argument by overstatement. I wish people would shut up about butter AND margarine. Neither is causing all the fat bastards.