If the fish oil issue is not true, despite the general media claiming it to be so, why has it not been challenged – until now – by the ‘responsible press’?

“Fish oil helps schoolchildren concentrate” (News), claimed that research published in the American Journal of Clinical Nutrition showed omega3 fish oil could combat hyperactivity and attention deficit disorder.

We wish to make clear that the study did not include ADHD. Additionally, algal DHA, not fish oil, was used in the study and the simple attention task employed during MRI scans did not specifically identify improvements in attention. We have removed the story from our website and apologise for these errors.

Yes and no. A small study will usually only a significant effect if the observed effect size is large. So you are right about the statistical power to _detect_ an effect. But, for the same reason, small studies that reach statistical significance tend to overestimate the true effect size (even if you discount important factors such as publication bias). An alternative conception of power is to think of the confidence interval around the observed effect. Typically it takes large studies (or meta-analyses of small studies) to get a narrow, precise confidence interval. So even having detected an effect you can't be too sure that it has any practical importance if the study is small.

This is neatly illustrated by the study that Dudeistan cites where the 95% CI for Cohen's d runs from 0.01–0.92. Cohen's d of .01 is probably equivalent to zero for practical purposes (being 1/100th of a standard deviation difference between groups). The study does not have sufficient power to estimate the size of effect precisely – so yes it is underpowered if the goal is to determine whether the treatment has any practical value.

Ben I believe you have already covered the study showing the lit-up brain parts of the dead salmon run through te MRI just for grins. Here is direct link to the pdf.

The lead author Bennett has more sober MRI/false positive stuff, but still awesome stuff, at his website…
prefrontal.org/blog/

Aren’t these some? Perhaps things aren’t totally black.

Telegraph: Fish oil capsules ‘don’t boost brain power in children’

Daily Mail: Fallacy of fish oil revealed as study finds supplements DON’T boost children’s brain power

BBC News: ‘More omega-3 research’ needed, says Prof Amanda Kirby

Further to what Dr DLD said (and I hope he’ll correct me if I’m wrong):

The 1-in-20 figure would be true if and only if all trials that yielded a “statistically significant” result yielded a p-value exactly equal to 0.05. However, 0.05 is the conventional cutoff point, not the result of the trial itself.

In other words, rather than “1-in-20 studies will give you a statistically significant difference between the samples, even if there is no effect”, it’s “1-in-at-least-20-and-probably-more-depending-on-the-p-value-you-actually-got studies will give you a statistically significant difference between the samples, even if there is no effect”.

(of course, this assumes your statistical techniques were valid in the first place, and you then have to look at bias and confounding, but those are separate issues)

It was me that moved the topic away a little, not you.

@ Guy

Well your explanation pretty much seals my fate as an amateur! Thanks

Of course, if you set your p value at .05 or .01 you have a 1 in 20 and 1 in 100 (respectively) chance of obtaining a significant result by chance, that is what the alpa is – aprobability statement. However, that is unrelated to ‘power’. This is one of the reasons why replication is very important – rather than sample size per se. Sorry to have moved away from the original article (which was great). On a seperate note – also great to see Ben likes Momus (I noticed from his tweet yesterday) – one of Britains most interesting and unsung lyrical geniuses.

Thanks for that explanation. Very helpful. I shall look into the book.

@tomrees

Is this referenced anywhere?

I’m a reasonable statistician, and I would argue that if a significant result was obtained, then by nature the study was adequately powered. Small sample sizes only really matter if no sig result was found (as by definition, power is just the probability that you will find a sig effect if one exists). In behavioural research, for some factors, you can obtain very large effect sizes on very small (n <10) sample sizes.

Hope that helps. Andy Fields (2009) book 'Discovering Statistics' is a very good start to the area.

I recently criticised a paper (see reference below) on acupuncture for depression in pregnancy. There were only 50 in each group. It seemed to me that the sample sizes were too small given that there was not a huge difference between intervention and controls.

Here are the stats breakdown:

Women who received acupuncture specific for depression experienced a greater rate of decrease in symptom severity (P<.05) compared with the combined controls (Cohen's d=0.39, 95% confidence interval [CI] 0.01–0.77) or control acupuncture alone (P<.05; Cohen's d=0.46, 95% CI 0.01–0.92). They also had significantly greater response rate (63.0%) than the combined controls (44.3%; P<.05; number needed to treat, 5.3; 95% CI 2.8–75.0) and control acupuncture alone (37.5%; P<.05: number needed to treat, 3.9; 95% CI 2.2–19.8).

I wonder whether any statisticians would argue that these results suggest that the sample sizes were adequate?

I know the point of Ben's article is that the Observer quoted a basic science study for sensationalised new that was not directly addressing efficacy of fish oil/omega 3 for underachieving kids. But that aside, looking at the study itself the sample sizes seem very small regardless of what the study was hoping to explore.

journals.lww.com/greenjournal/Abstract/2010/03000/Acupuncture_for_Depression_During_Pregnancy__A.7.aspx

I’ve linked the funniest paper i ever had to discuss. I promise it will make you laugh when you see a diagram of the machine custom built to get a reliable control (and maybe cringe).

www.scribd.com/doc/32587138/stensballe

I also asked the companys clinical nurse specialits in a big meeting what the different outcomes would really mean to a patient, since a nurse might ask.When she finally got the point that I didn’t care about the numbers themself, just the pain they would translate to my idiot national sales manager started glaring at me to shut up and stop being clever, muttering something about nurses never asking stuff like that. Anyway I though he was being a prick so I bet I’d personally do some comparative analysis if he would. He wouldn’t take me up on it though, wimp.

(seriously click the link it will make you laugh, and yeah, didn’t last at that company long)

Unfortunately I might have to return to sales for a bit till I can get on a decent PhD. Ben, I promise I wont even bother with the papers if I do, to be hosest it would probably mean I’d get better results!