Anyone should be able to get the gist of it, but Googling may be required for non-mathmos to understand some technical terms. Hell, I didn’t know what a Mantel-Haenszel test was until now.

I also like the look of this other paper about statistical errors in medical literature.

It’s called education, Steve. Or if you prefer the political form “Education, Education, Education”.

Ben’s point is not that analysing subgroups per se is bad science – but rather how you analyse the subgroups. Re-read post 44.

There is not just one disorder called “ADHD”, there are three subtypes of ADHD – inattentive – predominantly hyperactive and combined typed – ADHD is a relatively new disorder for research, so it is possible that maybe the three subtypes of ADHD are not even the same disorder.

So I disagree that this paper is a result of overanalyzing, when the results could also justify further research into the different subtypes, which may show that in fact the subtypes of ADHD are different disorders, with different etiologies, causations etc – and this which is why ADHD Inattentive type benefits from fish oils, and the others do not. Of course further research is required to replicate these findings, but I do think Ben, that this article is rather subjective due to your bias against previous research involving fish oils.

I also think to suggest that investigating subtypes is overanalyzing is a weak – would you claim that investigating effects on subtypes of gender, education etc is also over analyzing, too?

But I can understand the pressure on researchers, certainly in the medical profession. Names on papers is essential to progress up the career ladder towards consultanthood, and much research is done not because the medic involved has any particular interest in the outcome, but because it looks like a fruitful source of journal credits, much destined to be filed and forgotten except as a suffix to a CV.

Exactly. It also may be foolish, and possibly unethical, to NOT explore the data and discover what hypotheses subgroup analyses might suggest. But that means that some post-hoc finding is simply yet another way to discover a possibility, exactly the same as the way that broadly sampled epidemiological studies can be used to indicate possible avenues for future research. Ideas can come from anywhere.

The next step is to define the hypothesis, and test it as an a priori hypothesis: take the fish oil, define this supposedly specific subgroup effect, recruit a bunch of ppl in this profile, then randomize them to fish oil or olive oil.

Just to play with the true believers, take their declared effect size from this post-hoc analysis, add a bit of error band, and power the study to find at least that degree of benefit.

If it works, it works. If not, then: Good night! Thanks for playing science! Next?

Alpha level adjustment helps to some degree. But is not the total solution. You might just happen upon a true, worthwhile effect that would be ruled out by the stricter alpha, if that is your total criterion for judging the worthiness of data-mining hypotheses-worthy-of-future-study. Also, there will be spurious effects that are mathematically strong enough to survive the stricter alpha level.

Thus, best to dredge data for indications of true phenomena, re-read Bradford-Hill, and develop the next study.

‘“You can see we did pretty poorly overall,” they might say: “but interestingly our national advertising campaign did cause a massive uptick in sales for the Bognor region.”’

This is only a useful result if you had a solid theoretical reason why Bognor may be systematically different from the rest of the country before you started looking at the results. Otherwise it’s purely data dredging.

darn…and there I was thinking satisfaction was only available to people with really really smart dogs that had long long toes and that didn’t mind wasting a few hours reading about all the silly things that obsess people!

Can anybody post the actual data so that we can all find our own subgroups? The obvious ones are people who get worse or improvers in control group. But I am sure that we could squeeze out others.

The best ones could be published in the Saturday column.

@heng

So true. Richard Feynman is, and will always be my greatest hero. His writings are (dare I say it?) magical.

@13, 15 is it statistical significance? A minority of patients may respond well to a drug above a predetermined significance level, overall there is no significant effect, and outside the minority there is no significant effect. And this might be a real effect or a statistical accident. The drug could be doing everyone some good but too little to be detected in most, or nothing for most, or it could be doing a small amount of harm, undetected, as well as good. You’d have to do a larger trial to have a better chance to find out.

I thoroughly recommend reading some of Feyman’s books e.g. www.amazon.co.uk/Surely-Youre-Joking-Mr-Feynman-Adventures/dp/009917331X