Moments of genius

February 8th, 2010 by Ben Goldacre in bad science, onanism, podcast, systematic reviews | 25 Comments »

Sorry no column this week, I’ve got some fun stuff in the pipe, as they say, and a lot on. In case you miss me, here’s my shouty contribution to Radio 4’s “Moments Of Genius”, a eulogy to the startlingly new idea of systematic reviews.

Other bits and bobs…

…I’m on Quote Unquote this week (bizarrely) and also next week, pimping Momus and Cory Doctorow among others, but I recognise your mileage may vary.

…Also thanks to everyone who pointed out that I was the answer to a question on Brain Of Britain last weekend. Nobody got me. This is one instantiation of a wider phenomenon described by the great Danny O’Brien.

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

25 Responses

  1. Marky Mark said,

    February 10, 2010 at 4:54 am

    Look what some scallywag has done:

  2. nevajism said,

    February 10, 2010 at 4:56 am

    Hi ben- sound like a great idea- in the best of all possible worlds- but in this world we live in there is a big problem with relying on a few experts to sift through the data and interpret it for us- it puts to much power in the hands of a few people and leaves them exposed to the possibility of coersion or bribary from interested parties- far better have a de-centralised system something like www where people debate and share ideas – the fact that the government is trying to limit the scope of the internet is a pretty good sign that it is working well imo. For a pretty good analysis on the advantages of individualism over collectivism I would reccomend the works of Ayn Rand (of course the argument for collectivism is argued by people like Carl Marx and it is interesting to note how many contempory politicians have close links to this philosophy- e.g. look into the milliband brothers’ father and the books he wrote)

  3. calj said,

    February 10, 2010 at 10:41 am

    Yeah, I like cars. Now I suspect that most of us are far more likely to lose out (financially, definitely and, probably, physically) from our car malfunctioning, than we are at the hands of quack medicine.

    My solution? Everyone should take classes in car mechanics and read up about car engines in their spare time. However most people don’t have the time, there aren’t enough colleges in the country, and new innovations mean that what you learnt last year is already out of date (and if cars, why not plumbing, electrics, etc?)

    So…those of us with busy lives have to rely on the government to properly legislate the car mechanics industry. As a result, we expect our cars to be roadworthy if a government certified mechanic says so.

    The system isn’t perfect but it works most of the time. The same should be true for medicine. The vast majority of people have no interest in clinical trials, and will not, no matter what you say, learn how to analyse the 8 billion confusing or contradictory (or just plain dishonest) trials out there. Don’t you think we have better stuff to do with our time?

    It’s not up to the individual. It is up to the government to properly legislate this industry. We can’t be experts in everything. If something doesn’t work it’s up to scientists and doctors to say so and get it banned.

  4. paulhardy said,

    February 10, 2010 at 12:58 pm

    nevajism – did you listen to the clip? Systematic review is not interpretation; the authors predefine their method for finding papers, and their method for disregarding papers due to low quality. Following that the remaining trials are assessed statistically. This is an application of the scientific method.

    The www approach you advocate follows the legal model – everyone presents their biased arguments and the right answer is the one we mistrust the least. This model hasn’t resolved the MMR-autism hoax, or countless other examples.
    And anyone invoking Ayn Rand in their support has to be ragarded with circumpsection imho 🙂

  5. nevajism said,

    February 10, 2010 at 10:43 pm

    Quis custodiet ipsos custodes?

    I’m dissapointed in you guys, given a pile of bricks and cement you would probably build yourself a nice comfortable prison to live in. Just remember the book in orwells 1984:

    first chapter title: “Ignorance is strength”

  6. Ben Goldacre said,

    February 10, 2010 at 11:23 pm

    nevajism, you’re an idiot. systematic reviews are not about experts giving their opinion on research, they are the polar opposite of that, as i explained in the clip. go away.

  7. nevajism said,

    February 10, 2010 at 11:38 pm

    Hi ben

    the best contempory example of how this type of thing doesnt work is the IPCC (climate change expert body) wouldnt you say?

    do you think i didnt understand what you said in your clip?
    don’t you see the point I am making?

  8. nevajism said,

    February 10, 2010 at 11:41 pm

    guess theres not to much room on here for debate unless you agree with ben so im off- feel free to block me if it makes you feel better –

  9. nevajism said,

    February 11, 2010 at 2:21 am

    Hi Ben, just wondering , what do you think of what I said about the “danish study” on thimerasol in vaccines on the other thread, I know you are aware of the study cos ive seen you talk about it, did you notice the problem I pointed out with it? its not obvious if you just read the abstract but it is a pretty fundemental statistical error which only becomes clear once you start looking at the methods section.

  10. paulhardy said,

    February 11, 2010 at 7:51 am

    nevajism – we’re ready to debate with anyone who has something better than moronic to bring to the conversation. Ben discussed systematic review & you conflated this with collectivism. This isn’t so much apples and oranges, more like apples and snowboarding trousers.

    And no we still don’t think you understand what Ben said in his clip. Your espousal of the “danish study” demonstrates that; show us where this statistically flawed study has been incorporated uncritically into a systematic review and we’ll reconsider the view that you’re talking crap.
    Until then you have all the credibility of an alternative health practitioner.

  11. paulhardy said,

    February 11, 2010 at 8:20 am


    “Quis custodiet ipsos custodes?”

    Peer review.

    Delphinium natare doces.

  12. Welsh_Medic said,

    February 11, 2010 at 11:36 pm

    Hey, completely unrelated to the above post, but I’m a UCL medical student, and I’d just like to express my appreciation for the lecture that Ben gave today. As far as it’s possible to have a hero in this jaded, post-modern world, you’re one of mine, and it was great to hear you speak, especially as I’d been inattentive to the timetable, and didn’t know it was happening. Thanks. I hope this doesn’t come across too fanboy-ish.

  13. Psythe said,

    February 12, 2010 at 5:21 am

    I love the idea of systematic review since I read about it in Bad Science. However the bit you quoted about providing “clean, clear information” does appear to be optimistic in some cases. A case in point is the Cochrane study on Glucosamine.

    Now I was under the impression that Ben wrote somewhere that Cochrane had shown glucosamine was no better than placebo, and (as the Cochrane website has been down for maintenance for a number of months) had dutifully altered my recommendations to (veterinary) clients accordingly.

    When the website came back up I checked the abstract and figured “well, not all bad, but definitely dubious”.

    Then tonight I located the full text of that metanalysis – if anything, rather biased against glucosamine at this point by the fact that I was a dauntless crusader against fuzzy thinking colleagues and naughty drugs reps. But there appear to be plenty of studies which do say there is a benefit to glucosamine over placebo (about 20%), and overall to get a “no difference” result you have to exclude all the studies which use Rotta glucosamine – which seems somewhat heavy handed (even if those studies were all sponsored by Rotta).

    The four studies which compare glucosamine to NSAIDs suggest that they are of equivalent benefit (with much less risk of toxicity).

    So now I’m more confused than ever – and this is before you start adding in MSM, perna canaliculus, chondroitin and ground unicorn. Ben, you’ve obviously given this sort of thing a lot of thought – what do you tell your clients when they ask about glucosamine?

  14. Psythe said,

    February 12, 2010 at 5:23 am

    Addendum. I meant to point out that although I thought it was Ben who had written about glucosamine being no better than placebo, I’ve not been able to find the source (I got Bad Science out of the library 🙂 )

  15. nevajism said,

    February 12, 2010 at 6:43 am


    My own experience with the peer review process is that it is often far from ojective, a senior collegue of mine reckons that it is always easier to get a bad paper published than a groundbreaking one – because the latter will ruffle a few feathers.

    I would say that in the context of science publications the peer reviewer would be the judge and the question
    “who judges the judges?” would be the journal editor
    but in practice this doesnt really happen and even if it does the question can then be taken to the next level.

    if you want to see some of the indider dealings relating to the peer review proccess then the leaked emails from the University of East Anglia make interesting reading

  16. pberry said,

    February 12, 2010 at 9:08 am


    At the risk of feeding the troll, could I ask what you would propose as either a better system of peer review or where we need to make improvements to the existing system.

    The point is that peer review isn’t perfect–no-one’s claiming it is–but it’s the best system we’ve yet devised. Of course you’re going to get problems. It does, after all, have to involve people with their biases and flaws, even the conscientious ones. It’s like pointing out the flaws in democracy as a system of government: anyone can spot the cracks but putting forward solutions is far harder.

    Could you add something constructive? I can’t, but I being honest in stating so. How about you?

  17. nevajism said,

    February 12, 2010 at 10:38 am

    Hi pberry
    I’m glad you asked that question because it is of course completely valid. The solution is in both the political and medical cases I believe free market capitalism also known as libertarianism and is being championed by people like Ron Paul (who is a medical doctor btw) in the states, and people like Dan Hannan in this country. The basic idea is that big government centralises power and always gets hijacked by big business creating the form of monopoly capitalism which we have currently. Specifically with respect to published ideas I would say that you already have the makings of a good system on the internet where anything can be published and the ideas which resonate with people are the ones that spread- there is no need for intermediaries to decide what is and is not valid, a nice little summary related to this can be found here:

    sorry if you people consider me a troll- but a debating forum would be a bit dull if everyone had the same opinion don’t you think? have you ever heard of the concept of group think

  18. jweirmccall said,

    February 12, 2010 at 11:40 am


    Your link doesnt appear to be even remotely linked to this thread, just containing a wooly arguement regardiing collectivism vs individualism. And ‘no such thing as forests, only trees’, I’d best let my chemistry colleagues know that theres no such thing as compounds, only elements, the compound is merely a concept.

    But to get back to your arguement of spreading ideas due to resonance being a sign of rightness has quite a few holes, as multiple systematic reviews have shown. Such as steroids for head injury (a very resonant idea, head injury causes brain swelling which is a bad thing, ergo a drug which reduces inflammation will be a good thing), and beta agonists for asthma (an idea so incredibly resonant that it has spread throuhgout the world, and which does work on the immediate level [eg relief of asthatic symptoms] but which systematic reviews show to slightly increase long term risk of death, making it a second line drug now).

    The second point against this idea is that, to make a decision at any one point, would require knowing the most widely held opinion of the community. This would of course require a moderator to look at all views, and summarise them, otherwise how would the person know if they had accidently read a non-representative portion of threads that didnt resonate with the greater community. This moderator would wield an immense power, arguably like someone doing a systematic review, but with no details of their methodology and no reasoning behind their conclusion beyond it being ‘resonant and widely held’.

  19. pberry said,

    February 12, 2010 at 12:03 pm


    I categorically disagree. There is clearly not enough regulation of the free market (in any sphere you care to name). Even those organisations which do exist to supposedly oversee various systems seem to lack a solid foundation, a clear purpose, and are not armed with any genuine sanctions to keep in check those who fall foul of their guidelines.

    We’ll agree to disagree. Little point entrenching ourselves.

  20. paulhardy said,

    February 12, 2010 at 3:19 pm

    Psythe – see p9 column 2 first para; Rotta were contacted for papers & supplied 1 unpublished paper from an RCT. A published abstract was found for it so this appeared in the final analysis. Rotta also supplied 3 other unpublished technical reports of RCTs but this fact, and the lack of detail in them, meant that these failed to get into the meta-analysis. TBH the authors appear to be trying to be open by discussing these trials separately despite their shortcomings.

    nevajism – thank you for reinforcing the original point of the article with your comments about peer review; by means of a selective, impressionistic semi-anecdote/reported opinion about peer review you hope to offer a critique of it. A cursory glance at PubMed would reveal a large amount of literature about the subject which documents its shortcomings and engages in trying to tackle them.

    Similarly a cursory glance at the comments columns of these pages would reveal that groupthink is very far from what has gone on here.

    I think the reason why the word troll was used is that over a series of posts comprising what amounts to an off-topic polemic you have yet to convince that you know what a systematic review is, or why it is non-interptretive. Plus the fact that it seems like you are drawing the conclusions that your preconceptions expected, rather than what has been said.

    But to be frank I’m bored with this now so if you respond I’m sorry it will probably be for your own benefit.

  21. paulhardy said,

    February 12, 2010 at 3:22 pm

    jweirmccall – very good post

  22. nevajism said,

    February 15, 2010 at 11:03 pm

    Im sure I posted a reply here- has it been censored?

  23. Ben Goldacre said,

    February 15, 2010 at 11:05 pm

    nevajism: nothing has been censored, but i do find you very boring – by which i think i mean stupid, and arising from that also pointless, in fact, stupid in a way that is so confused that it’s not even coherent enough to be very informative on the interesting issue of how to deal with stupid people – and this could have been an interesting discussion about systematic reviews. as a matter of manners, it would be great if you could go back to the MMR thread to post your thoughts on the dangers of MMR, where you were before? thanks, bye.

  24. Psythe said,

    February 20, 2010 at 4:15 pm

    Paul Hardy – I agree that the researchers are trying (and succeeding) in being very open and unbiased. It just makes it hard to draw conclusions from the work.

    There appear to have been at least 8 studies involving a Rotta prep (pg 12) – five using Lequesne index and 3 using WOMAC scores (two of which also looked at joint space).

    One oddity is that they don’t appear to have separated those Rotta studies with adequate allocation concealment from the general Rotta subgroup. This would have seemed to be a sensible approach.

    However, leaving Rotta to one side for a moment and looking at the good quality studies, to make any conclusions one way or the other one still needs to decide that WOMAC is more relevant than Lequesne or vice versa.

    If (like I pretty much have) one gives up on making head or tail of the full paper and trusts the Cochrane researchers to make a fair summary in the plain text section, their conclusion does seem to be “grey area” rather than “no good”.

    I guess there’s nothing wrong with this in terms of what they could glean from the data available – it is to their credit in fact that after all that research they didn’t come up with something more concrete either way. I wish I could find the original source which made me think they’d said concretely that glucosamine was useless though, because this would seem to be just as flagrant a cherry-picking as the homeopaths are guilty of.

  25. nevajism said,

    July 12, 2010 at 12:32 pm

    somehow i dont think that in 100 years time the inventor of the metastudy will be remembered as a great innovator or genius – probably more a histirical footnote on how sophisticated propaganda methods evolved in order to push a coperate agenda towards a single and favorable view within the scientific community