Excluding Bias

November 26th, 2005 by Ben Goldacre in alternative medicine, bad science, homeopathy, references, statistics, telegraph | 67 Comments »

Ben Goldacre
Saturday November 26, 2005
The Guardian

The moment I saw the press release for the new Bristol Homeopathy study, I knew I was in for a treat. This was a fabulously flawed “survey”, no more, in which some doctors asked their patients whether they thought they’d got better a while after having some homeopathy. Not meaningless data in itself, but the action, as ever, is in the interpretation, and the interpretation was at its most cock-eyed in the Daily Telegraph.

“The other reason the survey is causing such delight,” gushed Elizabeth Grice, in her public love note to David Spence of the Bristol Homeopathic hospital, “is that it contradicts a scathing report published in the Lancet recently by Professor Matthias Egger.” Now scathing is not a word I would use to describe a rather sober and – by design – tedious meta-analysis. But she is also, weirdly, suggesting that a large systematic review and meta-analysis of a huge number of placebo controlled randomised trials is somehow contradicted by a survey from some homeopaths of their customers’ satisfaction.

Did she feel there were any flaws in the Bristol study, any need for balance? Yes: “This has led to a medical ding dong in the long-running debate about the value of homeopathy, with Egger (known in the profession as Eggy) accusing Spence and his colleagues of failing to use a ‘control group’ for comparison and Spence retorting that his huge observational study – the largest of its kind ever published – involving 23,000 consultations with no exclusions and no bias, is a pure measure of achievement. ‘It’s what I call a ‘real world’ analysis’, he told me. ‘It’s what happens.’ ”

Now the first lesson for sceptics here is, if you contradict the enemy, they will give you a funny name. And people call me childish. But on to business. Is it bad not to have a control group? Yes. Read the academic paper via www.badscience.net/?p=188: they were looking at a lot of chronic cyclical conditions, or time-limited ones, like the menopause, where people get better with time. If 70% get better that’s meaningless. 99.99% of people who get a graze to their knee will get better and 99% of people who get a cold will get better. It’s not enough to know that they got better. We need to know if they got more better, or better faster, than people who weren’t having homeopathy.

And what about this business of “no exclusions” and “no bias”? These are simple technical terms from evidence based medicine, and in fact there were stark staring heinous examples of both “exclusions” and “bias” in this study, Grice. Where did the patients come from? They were selected as patients who wanted homeopathy, and so were positively disposed towards it: this is “selection bias”, picking subjects who will give you a positive result .

And what about the data collected: did they measure how patients were at baseline, and compare how they were later in time, at follow up? No, they just asked patients later to remember how they were when they first came, and decide retrospectively whether they thought they were any better: this will give you “recall bias”, and also another form of “information bias”, as patients give the doctors the answer they think they want or deserve.

Lastly, a large number of patients never came back after their first appointment: and so they were simply, er, ignored in the analysis. That “exclusion” is the very opposite of a “real world analysis”, otherwise known as an “intention to treat analysis”. Did they get worse? Did they get better? Did they go home and die? We will never know.

Send you bad science to bad.science@guardian.co.uk

Learn more on how to critically appraise research with How to Read a Paper: The Basics of Evidence-Based Medicine by Trisha Greenhalgh (BMJ Books).


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

67 Responses



  1. Harry said,

    November 26, 2005 at 10:48 am

    I find it interesting that she puts ‘control group’ in scare quotes. And describes it as an ‘accusation’ that they didn’t have one, rather than an observation. Some of the ways a study can get distorted, like the biases you mention, are fairly subtle and un-obvious, but the idea of a control group is pretty straightforward, you’d have thought.

  2. Santiago said,

    November 26, 2005 at 11:04 am

    This is a good manner for testing the interpreting capabilities of people.
    It is really difficult to be blind for that clearly flawed study.
    What about doctors who don´t see the error?

  3. CB said,

    November 26, 2005 at 12:33 pm

    My personal favourite is the link to ‘debate about the value of homeopathy’ – which takes you to this:

    Homeopathy has been proved more successful and cost-effective than conventional medicine in the first comparison of the two approaches.

    Proof of its effectiveness has emerged from an extensive study of its use in treating chronic disorders such as back pain. The study ignored the question of how homeopathy might work and focused on how well it performs.

    Researchers in Germany recruited more than 400 adults and children with long-term health problems ranging from sinusitis to insomnia and depression. Half were treated using conventional therapy; the other half were treated homeopathically. After six months, the condition of the patients treated homeopathically had improved significantly more, and more quickly, than the others while the cost for each was similar.

    Yeah – thats a debate.

  4. Mojo said,

    November 26, 2005 at 2:06 pm

    I’ve seen some German studies comparing homoeopathy to “conventional” treatments (there have been quite a few using similar methods) discussed elsewhere quite recently. The suggestion was that they choose conditions likely to be susceptible to the placebo effect or that will resolve anyway (e.g. mild viral infections) and compare the results of homoeopathy to the results of an ineffective “conventional” treatment. Hey presto! Homoeopathy works as well as something that doesn’t work!

  5. Morag said,

    November 26, 2005 at 2:37 pm

    Not only that. Homoeopathy given after an hour-long ego-massage consultation works *better* than something that doesn’t really work given after a quick five minutes with an over-worked GP.

    You have to give them points for ingenuity, if nothing else.

  6. BSM said,

    November 26, 2005 at 4:29 pm

    www.trusthomeopathy.org/csArticles/articles/000000/000066.htm

    See how the despicable Faculty of Homeopaths has chosen to report this.

    “Homeopathy improves health of 70% of patients in hospital study

    The results of a six-year study at Bristol Homeopathic Hospital (part of United Bristol Healthcare NHS Trust), one of five NHS homeopathic hospitals in the UK, show that over 70% of patients with chronic diseases reported positive health changes after homeopathic treatment.”

    The body text is fairly careful about claiming a causal link between homeopathy and patient improvement and, to its slight credit, that is also true of the original article.

    But, just look at that headline. There is the Big Fat Lie

    This has been true throughout the mealy-mouthed PR that has accompanied the publication of this piece of trash research. The article itself is rubbish, but if you read it in detail you will find it is relatively circumspect: it even formally acknowledges its limitations;

    “Study limitations
    This study was designed as a longitudinal observational
    study that set out to examine a large cohort of patients (with
    no exclusions) to try to assess the health changes experienced
    by patients undergoing homeopathic treatment in realworld
    circumstances. Comparison groups were not included
    by design, although such a design has been successfully implemented
    in homeopathy research.34 Issues concerning observational
    study design are well known and are documented
    elsewhere.35
    The limitations of this study include potential bias introduced
    by a “patient-with-doctor” generated measure…” yadda yadda yadda you get the idea.

    We should note in passing that it does not admit that these limitations are utterly fatal to its purpose, but in writing the paper in this way they are pretending they have done a piece of grown-up science and want to talk to their science peers as equals.

    However this circumspection does not extend to the woos’ promotion of it where they would like us draw the inference that they did prove a causal relation. What they have proven, yet again, is that homeopaths are lying gits.

  7. BSM said,

    November 26, 2005 at 4:32 pm

    Morag and I wold like to apologise on behalf of our profession that the current President of the Faculty of Homs is a vet.

  8. Frank said,

    November 26, 2005 at 4:45 pm

    It strikes me that homeopathy will always be more cost effective than conventional treatments, as it’s hard to beat the price of mains water, currently coming in at about 60p per cubic metre. At that price it could have nothing greater than a placebo effect and still be more cost effective….

    Once I get some web space sorted out I’ll share with all of you the crazy take on control groups exhibited by one Dr Peter W. Joller, where he attributed the 75% drop in illness to a herbal elixir in the treatment group, and “participation effect” in the control group.

    Who is giving these people PhDs?

  9. BSM said,

    November 26, 2005 at 6:20 pm

    “Last year, there were 44,887 new or follow-up appointments at Britain’s five homoeopathic hospitals, in Bristol, London, Glasgow, Tunbridge Wells and Liverpool. ”

    To put this in context. 5 hospitals, 52 5-day weeks per year means 35 appoinments per “hospital” per day. Deeply impressive workload guys…not. How can they cope?

    To put it in context, this represents the equivalent of 1 real GP consulting for 6 hours. I hope it’s bloody nice tea they are serving with the amount of resources those institutions are soaking up.

    The Royal London Hom Hospital is undergoing a £20M refit. So 1 GP’s worth of work is having £20M lavished on it as a capital expense.

    Ben, do you know the annual budget of these sugar retail establishments? With that as the numerator and 44,887 as the denominator, I’d be very interested to know how much I, as a taxpayer, am paying for each appointment.

  10. Ben Goldacre said,

    November 26, 2005 at 7:05 pm

    “Who is giving these people PhDs?”

    I am. Anybody who wants a PhD, I will give one to them.

  11. Rosie said,

    November 26, 2005 at 9:13 pm

    Can I have one? I can’t afford one from McKeith’s alma mater and no-one seems to have opened Kwik Save U yet.

  12. Morag said,

    November 26, 2005 at 10:03 pm

    Back to what BSM said, I was thinking, 6000-odd patients, supposedly everybody who persevered with the water-shaking, over a 6-year period. Just over 1000 patients a year. Isn’t that more like 20 a week? For a whole hospital?

    At least Lewith has the honesty to admit that SCAM does increase overall costs to the NHS, unlike what jugears paid that PR company to say.

    BSM, talking of one Spence, D – do you remember seeing this?
    vetpath.co.uk/voodoo/vettimes5.html#baker8

    “A preliminary audit investigating remedy reactions including adverse events in routine homoeopathic practice.” Thompson, E., Barron, S. and Spence, D. (2004) Homoeopathy 93: 203-209. In this article we learn that: “Reactions were frequent: 28 out of the 116 patients (24 per cent) experienced an aggravation. Thirteen patients (11 per cent) reported an adverse event, even though five of those were patients who also reported an aggravation followed by an overall improve­ment of their symptoms. Thirty-one patients described new symp­toms (27 per cent) and 21 (18 per cent) a return of old symptoms”. Also, “Remedy reactions are common in clinical practice; some patients experience them as adverse events.”

    Fancy that!

  13. amoebic vodka said,

    November 26, 2005 at 10:26 pm

    Er…can we have a PhD? Each of us? Please?

    *goes to count how many amoebas we actually are*

  14. Michael P said,

    November 27, 2005 at 1:18 am

    Um, I think I have PhD already. If I’m wrong, can I have one anyway? You see, I have a BSc. (Hons, thank you very much!), but because I work for a university I get mail for Dr. P. all the time, just in case they insult me or something! It’s great! I don’t have to be on near-minimum wage for 4 years OR lay out loadsamoney and sit in waiting for the mail to arrive! Instant kudos :-D

  15. CB said,

    November 27, 2005 at 2:47 am

    I’ll have another PhD please – if only to piss the wife off!

  16. BSM said,

    November 27, 2005 at 8:58 am

    “BSM, talking of one Spence, D – do you remember seeing this?”

    I hadn’t made the connection until yesterday when I was searchng again through for old stuff with which to hit Saxton and Hoare, whereupon the name Spence, D jumped right out at me.

    Now here’s a funny thing, wouldn’t anyway halfway reponsible medical academic paper allude to the potential downsides in a report of a therapy- “DrugX appears to be effective in the management of congestive heart failure, but previous reports reveal that 38% developed complications and in 18% symptoms returned”

    Do Spence and co do this? Of course not. They keep this dirty little secret hermetically sealed away from the casual glance of Humanities Graduates In The Media (TM). Wouldn’t want to spoil a good story, would they.

  17. Deetee said,

    November 27, 2005 at 9:53 am

    BSM – your point about the startling lack of productivity/activity in these homeopathic hospitals is well made. The paper actually states “This study reflects real world circumstances in everyday clinical practice in a busy NHS outpatient unit.” And this is seeing in a single week the same number of patients a GP would see in one morning! And people dare talk of cost effectiveness and value for money?

    Frighteningly, the study authors go on to say “As a consequence..of this study…similar monitoring of clinical outcomes in routine practice has now been recommended by the UBHT Clinical Governance Committee for other specialties in this teaching Hospital Trust.” So now if one goes to Bristol, one can expect the oncology unit to forgo hard objective measurements on whether one’s malignancy is in remission (such as bone scans or CT scans), and will stand by with a pencil and clipboard, sweetly asking “Do you feel any better now, love?” I for one will not be going near the place, ever.

    I am also concerned with another aspect of the paper, in which the authors state “The appointment times for homeopathy align closely with those for such areas as rheumatology, neurology and respiratory medicine, and are substantially shorter than for psychiatry”, implying that time spent interacting with patients cannot be a relevant factor influencing the outcome of the homeopathic consultation (something they accept in the discussion may be an issue)

    Yet if one goes to the Bristol Homeopathic Hospital’s web site (www.ubht.nhs.uk/homeopathy/Patient%20info/Information%20for%20Patients.htm) you can read that: “The initial homeopathic consultation will involve more detail and will take longer than most other medical consultations you are used to, between 45 minutes and 1 hour in duration.”

    So, who is telling the truth, the study authors or the Hospital? One of them must be lying.

    The web site also shows how people should seek referral to the Hospital, including the advice “It is therefore necessary, as with most other treatments, that you ask your GP for a letter of referral.” The study authors do not state how many patients were referred by their GP (I wonder why not? – I suspect the vast majority) but it stands to reason that those referred must have been receptive to a homeopathic consultation in the first place – evidence of yet more selection bias.

    I am amazed by the paucity of data in the “results” section of the paper, which runs to a whopping 22 lines of text. There is absolutely no information on any of the objective outcomes they say they measured wherever possible (e.g. changes in forced expiratory volumes, which they say they measured in asthmatics) – why not? There are no data on contemporary conventional therapies either, just a small line trying to dismiss them as “outside effects”.

  18. Kumar said,

    November 27, 2005 at 1:30 pm

    Sorry, but when one ridicule homeopathy by interpreting its remedies as sugar pills, water etc. or by mentioning that these couldn’t yet be tested/studied in science, can’t it be thought that, he is just ridiculing science or telling its weakness, inabilities or ignorance, for not understanding science of homeopathic remedy’s working and their effects? Whether he thus aid something to make science as “Bad science” on this issue? Whose duty is it, to know science of any “mass existing system in well distributing people all over the world since long with least adversities–claimed, observed and experienced by its community in million”? Is it not the mandatory duty of science, to know it in its language for further benefit to humanity because most of the public means and work-power are allotted to science for this purpose. Therefore, let us avoid ridiculing and look this system as a good perspective to medical system as well as to people as a whole, esp. by thinking “incoming adversities” on our head in near future. Thinking bit logically & deeply, it may not be in the interest of homeopathy that science of its working and effects is known and endorsed by science due to fear/chance of its hijacking by science’s commercial people, big Pharmacies etc. It may then loose its entity and specialty which is somewhat evident in some other alt. medicines already hijacked. Even though, I feel that, first goal of anyone related to healing system should be “to provide best possible medications/treatments” to humanity as a whole, vet.etc. Just these are some of my thoughts.

    Best wishes

  19. Ben Goldacre said,

    November 27, 2005 at 2:04 pm

    “Just these are some of my thoughts.”

    Kumar, here and elsewhere, I simply cannot understand most of what you say, it’s getting to be a bit like having an interesting conversation with a screaming child in the same room.

    Do go away. Or go here:

    www.badscience.net/?p=190

  20. Kumar said,

    November 27, 2005 at 3:43 pm

    Hi Ben Goldacre,

    Ok, your choice.

    Lastly, to attend screaming child can be more impotant than conversations–inspite interesting.

    Best luck and best wishes. Bye.

  21. Richard said,

    November 27, 2005 at 4:19 pm

    I think Kumar is actually a computer, in which case we should all applaud the significant strides the AI researchers have clearly made in recent years. Almost there lads.

  22. Rosie said,

    November 27, 2005 at 5:21 pm

    Deetee – It doesn’t seem that far out of line, to me, for consultant-level first appointments in many specialities to be ~45 mins long… but I agree that it seems weird to then say this is much longer than you are probably used to.

    If they meant “than GP appointments” that’s fair enough, but you’d think most of the people referred in the study would already have seen a consultant before being referred to the homeopathy place.

  23. BSM said,

    November 27, 2005 at 7:17 pm

    Rosie- What we’re not sure of is how many medics work there. If these ‘hospitals’ each have more than about 4 doctors then they must still spend most of their time brewing those lovely cups of tea.

    I’d also like to know their annual budget to work out the cost per appointment.

  24. Rosie said,

    November 28, 2005 at 9:17 am

    Sure. I was only responding to the point about whether or not appointment times were particularly long.

  25. Rosie said,

    November 28, 2005 at 9:24 am

    Sorry to post again, but this

    www.ubht.nhs.uk/homeopathy/Referral%20information/Referral%20information.htm

    looks like they *do* only have 6 doctors. Who also help run the place as well as directly seeing patients…

  26. Morag said,

    November 28, 2005 at 9:57 am

    Of course they don’t have any inpatients to see to. Only the Glasgow Homoeopathic Hospital has (a few) beds.

  27. Edward Bozzard said,

    November 28, 2005 at 10:10 am

    Ben

    Have you made any progress on where to get phony PHds, or just decided that you issuing them is about as worth while as some of these other institutes?

  28. Dean Morrison said,

    November 28, 2005 at 10:14 am

    from Rosies link I note that at least one of these quacks deals with cancer patients?:
    Dr Elizabeth Thompson – Consultant Homeopathic Physician -Referrals for all morbidities including urgent oncology related referrals
    .. or is this just limited to ones that have excema, or are going through the menopause?
    … were these patients included in the ‘study’?….

  29. Rosie said,

    November 28, 2005 at 11:27 am

    Yeah – if you look in the study there’s a table of what types of problems the people in the study had & oncology is included. IIRC a few % of the patients had oncology referrals.

  30. JonnyW said,

    November 28, 2005 at 12:24 pm

    If any one whats to hear the twinkly quartz eyed guy Dr Spence and also Dr Elizabeth Spencer & others, then you can find them hear

    www.bbc.co.uk/radio4/science/casenotes_20050719.shtml

    TV Doctor Mark Porter visits Bristol Homeopathic Hospital for radio 4’s case notes (july this year) , its about a halfhour long mind but there is also a transcript. Although he ain’t exactly tough on them, (yep I probably deserve some jip for under exaggeration there!!) its worth listening, if you care to hear homeopaths talking about what they do. Lots of talk about “complex intervention” etc. plus plenty guff mixed with the usuall “common sense”.
    Also a mention of the possible “toxic” effects of homeopathic remedies by Dr Spence.
    Check out the links at the bottom of the page hardly balanced reporting!!
    Apologies if this has been posted before.

  31. Mathew said,

    November 28, 2005 at 12:48 pm

    Relevant letter in today’s Telegraph:

    Sir – Your report (Health, November 25) claiming that a recent study provides scientific support for homoeopathic remedies is misleading.

    The data presented by Dr David Spence and colleagues merely show that people reporting ailments to homoeopaths, when asked at a later date if they feel better, tend to say yes.

    Scientific trials need a control group. In this case, it would consist of an equivalent sample of individuals given sugar pills or water (as a placebo). None of the doctors or patients would know which group they were in.

    Only then would we discover whether reported improvements in health are a result of the medicine, the passage of time or the power of the mind.

    With no control group, Dr Spence’s report does not challenge the scientific consensus that homoeopathy has no effects other than those of a placebo.

    Baroness O’Neill of Bengarve, Principal, Newnham College, University of Cambridge, Professor Colin Blakemore, Chief Executive, Medical Research Council, London W1, Lord Taverne, Chairman, Sense About Science, London SW1

  32. Stephen said,

    November 28, 2005 at 1:26 pm

    Hi

    Following a few of the links above was real eye-opener. In particular whilst I am distressed that the NHS offers homeopathy, I was stunned to see that the Bristol Hospital offers a post-graduate course.

    What the hell do they teach?

    www.ubht.nhs.uk/homeopathy/The%20Academic%20Department/Academic%20Department.htm

    They claim this leads to

    MFHom for doctors,VetMFHom for veterinary surgeons and DDFHom for dentists

    Is that a medical masters doctorate?

    I have until recently thought homeopathy was a bit of a joke and given it little notice. Increasingly it worries me however that there are NHS institutes, professional qualifications, and many clinicians in positons of authority promoting it.

    How did it come to this? How can you have NICE on the one hand trying to promote evidence based best practice on difficult and complex treatment regimens, and on the other you have doctors promoting the most implausible, irrational and widely discredited flummery.

  33. BSM said,

    November 28, 2005 at 2:00 pm

    MFHom: Member of the Faculty of Homeopathy

    VetMFHom: Veterinary Member of the Faculty of Homeopathy

    DDFHom: Dental blah, blah, blah

    You have to be a medic, vet or dentist to get these so obviously tey’re really really impressive qualifications

  34. BSM said,

    November 28, 2005 at 2:06 pm

    “Yeah – if you look in the study there’s a table of what types of problems the people in the study had & oncology is included. IIRC a few % of the patients had oncology referrals. ”

    They probably had to ake sure they asked the cancer patients at the right time to make sure they could report them feeling better. Much easier to do when your report carefully excludes any requirement to assess patients at a set fixed point or define an end-point.

    Casual readers may be unaware of this. Homeopathic practice is completely dependent on declaring any reported upswing in the patient’s condition to be the result of homeopathy no matter when this occurs. The homeopath’s only job is to hang around long enough for a random improvement in a chronic condition or to browbeat the patient into telling them they have improved.

  35. Ben Goldacre said,

    November 28, 2005 at 1:02 pm

    Groovy. The cavalry have arrived.

  36. Morag said,

    November 28, 2005 at 2:16 pm

    I once told John Hoare (VetMFHom) in public that if he could tell a homoeopathically potentised remedy of his own choice from the blank sugar pills with sufficient repeatability to win the JREF million dollars, I’d undertake to study for the VetMFHom myself. Still he (and all his mates) refuse to try. You’d almost think they couldn’t really do what they say they can do, which is detect unmistakable and compelling proving symptoms from 30C belladonna and stuff like that.

    Q: What do you call the dickhead who graduated bottom of his class at medical school?
    A: MFHom.

  37. tom p said,

    November 28, 2005 at 2:40 pm

    To find out the Bristol Homeopathy Hospital’s budget, why not make a request to University of Bristol Hospital Trust (www.ubht.nhs.uk) under the freedom of information act? Unsurprisingly there doesn’t seem to be anything about the vast costs involved in this quackery on their website, but maybe some of you are more dilligent researchers than I (this wouldn’t be hard).

    What’s scary is that British Nursing News online uncritically parrotted the press release of the bristol survey here. Pity the poor nurses when they’re fed this flim-flam:
    www.bnn-online.co.uk/news_search.asp?TextChoice=homeopathy&operator=AND&TextChoice2=&Search.x=0&Search.y=0

  38. Teek said,

    November 28, 2005 at 3:29 pm

    Matthew, thanks for putting the Torygraph letter up, it shows that folks in powerful places within medicine and science are not likely to stand for any nonsense.

    one problem though – stable doors and bolting horses srping to mind. let’s assume x people read the original article. what propotion of x will have read the letter, what proportion of x will have been convinced by the reporting of the ‘study,’ and what proportion will change their mind having read the letter that rubbishes the report…?

    seems to me there’s little point in “Baroness O’Neill of Bengarve, Principal, Newnham College, University of Cambridge, Professor Colin Blakemore, Chief Executive, Medical Research Council, London W1, Lord Taverne, Chairman, Sense About Science, London SW1 ” trying to kick up a fuss now – once the story has been printed the damage is done, as in many other cases. the blinking horse could have won the Gold Cup by now, no point in double-locking the barn door by writing a letter.

    let’s hope the heavies that penned the letter want to take some affirmative action to prevent crappy journalism getting printed in the first place…

  39. Paul said,

    November 28, 2005 at 3:47 pm

    Teek
    In deference to Onora O’Neill, she’s been an enthusiastic debunker of media rubbish (see her Reith lectures, for example, collected in the book “A question of trust”).
    What more can such people do? Censor the hacks? The media have to be encouraged to take responsibility for what they spout but it seems unfair to take the signatories of the letter to task for having done only what they were able to do. Or have I misunderstood your point?

  40. Teek said,

    November 28, 2005 at 5:11 pm

    Paul, a little misunderstanding between us i think!

    the signatories have done the right thing, hats off to them. O’Neill is indeed “an enthusiastic debunker of media rubbish .”

    my point is this. if one wishes to debunk media rubbish, or one wants to uphold standards in journalism, there must be ways to do this *prospectively* (sorry, i cant do italics for emphasis) rather than by complaining *after* an innacurate article is printed. don’t get me wrong, i write letters too – i’ve had several published by the Guardian, and it’s definitely a good way to get views across. but the bigger picture involves a preventionn-better-than-cure strategy. in other words, do something BEFORE the stage at which articles like these get written.

    in my opinion (interested to know what everyone else thinks) what people like Ben Goldacre, the signatories to this letter, everyone that’s pissed off by Bad Science need to do is to make sure this kind of thing doesn’t make it to print in the first place. get more scientists recruited to national newspapers, get an ombudsman set up to ensure innacurate or false claims don’t make it into print, ensure science sub-editors (if not correspondents themselves) are either science-qualified, or have recieved formal training in how to critically evaluate publications.

    if we take these steps, letter-writing would be less necessary – my point was simply that despite O’Neill et al’s excellent letter, tomorrow the same newspaper will probably print some bollocks along the same lines, and it’s this that we need to stop.

    so i take your point about the guys doing their best, hope i clarified things and didnt make it worse…!!

  41. Teek said,

    November 28, 2005 at 5:13 pm

    by the way Paul, i also dont wanna see hacks censored, but surely there must be a middle-ground where scientific reporting is of i high standard without pre-or post-print censorship…

    suggestions welcome as to how to achieve this!!!

  42. Andrew T said,

    November 28, 2005 at 5:39 pm

    For those of you interested in taking more courses in placebo, try the University of Westminster
    www.wmin.ac.uk/sih/page-310
    I am curious as to whether the ‘B’ in Homeopathy BSc(Hons) stands for bollocks
    And cracking article by the way Ben. I tried to explain it last week, you did it so much better. Maybe thats why you have a guardian column and i don’t?

  43. Morag said,

    November 28, 2005 at 8:18 pm

    The trouble is, Teek, that anyone can set up a journal, and peer review means exactly that – reviewed by equals. Deluded quacks review the work of other deluded quacks. Even worse, I know of one case where a highly painstaking study debunking the claims that the ancient Chinese ever practised acuouncture on animals was refused by the Journal of the American Veterinary Association because they sent it for review to the very acupuncturists whose erroneous claims it was debunking!

  44. Morag said,

    November 28, 2005 at 8:19 pm

    Sorry, acupuncture.

  45. amoebic vodka said,

    November 29, 2005 at 9:50 am

    If the press release comes out before the study gets published, then scientists etc have no chance to do anything about it. The first they hear of stuff like this is when it appears in the media. Press releases only get circulated to the press after all. Ben’s column is only weekly, so he’s not really in a position to respond the same day these articles get written.

    Secondly, this does nothing about bad science interpretations of otherwise sound research.

  46. Teek said,

    November 29, 2005 at 12:28 pm

    Morag, good point about quacks reviewing quackery – but that wasn’t what i was referring to.

    think of science, and the reporting of science, as an assembly line manufacturing, say, a car. designers (i.e. profs) come up with an idea. engineers (i.e. bench scientists) carry out the work. bloke @ end of assembly line checks for faults or mistakes (i.e. peer review). Jeremy Clarkson reviews car in magazine (i.e Melanie Phillips). Even further than that, a press ombudsman checks that the reviewer (Clarkson) doesn’t make false claims about a 200mph Morris Minor or a fuel-efficent 4×4.

    Morag’s point (i believe) was about the stage at which peer review is done, and if a crap engineer checks a car then he could easily miss the fact that it’s missing an exhaust pipe (i.e. a vital control group). this is a cracking point, but quite separate to what i was saying – that we need more knowledgable people reviewing the work in the press (i.e. Clarkson) than crackpots like Phillips.

    sorry for the long-winded analogy, but my point was this. whatever the quality of the work, however flawed the ‘peer’ review process, the end effect can still be favourable if someone who knows the ins and outs of science writes about it in the press. this is the stage at which the current quality control is non-existant. imagine if, say, a politics correspondent, who’s graduated in politics, suddenly started reviewing cars instead of Clarkson – would you trust what they wrote? this is what happens to science – even good science gets written about badly, while sh!t science gets reported uncritically.

    amoebic vodka has a smashing point too – too many science stories are written on the back of a summary of a press release describing an abstract of a speculation. hardly any journos actually read papers and critically evaluate them before writng scathing and/or glowing headlines. some of the measures suggested in this thread are adequate to deal with crappy reporting, what i was suggesting was taking some measures to stop this kind of crap ever getting printed by having better science reporting – but then, Ben would have nothing to write about, so perhpas not…?!!

  47. Mathew said,

    November 29, 2005 at 1:08 pm

    I love the blurb on the University of Westminster’s site (as mentioned by Andrew T above). I have added a couple of notes:

    “HomÅ“opathy is a system of medicine…

    [Is it bollocks. A system of persuasion of the ignorant and vulnerable more like]

    …that aims to assist the natural tendency of the body to cure itself.

    [= does what the immune system does anyway]

    It works on the principle that whatever the symptoms a substance causes in a healthy person, minute doses of the same substance will cure those symptoms when they occur as illness.

    [Which makes perfect sense of course. Lead poisoning? Take some more lead. Mesothelioma? Inhale some more asbestos, etc. Nobel Prize stuff]

    HomÅ“opathic remedies are therefore prepared as minute doses. They cannot cause side effects and are not addictive.”

    Woah, hold on. Wait a moment. They cannot cause side effects? Not ‘they do not’ but they cannot. And yet they can cure illnesses. No side effects, no addiction, but still effective upon the cause of the illness. That, ladies and gentlemen, is what I would call a miracle.

    Which is quite apt, since homeopathy is a matter of faith and not science.

    I agree with Stephen, it is extraordinary that homeopathy is provided on the NHS.

  48. JK said,

    November 29, 2005 at 2:07 pm

    I have a plan for dramatically increasing the effectiveness of homoeopathic facilities in the NHS. It’s a dilution problem. First sack 90% of the staff and claw back 90% of the buildings. Repeat this several times until you have less than one person in less than one room. Remarkably, and yes I know this is counterintuitive, that sub-quack will cure more colds and bruises than you would ever have thought possible.

    (sorry if this very old joke has already been trotted out, I’m a bit of a stim-resp engine).

  49. Martin B said,

    November 29, 2005 at 3:19 pm

    There does appear to be one aspect of this homeopathy study which people haven’t picked up on, or only brushed up against it.

    It’s not that 70% of the people who went to homeopathy hospitals reported that their health had improved, it’s that 70% of the people who reported, reported that their health had improved. Therefore, 30% of those gullible (or desparate) people who thought that they’d try homeopathy, and still reported back, found that their health hadn’t improved. This study actually is a shock because it finds that homeopathy doesn’t work for nearly a third of people who are predisposed to expect it to work.

    Also, the reporting above doesn’t make it clear that everyone reported. I’d have thought that it was in the interest of the homeopathy peddlars to report if this was the case. Therefore, some people didn’t return to report on their condition. I can only think of three reasons why people didn’t return:
    1) They were miraculously cured and so didn’t need to see another doctor/quack.
    2) They weren’t miraculously cured and returned to their GP, who advised them against going back to the homeopathist (is that the right word?)
    3) They died.

    I’d say it’s a pretty big reporting bias if you only report on people with chronic conditions who survived to tell you that they feel better.

  50. BSM said,

    November 29, 2005 at 10:44 pm

    “They cannot cause side effects? Not ‘they do not’ but they cannot. And yet they can cure illnesses. No side effects, no addiction, but still effective upon the cause of the illness. That, ladies and gentlemen, is what I would call a miracle. ”

    Well, even homeopaths pretend to have side effects, it’s just that they call them “aggravations”. As ever, we know homeopathy better than its feckwit practitioners.

  51. tom p said,

    November 30, 2005 at 4:25 pm

    I’ve just made a request to the department of health under the freedom of information act 2000 to find out the annual budgets and annual number of consultations at each of the 5 homeopathic hospitals in the UK.
    I dunno about anyone else, but I’m rather intrigued to know how much this nonsense is costing us each year.

    I’ll keep you all informed whenever I hear anything, so we can make an accurate comparisson with proper medical services

  52. Mark said,

    December 5, 2005 at 8:25 am

    There is an exchange on homeopathy in the most recent issue of science and public affairs from the BA (British Associaiton for the Advancement of Science). Most of it is well worn territory but I am wondering if there something of interest in Micheal Hyland’s contribution.

    www.the-ba.net/the-ba/CurrentIssues/ReportsandPublications/ScienceAndPublicAffairs/SPADec05/_Homeopathy.htm

    Cheers

  53. Phil Wills said,

    December 6, 2005 at 1:26 pm

    I think that’s a reasonably well argued case generally Mark, even if I don’t necessarily agree with it’s conclusions, but why do psychologists resort to the likes of:

    “The other is there is some other ‘unconventional’ mechanism, such as quantum entanglement or electromagnetic radiation”

    If they don’t know about Physics why make themselves look daft to those who do by inserting random phrases to pretend otherwise? Electromagnetic radiation is largely well understood and any effects would be relatively easy to detect. Quantum entanglement is a fascinatingly strange phenenomena, but it’s only important in very specific circumstances and isn’t the new magical reason for everything it seems to be touted as these days.

  54. Rob Buckley said,

    December 9, 2005 at 8:39 am

    Can anyone spot the difference between Ben’s article above and this article over on Spiked? www.spiked-online.com/articles/0000000CAEB6.htm

  55. Delster said,

    December 9, 2005 at 11:53 am

    I’m thinking of setting up as a BScHom myself. i have all the qualifications after all.

    I know enough science to throw in the buzzwords, enough medical training to spout fancy names for conditions, easy access to tap water and small brown bottles and a very good line in Bull S***, which should also answer Andrew T’s question as to what the B stands for in BScHom

  56. Dean Morrison said,

    December 11, 2005 at 9:00 pm

    This type of survey is not a new invention of homeopaths – one from 1999 in Tunbridge Wells generated surprisingly similar results. Perhaps we should use this work as a ‘baseline’ of what people are likely to say when the are treated with sugar tablets:
    tinyurl.com/dnrkq

  57. Daniel R said,

    December 12, 2005 at 2:06 pm

    Hey guys

    I was wondering about where you mention that patients are more likely to give the response that the doctor wants, or that the patient thinks the doctor wants. This is something that makes sense, of course – but is there a literature reference to show this effect?

    I’m currently doing a literature review of homepathy (it’s not faring very well, in case you couldn’t predict that), and while me saying “patients gave the answers that were wanted” is all well and good, if I can’t back it up, it just sounds like I’m being a bitch.

    I’ve been trying to find something on this since Friday, and the closest sorts of things I can find are Milgram’s experiments, or maybe the Asch conformity experiments. But they’re not really saying what I want to.

    I don’t know any psychology students, and I have no idea where to look, or what to look for. Any help in finding a source/reference would be greatly appreciated.

    Oh, my project supervisor told me about this site, so he’ll probably read this. Hope I’m not breaking any rules >_>

  58. Ben Goldacre said,

    December 12, 2005 at 2:45 pm

    I’m sure you’ll find something on this in How to Read a Paper: The Basics of Evidence-Based Medicine by Trisha Greenhalgh. It’s a form of “information bias” I guess, as they weren’t blinded, and didn’t use standardised or more objective measurement tools like questionnaires, although I’m not sure about a term like “blinding” in a trial where there weren’t even two arms…

  59. Matt said,

    December 12, 2005 at 3:26 pm

    On the topic of side effects of homeopathy, in reporting Bristol Homeopathy study, Elizabeth Grice in the Telegraph made the claim that “Homoeopathic physicians have been prescribing it [homeopathic arsenic] for more than two centuries in Europe and America… apparently without a single case of poisoning”.

    Hmm… As I’ve already written to Ben, the idea that homeopathy doesn’t cause side effects isn’t true. Forget “aggravations”, these are bone fide “adverse effects”. Dipankar Chakraborti described three Indian patients in the Journal of Toxicology – Clinical Toxicology (now called Clinical Toxicology) who suffered arsenic poisoning after ingesting homeopathic arsenic [1]. Another study in the same journal nearly twenty years ago by Harry Kerr and Leon Saryan of the Wisconsin ACL Industrial Toxicology Laboratory noted that two of six homeopathic preparations tested contained “notable quantities” of arsenic [2]. Not good, since prolonged exposure to arsenic has been linked with cancer, diabetes, skin thickening, liver disease, digestive problems and nervous system disorders.

    The safety of other homeopathic remedies is also questionable: David Anick at the Harvard Medical School has found that commercially prepared homeopathic remedies can contain traces of ethanol, acetate, formate, methanol and acetone [3]; Spanish, German and Mexican researchers have all reported mercury poisoning due to homeopathic remedies, including in one six-month old baby [4-8]. I could find several other cases of severe side effects: homeopathic mandrake caused anaphylactic shock in one patient [9]; homeopathic slimming droplets (LOCO X112). were found to contain thyroid extract and diethylpropione, an amphetamine-like noradrenergic anorectic agent banned in Belgium [10]; in another case (abstract and paper sadly unavailable) a patient seems to have died from acute pancreatitis due to homeopathy [11]. Another remedy caused contact dermatitis [12], and the same remedy was in a course of homeopathic medicine given to a child who developed autoimmune blistering (whether it was caused by the homeopathy or by the failure to treat properly isn’t clear) [13].

    What was that about homeopathy having no side effects?

    Matt

    References:
    1. Chakraborti D, Mukherjee SC, Saha KC, Chowdhury UK, Rahman MM, Sengupta MK: Arsenic toxicity from homeopathic treatment. J Toxicol Clin Toxicol 2003;41(7):963-7.
    2. Kerr HD, Saryan LA: Arsenic content of homeopathic medicines. J Toxicol Clin Toxicol 1986;24(5):451-9.
    3. Anick DJ: High sensitivity 1H-NMR spectroscopy of homeopathic remedies made in water. BMC Complement Altern Med 2004, 4:15.
    4. Montoya-Cabrera MA, Rubio-Rodriguez S, Velazquez-Gonzalez E, Avila Montoya S: [Mercury poisoning caused by a homeopathic drug]. Gac Med Mex 1991, 127(3):267-70. Article in Spanish.
    5. Audicana M, Bernedo N, Gonzalez I, Munoz D, Fernandez E, Gastaminza G: An unusual case of baboon syndrome due to mercury present in a homeopathic medicine. Contact Dermatitis 2001, 45(3):185.
    6. Wiesmuller GA, Weishoff-Houben M, Brolsch O, Dott W, Schulze-Robbecke R: Environmental agents as cause of health disorders in children presented at an outpatient unit of environmental medicine. Int J Hyg Environ Health 2002, 205(5):329-35
    7. Olujohungbe A, Fields PA, Sandford AF, Hoffbrand AV: Heavy metal intoxication from homeopathic and herbal remedies. Postgrad Med J 1994 Oct;70(828):764.
    8. Spoerke DG: Toxicity of homeopathic products. Vet Hum Toxicol 1989, 31(3):259-60.
    9. Helbling A, Brander KA, Pichler WJ, Muller UB: Anaphylactic shock after subcutaneous injection of mandragora D3, a homeopathic drug. J Allergy Clin Immunol. 2000 Nov;106(5):989-90.
    10. Mortelmans LJ, Biesemans L, Van Rossom P: Homeopathic products, not as innocent and safe as they seem? A case report. Eur J Emerg Med. 2004 Aug;11(4):242
    11. Barquero Romero J, Redondo Lopez JM, Galeano Diaz F, Perez Miranda M: [Fatal acute pancreatitis in a patient who received an homeopathic treatment]. Med Clin (Barc). 2004 Mar 6;122(8):318-9. Spanish.
    12. Cardinali C, Francalanci S, Giomi B, Caproni M, Sertoli A, Fabbri P: Contact dermatitis from Rhus toxicodendron in a homeopathic remedy. J Am Acad Dermatol. 2004 Jan;50(1):150-1.
    13. Kuenzli S, Grimaitre M, Krischer J, Saurat JH, Calza AM, Borradori L: Childhood bullous pemphigoid: report of a case with life-threatening course during homeopathy treatment. Pediatr Dermatol 2004, 21(2):160-3.

  60. online graduate courses said,

    February 14, 2006 at 8:04 pm

    Great blog. Found your blog while searching for more information at yahoo about online graduate courses . Your blog has quite a lot of interesting thoughts. Keep up the good work.

  61. kallo said,

    October 23, 2006 at 1:09 pm

    @ Phil Wills

    ““The other is there is some other ‘unconventional’ mechanism, such as quantum entanglement or electromagnetic radiation”

    If they don’t know about Physics why make themselves look daft to those who do by inserting random phrases to pretend otherwise?”

    Michael Hyland is one of my lecturers, let me assure you he was only joking about that. He’s a funny guy. Insane… but funny. His current work on spirituality and its effect on health is interesting stuff.

  62. It’s Evidence Jim, But Not As We Know It! « A canna’ change the laws of physics said,

    November 8, 2007 at 8:52 pm

    [...] This debacle has been covered in detail here. [...]

  63. Blog Atopowy » Blog Archive » Koniec homeopatii? said,

    November 22, 2007 at 10:26 pm

    [...] to jakie brednie. Jakoś, zupełnie niewytłumaczalnie, ankieta satysfakcji konsumenta z kliniki homeopatycznej jest promowana w mediach, tak jak gdyby [...]

  64. Publish and be damned « A canna’ change the laws of physics said,

    August 16, 2008 at 5:00 pm

    [...] is that for communities of True Believers, the approach is very different.  Testimonials and customer satisfaction surveys are hailed as hard evidence.  Really silly books are cited.  Pseudo-journals controlled by cabals [...]

  65. wayscj said,

    November 21, 2009 at 7:30 am

    ed hardy ed hardy
    ed hardy clothing ed hardy clothing
    ed hardy shop ed hardy shop
    christian audigier christian audigier
    ed hardy cheap ed hardy cheap
    ed hardy outlet ed hardy outlet
    ed hardy sale ed hardy sale
    ed hardy store ed hardy store
    ed hardy mens ed hardy mens
    ed hardy womens ed hardy womens
    ed hardy kids ed hardy kids ed hardy kids

  66. iphone revolution said,

    December 30, 2009 at 8:30 am

    iphone wireless

    iphone

    Apple iphone

  67. It’s Evidence Jim, But Not As We Know It! | CAMcheck said,

    June 25, 2010 at 8:56 pm

    [...] This debacle has been covered in detail here. [...]