Is it okay to ignore results from people you don’t trust?

March 6th, 2010 by Ben Goldacre in bad science, media, media research, tobacco | 39 Comments »

Ben goldacre, The Guardian, Saturday 6 March 2010

If the media were actuarial about drawing our attention to the causes of avoidable death, your newspapers would be filled with diarrhoea, Aids, and cigarettes every day. In reality we know this is an absurd idea. For those interested in the scale of our fascination with rarity, one piece of research looked at a 3 month period in 2002 and found that 8,571 people had to die from smoking to generate one story on the subject from the BBC, while there were 3 stories for every death from vCJD.

So you’ve probably heard that smoking might prevent Alzheimers. It comes up in the papers, sometimes to say it’s a true finding, sometimes to say it’s been refuted. Maybe you think it’s a mixed bag, that the research is contradictory, that “experts are divided”. Perhaps you smoke, and joke about how it’ll stop you losing your marbles, at least.

This month, Janine Cataldo and colleagues publish a systematic review on the subject, but with a very interesting twist. First they found all the papers ever published on smoking and alzheimers, using an explicit search strategy which they describe properly in the paper – because they’re scientists, not homeopaths – to make sure that they found all of the evidence, rather than just the studies they already knew about, or the ones which flattered their preconceptions.

They found 43 in total, and overall, smoking significantly increases your risk of Alzheimers. But they went further. 11 of the studies were written by people with affiliations to the tobacco industry. This wasn’t always declared, so to double check, the researchers searched on the University of California’s Legacy Tobacco Documents Library, a vast collection of scanned material which has been gathered over decades of legal action.

If you ever want to spend a chilling afternoon living in the head of an industry whose product has been proven to kill a third of their customers, this is the place for you. “The importance of younger adults” uses financial modelling to explain the importance of recruiting teenage smokers to replace the dying older ones before it’s too late, and explains that “repeated government studies have shown less than one third of smokers start after age 18 [and] only 5% of smokers start after age 24.” “Youth cigarette – new concepts” from Marketing Innovations Inc takes these ideas further, into cola and apple flavour cigarettes, because “apples connote goodness and freshness”.

How much did it matter if the researchers had worked for the tobacco companies? A lot: the risks of Alzheimers associated with smoking reported by these papers were on average about a third lower than those conducted by other researchers, and they produced many papers showing cigarettes were actively protective. If you exclude these 11 papers, and look only at the 32 remaining, your chances of getting alzheimers are vastly higher: for the gamblers out there, comparing a smoker against a non-smoker, the odds of getting Alzheimers are higher by 1.72 to 1.

So does that mean we can comfortably ignore all research that comes from people who disgust us? In the 1930s, identifying toxic threats in the environment became an important feature of the Nazi project to build a master race through ‘racial hygiene’. Two researchers, Schairer and Schöniger, were working on biological theories of degenerate behaviour under Professor Karl Astel, a scientist who helped organise the vile “euthanasia” operation that murdered 200,000 mentally and physically disabled people.

In 1943 his two researchers published a well conducted case-control study demonstrating a relationship between smoking and lung cancer almost a decade before any researchers elsewhere. Their paper wasn’t mentioned in the classic Doll and Bradford Hill paper of 1950, and if you check in the Science Citation Index, it was referred to only four times in the 1960s, once in the 1970s, and then not again until 1988, despite providing a valuable early warning on a killer that would cause 100 million early deaths in the 20th century. It’s not obvious what you do with evidence from untrustworthy sources, but it’s always worth appraising its untrustworthiness with the best tools available.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

39 Responses



  1. ellieban said,

    March 6, 2010 at 3:12 am

    We should NEVER ignore the research of people who offend us or whose research is suspect. Look at the mess we are in with global warming: all because those who had the power to do it didn’t nip the denialism in the bud. We have to fight this stuff tooth and nail where ever it appears, if we don’t, it gets a foot hold and takes over.

  2. Phaedron said,

    March 6, 2010 at 4:11 am

    According to the latest research, however, smoking still makes me cool. I should really get to quitting…

  3. wolfkeeper said,

    March 6, 2010 at 5:32 am

    @Phaedron Yes, and half the time it cools you all the way to room temperature; great huh? Each cigarette shortens your life by an average of 11 minutes.

  4. Thimble said,

    March 6, 2010 at 7:01 am

    Wouldn’t actuarial UK newspapers also prominently cover pedestrians being killed by cars?

    Also, just like it is important to distinguish between bullies, victims, and bully-victims, I think that it is important to distinguish between the quacks who are just ignorant from the ones who are stopping quotes mid sentence and all that, which really can only be fraud. The bha has recently been caught out at this has it not?

  5. theonlyrick said,

    March 6, 2010 at 9:27 am

    @Thimble I’m not trolling (well, maybe I am), but what’s your point?

  6. theonlyrick said,

    March 6, 2010 at 9:41 am

    IIRC, Al Gore made it very clear that the PR companies that lobbied governments for non-intervention and pushed cigarette denialism for years (and years and years) are the same PR companies that have done the same thing with oil and climate change.

    It’s funny/tragic how large industry, and their very rich bedfellows (rightwing nutjobs and pro-corporate figures in the public eye) claim that climate research is just a gravy train – an excuse for those fat cat researchers to score another holiday seal clubbing in Northern Canada.

  7. stvb2170 said,

    March 6, 2010 at 10:07 am

    Thimble, I believe the answer to your question is ‘no’. A quick look at the national statistics (www.statistics.gov.uk/cci/nugget.asp?id=1208) shows 646 pedestrians killed by cars in 2007.

    According to the Department of Health (www.dh.gov.uk/en/Publicationsandstatistics/Pressreleases/DH_4047360) smoking kills 120,000 people in the UK each year.

    Meaning that we’d need 185 stories about smoking deaths before we covered a single pedestrian death

    Ben has made a mistake with his facts here though (link broken, but if you look elsewhere (www.kingsfund.org.uk/applications/site_search/?term=bbc+smoking&searchreferer_id=20250&submit.x=0&submit.y=0 13 May 05 story) Ben is misreporting the timeframe. BBC investigation was over the year to September 2001, newspapers were investigated in the 3 month period he quotes).

  8. Abahachi said,

    March 6, 2010 at 10:08 am

    At the risk of being shot down as a typical wishy-washy humanities type, this seems to be blurring the distinctions between different kinds and effects of bias resulting from dubious a priori assumptions. Schairer and Schoeniger had extremely dodgy reasons for being interested in environmental toxicity, but there’s no obvious reason why those motives should have prejudiced them in favour of particular results (whereas the moment they start discussing the different responses of Aryans and Jews to tobacco smoke we should stop listening to them). Researchers with affiliations to the tobacco industry, on the other hand, have obvious reasons for preferring certain results over others; whether or not they deliberately set out to deceive, we should take their results with a few shovels of salt.

  9. osipacmeist said,

    March 6, 2010 at 10:31 am

    “This is right even if Hitler says it is” is a difficult argument to win… It’s ugly but true: sometimes you have to think of the effective rhetoric that will be used against good science. Unfortunately it leaves you feeling very dirty…

    Still you should always distrust a good rhetorical point, no matter whose side it’s made on. Ain’t life complicated?

  10. JH said,

    March 6, 2010 at 11:26 am

    Putting Alzheimers to one side for a moment, we shouldn’t forget that smoking dramatically increases risk of vascular dementia, the second commonest form of dementia.

  11. geodoc said,

    March 6, 2010 at 1:19 pm

    “If the media were actuarial about drawing our attention to the causes of avoidable death, your newspapers would be filled with diarrhoea, Aids, and cigarettes every day.”

    Indeed.

    The ever-excellent Hans Rosling came up with the ‘news/death ratio’ as a measure for quantifying this effect, at the height of the swine flu epidemic:
    www.gapminder.org/videos/swine-flu-alert-news-death-ratio-tuberculosis/

    I did some more back-of-the-envelope ratios for other news/non-news events, just for the hell of it:
    geodoctor.wordpress.com/2010/02/14/newsdeath-ratios/

  12. rhurras said,

    March 6, 2010 at 2:14 pm

    @woolfkeeper
    where did you get the 11 minute number from?

    @geodoc
    I’d rather use news/death ratio instead of death/news ratio. That way you get a proportional measure of the news worthiness of a particular death. Seems to sell the point better.

  13. geodoc said,

    March 6, 2010 at 3:46 pm

    rhurras, yup that’s what the news/death ratio does (news stories per death). Just to confuse things I did mislabel the relevant column in the table in my post though- will correct!

  14. richardelguru said,

    March 6, 2010 at 8:19 pm

    “newspapers would be filled with diarrhoea, Aids, and cigarettes every day.”

    Seems to me that the media ARE full of diarrhoea most days.

  15. Mark P said,

    March 7, 2010 at 2:50 am

    We should NEVER ignore the research of people who offend us or whose research is suspect.

    Fine.

    Look at the mess we are in with global warming: all because those who had the power to do it didn’t nip the denialism in the bud.

    Too funny. I thought we were to NEVER ignore the research of people who offend us or whose research is suspect.

    It’s not funny though how this one particular issue makes people otherwise allegedly in favour of scepticism and open discussion close their minds.

    (BTW, the world is warming. The real issues are 1) are humans responsible to any great extent, and 2) can we stop it. I have to suspect you are deliberately going at straw men by conflating the issues.)

  16. DevonDozer said,

    March 7, 2010 at 1:50 pm

    Mark P is on the money at 2:50 a.m.

    You probably saw a link from the NHSblogdoctor to this depts.washington.edu/gim/calendar/hmcjc_abstracts/JCJul04Article1.pdf .

    If ‘the law’ is against Evidence Based Medicine, then this is probably an even greater threat than the MSM. Do you have any views on this chap’s experience? Slander laws pale to nothing next to this kind of stuff.

    The notion that “Lawyers trump science” makes me very uncomfortable – and we aren’t talking about Sharia or other ‘supersitition oriented’ countries. I’m sure a lot of Brusselcrats will have seen its potential, too.

  17. DrJG said,

    March 7, 2010 at 5:32 pm

    @ Mark P – I disagree with your assessment of which are the real issues. How about these two:

    1. What level of evidence would it take to convince the majority of current sceptics that global warming is a) real and b) significantly worsened by human activity?

    2. How much more difficult would it be to take effective action if, as many sceptics want, we take no action until that level of proof can be gathered?

    Do I regard AGW as conclusively proven? No. Do I think that we can afford to wait until it IS conclusively proven before considering action? No again.

  18. pictonic said,

    March 7, 2010 at 6:17 pm

    I agree with DrJG. Global warming is a (very) high stake game.
    Waiting (possibly a few scores of years) for a high standard of ‘proof’ that the world will be badly affected before considering action to anticipate and compensate is a bit too casual for me. Let’s plan for a future – as you do when you buy insurance for you and your family.

  19. DougieJ said,

    March 7, 2010 at 9:34 pm

    @pictonic – yes, but if you own a Hyundai would you take out Bentley-level insurance ‘just to be on the safe side’?

    @MarkP – I actually think you’re being too generous. The BBC’s Roger Harrabin (a fully paid-up member of the church of ManBearPig, but to be fair he’s at least trying to build bridges to some extent) recently asked a certain person some pertinent questions:

    Q. Do you agree that according to the global temperature record used by the IPCC, the rates of global warming from 1860-1880, 1910-1940 and 1975-1998 were identical?

    A. …I have also included the trend over the period 1975 to 2009, which has a very similar trend to the period 1975-1998. So, in answer to the question, the warming rates for all 4 periods are similar and not statistically significantly different from each other.

    Q. Do you agree that from 1995 to the present there has been no statistically-significant global warming

    A. Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level. Achieving statistical significance in scientific terms is much more likely for longer periods, and much less likely for shorter periods.

    If anyone can read into the above that the end is nigh, well…

    You’re right to highlight the strawman arguments being consistently aired by the alarmists though. The real issue now at last being vigorously debated is not ‘global warming’ but what is the extent of catastrophic, man-made global warming.

    BTW – the person Harrabin was interviewing was a Professor Phil Jones. Anyone heard of him at all?

  20. fluffy_mike said,

    March 7, 2010 at 10:48 pm

    @stvb2170

    Perhaps @Thimble was referring to the prevalence of ‘killer cyclists on pavement’ stories that seem to populate the likes of the Daily Mail and GLR radio every week.

    Truth is you’re as likely to be killed by a golf ball as you are by a cyclist on a pavement (about 2 or 3 per decade), whereas motor traffic kills 3000 people per year, including a large number of pedestrians on footpaths (about 40 each year).

    Yes, these figures are many orders of magnitude removed from smoking fatalities, but that doesn’t mean the misreporting is in any way acceptable.

  21. Uther said,

    March 8, 2010 at 12:43 am

    Two quick points:

    i) I heard that many Nazi medical experiments were ignored, not because they were done by unethical people or for unethical reasons, but because the experiments themselves used unethical methodology. I find this an interesting area. If someone murders 40 people, and in the process finds out some useful information, should you use that information? One strong argument says ‘no’ because even if you punish the psycho who murdered 40 people, others might come along who decide that they’ll do unethical experiments and then take the punishment ‘for the good of humanity’. If you refuse to use the results, then you remove that justification.
    This issue is, I believe, not relevant to the above smoking example though. I also like Abahachi’s point.

    ii) Was looking at journals today and got some interesting Google Ads. Apparently looking at the contents page of a journal where they list ‘Numer 1’, ‘Number 2’, etc. means you’re interested in numerology.

    picasaweb.google.com/lh/photo/OxSJPoIFcwU2vVyNZrj1dw?feat=directlink

    Be well all.

  22. ellieban said,

    March 8, 2010 at 1:35 am

    Mark P

    Too funny. I thought we were to NEVER ignore the research of people who offend us or whose research is suspect.

    Errr, yes… By instigating an open and honest debate that shows these charlatans for what they are before they get a toe hold in the press.

    I don’t see your point.

  23. paddyfool said,

    March 8, 2010 at 9:54 am

    “lower risk estimates for studies done by authors affiliated with the tobacco industry (by −0.37 ± 0.13, P=0.008), ”

    Whether or not results from those with a strong source of bias should be ignored completely, I applaud the ingenuity of considering such a funding source as an effect modifier – perhaps this could also be done for other contentious fields?

  24. ChrisPartridge said,

    March 8, 2010 at 12:36 pm

    I know a guy who worked on the first ejector seats just after the war. To get it right, they needed to know exactly how much force they could use on a pilot to get them out of the cockpit without killing them. The faster they could propel them, the more pilots’ lives they would save.
    Research existed, however – the Nazis had established exactly the data they needed by dropping Jews down mineshafts.
    There was fierce debate as to whether this data could ethically be used. It was, of course, partly because it was there and would save our pilots’ lives. And it was the height of the Cold War.

  25. MedsVsTherapy said,

    March 8, 2010 at 3:03 pm

    News/death ratio? Abortion is by far and away the leader. 800 each workday in UK. The occasional story is sure to ignore the fact that these are actual lives, but focus on a vague notion of “tragedy” undefined. Without legal abortion, the death rate would go down drastically, close to 3 orders of magnitude, even considering the lives lost to illegal abortion.

  26. pv said,

    March 8, 2010 at 7:02 pm

    MedsVsTherapy…

    so what’s your agenda then?

  27. DrJG said,

    March 8, 2010 at 9:39 pm

    @ DevonDozer

    A truly horrible story in your link. And what is worse is that it is easy to imagine the same lawyer going for his next medical victim for failing to inform their patient that the PSA test might lead down a path of prostatic biopsies etc with no practical benefits in the end.

    I’m with Will Shakespeare when it comes to some of our legal friends.

  28. Delster said,

    March 8, 2010 at 10:24 pm

    Re the sentance “We should NEVER ignore the research of people who offend us or whose research is suspect” in the very first comment.

    Only part of this is true.

    If somebody is offensive to us, for whatever reason, this does not affect the data that they have produced in any way. So yes the first part of the statment is true in so far as it goes. Provided of course, they don’t fall foul of the 2nd part of the sentance.

    However, the second part; “Whose research is suspect” is, in itself, a warning that their results should, at the very least, be ripped apart down to the raw data and methodoloy used and re-examined from the ground up.

    Dodgy research is dodgy research no matter who carries it out and of course the same applies to good research.

  29. Mark P said,

    March 9, 2010 at 9:31 am

    By instigating an open and honest debate that shows these charlatans for what they are before they get a toe hold in the press.

    Why are “charlatans” of other sorts of other sorts not subjected to this though? We let Marxists publish total rot without anything like this venom. Have done for decades. Marxism has killed millions BTW, so it isn’t “harmless”.

    Actually, why are people who do actual research and publish finding contrary to the approved view called “charlatans”?

    What you are really saying is — I’ve made up MY mind, the debate is over.

    What level of evidence would it take to convince the majority of current sceptics that global warming is a) real and b) significantly worsened by human activity?

    It depends on the sceptic.

    The hard-core ones who are opposed to any action on political grounds will never be persuaded. (Just as some eco-greenies will never budge on GM foods, no matter how much science is thrown at them.)

    Most people who call themselves sceptics haven’t done any research at all. (Just as some AGW proponents haven’t either, in the sense of actually looking for evidence of the opposite, which is the heart of real scepticism. And science for that matter.)

    Others, like me, are far more concerned about the deliberate panic being instigated about it, for political reasons, than the problem warrants. All these “it’s our last chance” doom-sayers. I watched the Club of Rome try that lark over 30 years ago.

    I’m hesitant on the warming, and persuadable. But the need to immediately choke all economic growth as the only solution is too stupid for words.

  30. theengineer said,

    March 9, 2010 at 12:29 pm

    @markp. Good news then, we don’t need to immediately choke all economic growth to ‘solve’ global warming (although there are probably a fair numer of ‘eco-geenies’ who would disagree with me). Swapping across to a low-carbon economy should be economically stimulating – imagine the massive investment required throughout our society to acheive the changes needed – this is all economic activity. What is required is to immediately choke all economic growth in those industries that are fundamentally carbon driven, but that in definitely NOT the same thing.

    I’d also argue, since your a sceptic leaning towards ‘not convinced of global warming’ but concerned about economic activity, that we are faced with a choice regardless – oil/gas/coal are finite resources and will, pretty much guaranteed, become more expensive over time. The effort involved in finding and exploiting them (and corresponding research and development) would, potentially, contribute to economic growth as much as the alternative and should we stay on the ‘carbon’ path would become more and more significant proportion of our economy – but the exploited resource would continue to be finite and reducing – so the choice is this swap a chunk of our economy over to something that will continue to work in the long run OR don’t and allow a similar chunk to develop with a guarantee that it will fail at some point.

    The words ‘sustainable development’ might come with alot of emotional baggage, but if you really think about it this is the root of my argument.

  31. pray111 said,

    March 9, 2010 at 2:45 pm

    The words ’sustainable development’ might come with alot of emotional baggage, but if you really think about it this is the root of my argument.

  32. jwm said,

    March 10, 2010 at 8:18 pm

    Mark P:

    “Why are “charlatans” of other sorts of other sorts not subjected to this though? We let Marxists publish total rot without anything like this venom. Have done for decades. Marxism has killed millions BTW, so it isn’t “harmless””

    Choosing a social science philosophy as a comparison to biological science fact is a rather weak point, and a poor platform for a cheap shot against Marxism.

    Killed millions? Are you confusing it with stalinism or maoism per chance? And for fair comparison should we be comparing deaths with those resulting from British Imperialism where we managed to wipe out more people than the Nazis could even dream off, or more recently, American Imperialism?

  33. timboson said,

    March 11, 2010 at 12:13 pm

    How to hi-jack a comments column? Answer throw in a comment about global warming.

  34. MedsVsTherapy said,

    March 11, 2010 at 3:22 pm

    jwm has a great point. Centralized, unchecked power leads to bad outcomes (Stalinism, Maoism, American Imperialiasm, etc.). I believe the solution would be to create a centralized, unchecked world power governing economic production through global tax and economic regulation. That is sure to solve the problem of pollution, while avoiding the dangers of centralized, unchecked power.

  35. What said,

    March 11, 2010 at 8:00 pm

    I wonder how firm the diagnoses are. There are many kinds of dementia, and differentiating Alzheimer’s from vascular dementia before death has been very difficult.

  36. chrisdeane said,

    March 12, 2010 at 6:09 am

    I worry about comparing news stories just with numbers of deaths. Some things may on average remove many more years from people’s lives on average and could therefore be considered much more important for the same number of deaths. For example traffic accidents are likely to be killing a higher proportion of people in their teens and 20s and therefore removing many more years per death than smoking which kills people in their 50s to 80s. Although I agree with the general sentiment and the effect probably remains after taking my concerns into consideration, this site is supposed to be about doing things properly and scientifically.

  37. jazz_the_cat said,

    March 12, 2010 at 1:55 pm

    You can’t ignore studies that you don’t trust in the scientific literature, but you can discount them. It’s not uncommon to see a paper where a contradictory or untrustworthy paper (result) is referenced, but then followed with an explanation of why the result is untrustworthy or irrelevant to the work at hand.

  38. milli said,

    March 19, 2010 at 3:29 am

    fascinating piece and goes to the heart of scientific research.

  39. maninalift said,

    May 5, 2010 at 11:04 am

    the relevant question is of course “do the researchers prejudices effect this research”.

    the problem is how do you control for that. Excluding research based on your judgement of the character of the researches is a dark road which would lead to the total implosion of the common forum for sensible scientific debate. Just look at how the Creation “scientists” and even some of the industry-research-institute aligned scientists already refer to the top universities as “(extreme) left-wing”.

    The only protection is to continue to refine the processes of science and scientific publication to be robust enough to make the buffeting of vested interests and preconception almost impotent. And I think that almost is the most we can hope for – or rather the best, which in this case is I suppose the least 🙂