Venal, misleading, pathetic, dangerous, stupid, and busted

March 21st, 2009 by Ben Goldacre in bad science, evidence, media, screening | 69 Comments »

Ben Goldacre
The Guardian
Saturday 21 March 2009

Science is not difficult to explain. Today we will see how British journalists go out of their way to cherry pick which evidence they cover, and then explain the risks and benefits in what has been shown to be the single most unhelpful way possible.

“Screening all older men for prostate cancer ‘could reduce deaths by a fifth’” said the Mail. “Prostate cancer hope” said the Mirror. Calls for new policies on NHS cancer tests said the Independent. Prostate cancer screening could cut deaths by 20% said the Guardian. Better cancer screening is every man’s right was the editorial in the Scotsman, where they wound themselves into a froth of indignation.

But was this just British journalists finding something to complain about? Because all around the world, people were saying something completely different, on the same day, about the very same academic publication: Prostate Cancer Screening May Not Reduce Deaths said the Washington Post. Studies cast doubt on leading prostate cancer test said USA Today. PSA testing may not save your life, after all said Scientific American. Prostate cancer blood test does little to decrease death rate said the Sydney Morning Herald. And so on.

Why would the American and the Australian journalists say something completely different to the British ones, about the very same evidence?

Firstly, our journalists were simply confused. Not a single newspaper managed to clearly explain the risks and benefits of screening in the trial they were writing about. It’s very simple: the study took over 160,000 men between the ages of 55 and 69 and randomly assigned them either to get PSA screening, or to be left alone. The differences were marginal. Yes, there were 20% fewer deaths in the screening group. What does that mean in terms of real people, in real numbers you can understand, not percentages?

1410 men would need to be screened to prevent one death. For each death prevented, 48 people would need to be treated: and prostate cancer treatment has a high risk of very serious side effects like impotence and incontinence.

These figures are not hard to find: they are in the summary of the research paper.

For complex risk decisions like screening, it has been shown in three separate studies that patients, doctors, and NHS purchasing panels make more rational decisions about treatments and screening programmes when they are given the figures as real numbers, as I did above, instead of percentages. I’m not saying that PSA screening is either good or bad: I am saying that people deserve the figures in the clearest form possible so they can make their own mind up.

Journalists meanwhile – professional communicators no less – all used impenetrable percentages, called “relative risk differences”, or worse. The piece by the Guardian’s health correspondent quoted several entirely meaningless numbers. “214 prostate cancer deaths had occurred in the screening group and 326 in the “control” group.” Without explaining the other numbers in each group, these digits are hardly more than decorative. It went on: “The rate of overdiagnosis – defined as diagnosis in men who would not have clinical symptoms during their lifetime – was as high as 50% among those who were screened.” I cannot see how this number helps me understand the risks.

image But it gets worse. British journalists also deliberately ignored one whole half of the research, and I’ll confess I’ve slightly lost my sense of humour over this. There were in fact two large studies on PSA testing published in the New England Journal on the 18th of March 2009, not one. They were both published on the same day, in the same journal, they are side by side on the same contents page. British journalists discussed only one of them: the one that said PSA screening does reduce deaths.

The study they ignored was huge too: it took over 75,000 men and randomly assigned them to either a screening programme, or no screening. It found no difference in death rates between the two groups at all, and in case you think it was a close thing, in fact, there was a non-significant trend towards more deaths in the screening group. Not one UK newspaper mentioned this trial.

So newspapers ignore one half of the evidence, and they fail to explain the other half properly. In the past, nobody could catch them, and nobody could compete with them. That has now changed. Anyone can write, and publish online, and appear in Google news alerts: the NHS, medical research charities, individual academics, journals’ press offices.

These people are intelligent, informed, and they can explain things clearly, without worrying about eyecatching hysteria, or space, or hyperbole. Some will be silly, some won’t be. If they ignore half the evidence, they will be busted in the comments, mocked, and sensible visitors will never come back. They can also link directly and transparently to scientific papers, which mainstream media still refuses to do. Journalists insist that we need professionals to mediate and explain science. From today’s story, their self belief seems truly laughable.

References:

Andriole, G.L. et al. Mortality Results from a Randomized Prostate-Cancer Screening Trial. N Engl J Med NEJMoa0810696(2009).doi:10.1056/NEJMoa0810696

Schroder, F.H. et al. Screening and Prostate-Cancer Mortality in a Randomized European Study. N Engl J Med NEJMoa0810084(2009).doi:10.1056/NEJMoa0810084


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

69 Responses



  1. The Biologista said,

    March 22, 2009 at 2:35 pm

    @45 “Ahem, Yes, well, aside from that fact that the journal content will be over the head of most journalists, the NEJM is only (generally) free after six months.”

    If academic institutions can provide full Pubmed access, then surely enormous multinational media conglomerates can pull it off too. As thepoisongarden points out, arts critics are expected to fully understand their medium. Sports journalists must understand the rules of most sports. If a science or health journalist cannot read the primary literature, he or she is in the wrong job.

  2. DrJG said,

    March 22, 2009 at 2:40 pm

    It is not just the 47 other diagnosees per life saved who worry me.
    With a PSA cutoff of 3.0 (necessary as using the common “upper limit of normal” of 4.0 will miss a significant proportion of still-potentially-treatable cancers), there will be many with nothing but benign prostatic hyperplasia. To prove this, they would need further investigation, likely to be an ultrasound-guided rectal-route prostate biopsy, or, to be at all reliable, multiple biopsies, sometimes repeated at a later date. Perhaps we should be asking these journalists how many such biopsies they would be prepared to undergo to save one man’s life, or what odds they would demand that it was going to be Their life which would be saved?

    I would imagine that these false positives would outweigh even the 47 who still die of the disease or, as is most often the case, die With the disease Of something completely different. I note the claimed figures that 35000 men a year are diagnosed with prostatic carcinoma, and “one an hour” (ie between 8 and 9000) die of it annually. Well, I know that very few of the rest are ever actually cured of the disease, which suggests that maybe around 25000 sufferers a year die of something else.

    The Prostate Cancer Charity, from the press releases referenced, clearly believe that GPs are not taking the matter seriously (like many other single-issue health-related organisations seem to think, we should obviously be putting Their agenda higher than the agendas of every other issue, maybe).
    Perhaps if they had had to deal with many years of patients being sent to us after some private medical with a marginally raised PSA, they might understand it better: “Ah, yes, we’ve done your prostate cancer test, and it is a bit high, aren’t you lucky you came our way? Now, be a good fellow and just pop along to your GP and get them to sort it all out for you.”
    Thankfully, that seldom happens now, but there is no doubt that dealing with the fall-out from such thoughtless testing will tend to colour a GP’s view.

    And the last time, admittedly before this latest round of research, that I went to a clinical debate on the subject, the Urologist arguing the case for screening freely admitted during the discussion afterwards that he was firmly anti other than for the sake of the meeting.

  3. Andrew Clegg said,

    March 22, 2009 at 5:22 pm

    The issue of journalists not routinely reading, or having access to, full texts of journal articles is a complete red herring here.

    Not just because the full text of the ignored article is open access, as others have pointed out already, but more importantly because the abstract — which takes about 20 seconds to read — spells out the key point in no uncertain terms.

    “Conclusions: After 7 to 10 years of follow-up, the rate of death from prostate cancer was very low and did not differ significantly between the two study groups.”

    I don’t see how you could spoon-feed lazy journalists a message any more effectively than that!

  4. Dr Jim said,

    March 22, 2009 at 6:00 pm

    #49 / #51: I agree, and absolutely expect that science correspondents should be capable of reading the primary literature. But, as Ben has described in his book:

    “Nobody dumbs down the finance pages. I can barely understand most of the sports section. In the literature pull-out, there are five-page-long essays which I find completely impenetrable, where the more Russian novelists you can rope in the cleverer everybody thinks you are. I do not complain about this: I envy it.”

    The financial and literature review sections of broadsheet newspapers can be well over the head of most readers, the same expectations are not made of science.

    Not sure whether we could ever expect them to rise to the challenge, but I guess we can encourage the media to seek appropriate consultants.

  5. Jeesh42 said,

    March 22, 2009 at 9:41 pm

    Bob Ward,

    Surely journalists should (in theory) be doing something more than regurgitating the press release. Otherwise why don’t newspapers just give us the press releases and fire the lot of you?

    The fact that nurses work under a lot of pressure doesn’t give them free reign to neglect all their patients.

  6. adamcgf said,

    March 23, 2009 at 8:25 am

    I understand that it might slow the news down but surely journalists should operate under the kind of ‘strict liability’ that athletes do – if it’s under your name then it’s your responsibility. I imagine this kind of thing can work itself out in responsible journalists in time – if you’re fed information from a source that proves unreliable you will come to question it at some point – but it should always be the case that if you are ‘publishing’ it it’s up to you to make sure it’ right.

  7. Jessicathejourno said,

    March 23, 2009 at 10:51 am

    If you wanted replies from the actual journalists involved, you should have included their full names in the article so they catch it whilst vanity googling. Because sadly that’s something most of us can be relied on to research thoroughly.

    Anyways, Bob Ward’s arguments piss me off. I write (and far, far more than one article a week) for a B2B concern and if I just copied out a press release without looking into it and adding detail I’d get slapped, and rightfully so. My readers pay about 700 pounds a year for this crap and they deserve value. I don’t think Ward would argue with that.

    So his position annoys me, because while I doubt anyone would argue I shouldn’t do background checking into the releases that cross my desk for my kajillion articles a week because people pay so much for what I write, he seems to be arguing that the public, who pay much less than 700 pounds a year for their news, don’t deserve value – that the system of journalists regurgitating press releases for pop markets is a functional way for things to work because it gets the most information out the fastest.

    The information angle is nonsense, obviously. Press releases aren’t information, they’re ads. That’s their job. That should be obvious to the meanest of intelligences – even to all the non-humanities grads out there.

    But the suggestion that pop audiences don’t deserve the same checking into the material for lifestyle articles, especially the ones about health regimens, is what really gets my goat. They might not be paying however many hundreds of pounds a year but they’re certainly making the business in question however many hundreds of pounds a year. It’s exploitative and gross.

    Oh, and it’s not that damn hard to check a damn press release. It really isn’t. You call the contact people, you look at the company or organization, you have a few quick searches to see if anything else is being done along the same lines somewhere else or by someone else; I mean, the whole damn thing takes 45 minutes at the topmost. Stop being such a big bunch of sucks. Stop playing solitaire or dropping overly-long comments like this one on Bad Science and pick up the damn phone. Damn it.

  8. The Biologista said,

    March 23, 2009 at 3:15 pm

    @54

    Perhaps a distinction needs to be made between “front page” scientific coverage and some sort of “science section” coverage. I wonder if the front page coverage of the economic crisis reflected the accuracy (if not the depth) of the financial pages of most papers? If the second aids the first then perhaps what we really need is hardcore scientific coverage in papers which will then pull up the socks of the flashy front page stuff.

  9. Thegoodhack said,

    March 24, 2009 at 3:25 pm

    It’s a pack thing. After a press conference, the boys and girls from the nationals, popular and unpopular, gather together to agree what the angle is. The last thing they want is to write one story and to see a better angle in another paper, that will invite a bollocking from the news editor. The same thing happens virtually.
    It’s not necessarily venal etc etc, it is the nearest thing to being collegiate in the dog eat dog world of national news reporting.
    The NEJM, uniquely among the big general medical journals, doesn’t issue press releases, but sometimes other people (universities, charities) do, based on early sight of a paper. I suspect (but I don’t know) that this is what happened here, and that the Prostate Cancer Charity issued a release based on one of the papers that supported their activist interventionist agenda.
    Journalists tend to believe charities over the NHS/department of health/government. They just do. It’s genetic. And normally not a bad rule of thumb.
    Except, obviously in cases like this.
    It is possible, indeed likely, that the writers wrote longer more balanced pieces, including stuff from boith studies and they were cut and edited because news editors want simple narratives where there is a villain: in this case the government.
    Important stories on the front page will be honed and honed by various sub-editors and editors to achieve that simple compelling narrative. When it goes right it is a brilliant way of communicating complex information and ideas to millions of people with varying levels of literacy/health literacy.
    And it’s a hard job to do in the few hours available to the average reporter competing for space against celebrinews, murder and the rest of the news agenda. Try it some time. I’m amazed how often they do a good job in reporting complex science in 350 words.
    I prefer newspapers to be energetic vital popular (and sometimes wrong) to dreary, dead, unread, bankrupt and gone (as they increasingly are in the US outside a few metropolitan hotspots).
    Not a popular view among the BAd Science possee, I suspect

  10. Isoelectric said,

    March 24, 2009 at 3:46 pm

    Some of the hubris above sounds like dogs that have treed a raccoon rather than going after the bear they were supposed to be chasing (Ken Kesey: Sometimes a Great Notion). The impact of inaccuracies and inconsistencies of popular press reporting is tiny compared to the systematic abuse of overdiagnosis and ruination of patients due to unnecessary surgery and treatment. Such nonchalant abuse and collateral damage continues because a large number of narcissistic agencies benefit from indolence and disinformation despite years of evidence. A day of reckoning similar to that being experienced in another field of human endeavour may be nigh. But I doubt it.

  11. Jessicathejourno said,

    March 24, 2009 at 4:16 pm

    “I prefer newspapers to be energetic vital popular (and sometimes wrong) to dreary, dead, unread, bankrupt and gone (as they increasingly are in the US outside a few metropolitan hotspots).”

    You’re welcome to your preferences, but it’s a false comparison: many newspapers have gone and will presently be gone that were/are also energetic, vital, and popular.

    Dry troubled newspaper or wet troubled newspaper, their ownership structure has tended to be share-based. So when a bunch of shareholding morons start panicking about the Internet and everything else stealing advertisers and readers away, their knee-jerk reaction is:

    -To cut costs to boost (often already healthy, in the sense a privately-owned enterprise would be happy with them) margins, thereby screwing their quality in a way that is very obvious to readers and advertisers, who then move on, and kill the margins

    -To dumb everything down, chasing what they believe is the lowest common denominator, who they think cares more about reading something entertaining than being informed, and who they nonetheless think advertisers are interested in reaching

    But the problems start right away: this stereotype of the lowest-common-denominator reader isn’t going to be the sort of stereotype high-disposable income stereotype advertisers want to chase.

    And even if they did, it’s obvious to anybody besides panic-stricken shareholding morons that there are a MILLION things more entertaining than an ‘entertaining’ newspaper. It’s just not the right format for lowest-common-denominator entertainment. Why go to all the trouble of READING pablum when you can watch just about any idiocy roll out on gossip sites, reality shows, scaremongering current affairs programmes with presenters with nice tits, etc.?

    Is any news group more ‘entertaining’ and less concerned with dry facts than the Daily Mail group? They’ve just slashed the costs to hell; the only problem is they can’t dumb down anymore. Predictably enough, they’re losing ad revenue and will almost certainly lose a great deal more.

    I don’t disagree with your explanation of why science reporting can get so crappy and homogenous, but I really doubt competing with celebrinews and murder splats is the best route forward for science reporting or other news reporting in the papers . . . either for the papers or the readers. The ‘lowest common denominator’ has better things to do than be condescended to in print, and the newspaper medium has too many other possibilities.

  12. 10channel said,

    March 24, 2009 at 10:47 pm

    In something as important as health and medicine, one would think that at least that caution is the best method of reporting the facts – but no, most British newspapers seem to have no restraint in writing laudatory tracts for this or that related to health. If this is what sells, and if this is what the people want, then it is also a fault of the people. Do the people in the U.K. just fail to be sceptical about health and medicine or what?

  13. SimonCox said,

    March 25, 2009 at 12:22 pm

    @BobWard

    “It may not be desirable that journalists have such ridiculously short deadlines (even shorter these days with online news, 24-broadcast news and shrinking incomes) but it is a fact. Newspapers could of course employ two or three times more journalists, but then readers would need to stump up five or six times more money each day to buy the newspaper.”

    This is a straw man – you assume that research results must be reported as soon as possible. Why must they?

    The scientific research being discussed takes months or years to carry out, and the results, when published, generally take weeks, months or years to filter into any kind of technology or policy that will actually have an effect our lives. Given this, there is no credible reason that I can think of for newspapers to report scientific research findings on the day that the relevant press release is issued.

    Put it like this: if you were given the choice between reading a misleading, error-strewn science article on Monday and reading a meaningful, factually correct science article on Tuesday, which one would you choose?

  14. NeilHoskins said,

    March 27, 2009 at 3:29 pm

    Silly me: I thought that headline was referring to the pope and his views on condoms making the HIV/AIDS epidemic worse. You’re oddly silent on that one, Ben: is it OK to talk scientific bollocks if you wear a pointy hat?

  15. heavens said,

    March 27, 2009 at 4:24 pm

    @Bob Ward, who said, “And frankly, if journalists had to assume that all parts of a press release were wrong until proven otherwise, then…”

    Then they would be doing their jobs properly.

    I’ve been on both sides of a press release. What’s included in the press release is usually technically accurate, but what’s left out is often much more important. The level of “honesty” is often the level that you get from asking a young boy whether he got his clothes muddy.

    The boy might look at the mud caked on his shoes, the filth on his hands, and the streaks in his hair, but note that from collar to socks there’s not a speck of mud, and “honestly” tell you that he didn’t get any mud on his clothes.

    If I were writing for a pro-PSA group, I’d never tell you that you have to screen 1400 men to save one life. I’d never tell you that saving one life requires putting 47 other men through useless treatment. You might not make the “right” choice if I did that.

    It’s not just prostate cancer; we could use any number of examples:

    If I were writing for a pro-HIV testing center, I’d never tell you that testing people (in the developed world) with no identifiable risk factors (e.g., no anal sex) is going to produce an enormous proportion of false positives.

    If I were writing for a pro-choice group, I’d never tell you that the reason abortion isn’t associated with breast cancer is because they compare the risk of a woman that has an early abortion at age 20 and her sole full-term pregnancy at age 30 to a woman that never got pregnant until she was 30 — instead of to a woman that had two children, the first at age 20 and the second at age 30. We all know that “age at first full-term pregnancy” and “number of children” have a significant impact on breast cancer risk, but they set up the studies to compare the risk to someone that never got pregnant in the first place, instead of comparing it to a woman who got pregnant and gave birth instead of getting an abortion (i.e., the two actual choices facing the woman in question, given that time travel has not yet been perfected).

    Similarly, I’d never tell you that young women who refuse abortions typically do better in life (including income, education, housing, staying out of criminal justice, and more) than the ones that have an abortion. The facts are real, and every person whose grandfather said, “I realized when your uncle was born, in the bad old pre-abortion days, that I needed to grow up in a hurry” knows that it’s not entirely a matter of already-successful women choosing to give birth while losers get abortions.

    If I were writing for a Chronic Fatigue Syndrome charity, I’d never tell you that 90%+ of CFS patients actually have undiagnosed orthostatic hypotension. Who wants to give up 90%+ of their support base?!? Most “CFS” patients need a tilt-table test and three litres of water every day, not a poor-mysterious-us group, but you won’t get that information from the charity.

    All rare diseases are underdiagnosed — if you believe the press releases.

    I could go on, but I think you get the picture. It’s irresponsible to trust in the completeness of a press release. It’s unprofessional to assume that the press release is the best method of presenting the information.

  16. mikewhit said,

    April 1, 2009 at 12:37 pm

    but they set up the studies to compare the risk to someone that never got pregnant in the first place, instead of comparing it to a woman who got pregnant and gave birth instead of getting an abortion

    Surely they should be comparing like with like – the woman who had the abortion did not have a full-term baby. Otherwise you have two factors instead of one: a) having an abortion b) having a full-term pregnancy

    They should instead perhaps compare with someone who had a miscarriage at the same (st)age.

    Or, the study you propose should include miscarriages as well as abortions.

  17. mikewhit said,

    April 2, 2009 at 9:03 am

    But in any case I don’t think anyone desperate enough to consider having an abortion is going to be swayed in their decision by some marginal difference in long-term breast cancer statistics !

  18. ivy said,

    April 5, 2009 at 4:28 am

    How do I track my posts? I remember posting somwehere on this board but I cannot find it.

  19. wayscj said,

    November 21, 2009 at 6:24 am

    ed hardy ed hardy
    ed hardy clothing ed hardy clothing
    ed hardy shop ed hardy shop
    christian audigier christian audigier
    ed hardy cheap ed hardy cheap
    ed hardy outlet ed hardy outlet
    ed hardy sale ed hardy sale
    ed hardy store ed hardy store
    ed hardy mens ed hardy mens
    ed hardy womens ed hardy womens
    ed hardy kids ed hardy kids ed hardy kids