The caveat in paragraph number 19

October 16th, 2010 by Ben Goldacre in bad science, irrationality research, media | 19 Comments »

Ben Goldacre, The Guardian, Saturday 16 October 2010

You will be familiar with the Daily Mail’s ongoing project to divide all the inanimate objects in the world into the ones that either cause or prevent cancer. Individual entries are now barely worth documenting, and the phenomenon is best appreciated in bulk through websites such as the Daily Mail Oncological Ontology Project and Kill Or Cure, with its alphabetised list: from almonds, apples and artificial light; through horseradish, hotdrinks and housework; to wasabi, water, watercress, and more.

But occasionally one story pops up to illustrate a wider issue, and “Strict diet two days a week ‘cuts risk of breast cancer by 40 per cent’” is a good example. It goes on: “a strict diet for two days a week consisting solely of vegetables, fruit, milk and a mug of Bovril could prevent breast cancer, scientists say.”

Now, obviously, if you have the time to track down the academic paper which this news article is describing, from the October edition of the International Journal of Obesity, you will immediately discover that it is not a study of breast cancer, and it does not find that the risk of cancer is reduced by 40% (although it does measure a couple of hormones). The press release wasn’t exactly a masterpiece of clarity either, as CRUK’s excellent science blog immediately pointed out, but in any case, the study doesn’t measure breast cancer as an outcome, at all.

But if I was to leave it there, then the journalist would correctly complain: because after all the grand and misleading claims, firstly, in the body of the piece, they do mention that the outcome is not cancer, but some hormones related to cancer (with no explanation of how tenuous that relationship is). Then, finally, at the very bottom of the piece, they have the reality. Although it’s not spoken in in the authoritative third person of the paper itself, it’s there, in a quote, at paragraph number 19.

“But Dr Julie Sharp, senior science information manager at Cancer Research UK, said: ‘This study is not about breast cancer, it’s a study showing how different diet patterns affect weight loss and it’s misleading to draw any conclusions about breast cancer from this research.’”

The late caveat, torpedoing the central premise of a news piece, is a common strategy in many newspapers. But what use is this information, at the end of a long article, in paragraph number 19?

The way that people read newspapers has been studied widely using eyetracking technology. It’s through this that we discover, for example, that when presented with a full length photograph of a man, men are more likely to look at the penis area than women.

Most of this research is more preoccupied with ads than news, because research in so many fields is funded by people with both questions and money – page top left is best, probably – but there is plenty of useful stuff, much of it by the Poynter Institute.

They did an early study in 1990, finding some predictable stuff: that photos attract attention; eyes travel from the dominant photo to the biggest headline, then teasers, and finally text; text is read the least, headlines the most; and so on.

Their most recent project was far bigger: they took a representative sample of 582 people from 4 cities in the US, and invited them in to read a newspaper and website as they normally would, wearing the eyetracking equipment, over 5 days in 2006, for 15 minutes each. This yielded a dataset of more than 102,000 eye stops.

This is what they found: by the time you get to a story length of 8 to 11 paragraphs, on average, your readers read only half the story. A minority will make it to paragraph number 19, where, on this occasion, a fraction of the readers of the Daily Mail would have discovered that the central premise of the news story – that a new trial had found a 40% reduction in cancer through intermittent dieting – was false.

Caveats in paragraph 19 are common. This evidence strongly suggests that they are also a sop: they permit a defense against criticism, through the strictest, most rigorous analysis of a piece. But if your interest is informing a reader, they are plainly misleading.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

19 Responses



  1. Mark Wainwright said,

    October 16, 2010 at 3:51 am

    I don’t think you even need to ask which paragraph the disclaimer is in, or bring out the heavy guns on reading research, to note that the headline is “Strict diet two days a week ‘cuts risk of breast cancer by 40%'”, which is a clear statement that at least according to someone quoted in the article, a strict diet, if adhered to on two days a week, cuts the risk of breast cancer by 40%. But it doesn’t quote anyone saying that – the only relevant quote is that no such conclusion can be drawn. So sure, the text of the article would be fine, if it appeared under the headline “Strict diet two days a week ‘cannot be meaningfully linked to breast cancer'”, but that’s not the way it went.

  2. BobP said,

    October 16, 2010 at 8:35 am

    Boring. I only read half of it.

  3. JimG said,

    October 16, 2010 at 9:22 am

    Yeah, that’s not a caveat so much as a 100% contradiction. Maybe the Mail think that as long as they include the truth *somewhere* in the article it is perfectly acceptable to make the headline and opening paras say anything they like, however untrue.

  4. muscleman said,

    October 16, 2010 at 10:01 am

    Ben can you get the Graun to turn on comments on the version on CiF please?

    When designing posters at scientific meetings I would advise colleagues to put most thought into their title, then the abstract and to make the pictures take up more space than the text. Because most people will just read the title (the rest will see if the pictures are interesting), a subset of them will read the abstract and only a minority will read the text.

    A room full of posters is actually quite analogous to browsing a website.

  5. walks with tench said,

    October 16, 2010 at 12:26 pm

    Surely the gist of this is at para 11: “Not being a lard-arse reduces risk of cancer”. They just dressed it up a bit so they could talk about bosoms!

  6. spk76 said,

    October 16, 2010 at 2:07 pm

    tldr

  7. Seye said,

    October 16, 2010 at 5:40 pm

    As far as the PCC are concerned this kind of ‘reporting’ does not break their code even though it’s obvious that it’s intentionally misleading.

    I made a complaint about a similar situation a while ago and as far as the PCC are concerned you can make any claim in a headline and article as long as any statements you make are non-committal (apparently the headline escapes this rule) and you include at least one quote at the end from someone disputing the nonsense above it.

    The PCC is an absolute disgrace and the Daily Mail is all the evidence that anyone should need to prove that the press are incapable of self regulation.

  8. Xobbo said,

    October 17, 2010 at 3:15 pm

    Well said, Seye.

    Ben, you’ve given us a good analysis of the pointlessness of adding the truth right at the end of a false article (I would describe it as “misleading” if it was ambiguous, but as usual the statements here are clearly not based on any facts), but I’d say that if the newspapers are allowed to print false headlines, it makes little difference whether it’s contradicted in paragraph 19 or paragraph 1, because most people who see the headline don’t even buy the paper.

    I think the fact that these papers are allowed to print things demonstrably at odds with the truth is one of the main causes of many of the problems we see these days, from confusion or distrust of science to the scapegoating of Muslims, the EU, asylum seekers or wheelie bins for whatever goes wrong. And the root of this problem is the fact that unlike other media, the papers are self-regulated, by the shamelessly biased PCC.

    I just hope you get as angry about it as I am, then convince all your showbiz mates that we need an urgent campaign to fix this, in the name of rationalism! ;-)

  9. fluffy_mike said,

    October 17, 2010 at 11:57 pm

    Many of you will have seen this excellent (though infuriating) website exposing daily tabloid excesses – sorry, lies:

    tabloid-watch.blogspot.com/

    The establishment’s complicity in the piss-poor journalism tolerated in the UK is utterly shameful

  10. HowardW said,

    October 18, 2010 at 12:47 pm

    If somebody wrote a newspaper article with the headline “All journalists at the Daily Blah are lying c*nts who deliberately mislead the public with fake health stories in order to sell papers”, and then added a caveat in paragraph 19 that said:

    “But Fred Smith said: Actually they’re all decent journalists at the Daily Blah who always work to the highest journalistic standards.”

    Would this stand up as a valid defence if the paper or its journalists sued for defamation or libel?

    Howard

  11. Norman said,

    October 18, 2010 at 1:26 pm

    Sigh! The Daily Mail, telling lies – well I never!

    I always maintained, and still do, that in the extremely unlikely event of me ever becoming prime minister, that the red tops would be the first to go.

    Any other newspaper, or anything claiming to be a newspaper, would have to stick to the new rules:

    * You only report new, not speculation, or what you have just made up.
    * See rule 1 above!
    * Horoscopes are not allowed.

    You catch my drift.

    Cheers,
    Norm.

  12. Joel said,

    October 19, 2010 at 10:17 am

    Why is there a big picture of Gillian Haddock illustrating the article, when she wasn’t one of the authors of the paper and isn’t even quoted in the article?

  13. IMC said,

    October 19, 2010 at 6:15 pm

    And what is “C-reactor protein”? Do they mean C-reactive protein? I thought there was, at best, a tenuous link between elevated CRP and incidences of colon cancer (not withstanding CRP’s possible role in inflammatory atherosclerosis). Here the Daily Fail seems to have linked it to breast cancer as well…

    Overall, this article is so badly researched and written it could only come from one newspaper.

  14. Wid said,

    October 21, 2010 at 12:35 am

    Is it possible that the PCC actually makes it easier to mislead people than if the press was utterly unregulated? Presumably the spectacle of punishment and regulation the PCC provides (as well, incidentally, as the practice of unread but noticed voluntary retractions) gives the papers more credibility that they would have if utterly unregulated. But since the PCC provides little substantive regulation, it does not effectively cut down on the nonsense. Therefore in terms creating false credibility there is a fair bit of gain and very little loss in the PCC for papers. I can’t decide whether the answer is just to regulation all together or to set up a new form of regulation. Giving powers under law to a regulatory body for the press is a concept I instinctively shy away from, but perhaps I am being paranoid?

  15. skyesteve said,

    October 22, 2010 at 8:59 am

    Perhaps I’m just naive and simplistic but for me the solution to the problem of the press telling lies has always been easy and straight forward.
    The emphasis should be on the press to check the facts before they publish. On that basis I would simply ban from publication any newspaper shown to be deliberately telling lies.
    At the very least we should have a rule that says any retraction or apology printed by a newspaper should occupy the same page number and have the same level of prominence as the original article.

  16. ianbright said,

    November 3, 2010 at 4:16 pm

    Surley one implication is to read most medical and science articles in the mainstream press from the bottom to the top. Saves a lot of time and gets to the point more quickly.

  17. Jono4174 said,

    November 15, 2010 at 10:48 pm

    19 get

  18. Jono4174 said,

    November 15, 2010 at 10:48 pm

    19 get get

  19. Jono4174 said,

    November 15, 2010 at 10:49 pm

    bad science by Ben Goldacre is lies

    :P