Ben Goldacre, The Guardian, Saturday 16 October 2010
You will be familiar with the Daily Mail’s ongoing project to divide all the inanimate objects in the world into the ones that either cause or prevent cancer. Individual entries are now barely worth documenting, and the phenomenon is best appreciated in bulk through websites such as the Daily Mail Oncological Ontology Project and Kill Or Cure, with its alphabetised list: from almonds, apples and artificial light; through horseradish, hotdrinks and housework; to wasabi, water, watercress, and more.
But occasionally one story pops up to illustrate a wider issue, and “Strict diet two days a week ‘cuts risk of breast cancer by 40 per cent’” is a good example. It goes on: “a strict diet for two days a week consisting solely of vegetables, fruit, milk and a mug of Bovril could prevent breast cancer, scientists say.”
Now, obviously, if you have the time to track down the academic paper which this news article is describing, from the October edition of the International Journal of Obesity, you will immediately discover that it is not a study of breast cancer, and it does not find that the risk of cancer is reduced by 40% (although it does measure a couple of hormones). The press release wasn’t exactly a masterpiece of clarity either, as CRUK’s excellent science blog immediately pointed out, but in any case, the study doesn’t measure breast cancer as an outcome, at all.
But if I was to leave it there, then the journalist would correctly complain: because after all the grand and misleading claims, firstly, in the body of the piece, they do mention that the outcome is not cancer, but some hormones related to cancer (with no explanation of how tenuous that relationship is). Then, finally, at the very bottom of the piece, they have the reality. Although it’s not spoken in in the authoritative third person of the paper itself, it’s there, in a quote, at paragraph number 19.
“But Dr Julie Sharp, senior science information manager at Cancer Research UK, said: ‘This study is not about breast cancer, it’s a study showing how different diet patterns affect weight loss and it’s misleading to draw any conclusions about breast cancer from this research.’”
The late caveat, torpedoing the central premise of a news piece, is a common strategy in many newspapers. But what use is this information, at the end of a long article, in paragraph number 19?
The way that people read newspapers has been studied widely using eyetracking technology. It’s through this that we discover, for example, that when presented with a full length photograph of a man, men are more likely to look at the penis area than women.
Most of this research is more preoccupied with ads than news, because research in so many fields is funded by people with both questions and money – page top left is best, probably – but there is plenty of useful stuff, much of it by the Poynter Institute.
They did an early study in 1990, finding some predictable stuff: that photos attract attention; eyes travel from the dominant photo to the biggest headline, then teasers, and finally text; text is read the least, headlines the most; and so on.
Their most recent project was far bigger: they took a representative sample of 582 people from 4 cities in the US, and invited them in to read a newspaper and website as they normally would, wearing the eyetracking equipment, over 5 days in 2006, for 15 minutes each. This yielded a dataset of more than 102,000 eye stops.
This is what they found: by the time you get to a story length of 8 to 11 paragraphs, on average, your readers read only half the story. A minority will make it to paragraph number 19, where, on this occasion, a fraction of the readers of the Daily Mail would have discovered that the central premise of the news story – that a new trial had found a 40% reduction in cancer through intermittent dieting – was false.
Caveats in paragraph 19 are common. This evidence strongly suggests that they are also a sop: they permit a defense against criticism, through the strictest, most rigorous analysis of a piece. But if your interest is informing a reader, they are plainly misleading.