Ben Goldacre
The Guardian,
Saturday June 21, 2008
The best thing about this job is you have an excuse to read the Daily Mail every day: but sometimes, out of the corner of my eye, I worry that it might infect me. We are all biased by the information we expose ourselves to, through our friends, our reading, and our choices in life. I think science coverage is pretty poor, and a lot of it is plainly wrong. Am I biased by my work? Do I see only the bad, in a very literal sense? Like many before it, this is a problem which can only be cracked with an ingenious idea from 20,000BC called “counting” (quibblers are welcome, my date is from the notched “Ishango bone“).
Gary Schwitzer used to be a journalist, but now he has turned to quantitative analyses of journalism, and this month he published an analysis of 500 health articles from mainstream media in the US. Theresults were dismal. Only 35% of stories were rated satisfactory for whether the journalist had “discussed the study methodology and the quality of the evidence”: because in the media, as you will have noticed, science is about absolute truth statements from arbitrary authority figures in white coats, rather than clear descriptions of studies, and the reasons why people draw conclusions from them. Only 28% adequately covered benefits, and only 33% adequately covered harms. Articles routinely failed to give any useful quantitative information in absolute terms, preferring unhelpful eye-catchers like “50% higher” instead.
Was this new? No. The same thing has been shown in Canada and Australia, and in the US almost a decade ago. Does it matter? Yes. Regardless of what they say in surveys about trusting doctors and priests, and despising hacks, in reality, people listen to journalists. This is not idle speculation.
A 2005 study in the Medical Journal of Australia looked at the impact of Kylie Minogue’s breasts on mammogram bookings, looking at appointments made before, during, and after the peak of publicity over her cancer. This period saw a 20-fold increase in news coverage of breast cancer and that, in turn, had a significant impact on health behaviour at a population level. Bookings rose by 40% during the 2 week peak, and six weeks later they were still up by a third. The increase among previously unscreened women in the 40-69 year age group was 101%. These surges were unprecedented.
Am I cherry picking? A systematic review from the Cochrane Collaboration found 5 studies looking at the use of specific health interventions before and after media coverage of specific stories, and each found that favorable publicity was associated with greater use, and unfavorable with lower.
And it’s not just the public: medical practice is influenced by the media too. Coverage of a flesh-eating bacteria outbreak led to a massive increase in group A streptococcus screening in one A&E department (from 55 to 103 per 1000 visits), despite no change in the number of children presenting with appropriate symptoms. Fair enough, you might say.
But even academics are influenced by media coverage: Phillips et al showed, in a seminal paper from the New England Journal of Medicine in 1991, that if a study was covered by The New York Times, it was significantly more likely to be cited by other academic papers. Was coverage in the NYT just a surrogate marker for the importance of the research? History provided the researchers with a control: for 3 months, large parts of the NYT went on strike, and while the journalists did produce an “edition of record”, this was never published. They wrote stories about academic research, using the same criteria of importance as ever: the research they wrote about, in articles which never saw the light of day, saw no increase in citations.
People read newspapers. Despite everything we think we know, their contents seep in, we believe them to be true, and we act upon them.
Schwitzer G PLoS Med 2008 5(5): e95
Chapman S et al. Med J Aust. 2005 Sep 5;183(5):247-50.
Grilli R et al. Cochrane Database of Systematic Reviews 2001, Issue 4: CD000389.
Sharma V et al Arch Pediatr Adolesc Med. 2003;157:257-60
Phillips DP et al. N Engl J Med. 1991;325:1180-3.
Jessicathejourno said,
June 21, 2008 at 8:10 am
1) While agreeing with your drift, your quibble isn’t with journalists; it’s with marketing and ownership structures. As a B2B journalist I’m not allowed to make ‘mistakes’ in my science coverage, on the assumption that serious people with investment dollars read it. Wide media journalists are ENCOURAGED to make mistakes in their coverage on the assumption that it can (as you prove it does) get people really excited about stuff and, one hopes, buy more newspapers/watch more news programmes/sink their money into products that benefit members of the board of directors.
I assume you’re making some money from the marketing and ownership structure of the Guardian, but if you really want to avoid being biased in your work you’ll have to care a little less whose toes you step on. Sure, journalists are easy targets and they’re foolish enough to attach their name to rubbish, so it’s easy to make the Stupid Humanities Science Illiterate badge stick. But if you want to make a difference, you have to target the pimps, not just the hos.
2. I like that this column came out the same day the Independent front-paged about how measles is back.
alexlockwood said,
June 21, 2008 at 10:07 am
Hi
1. As a journalist turned academcic, I agree with Jessica that the ownership structures of journalism and the increasing economic and speed pressures are, if blame were a quantitative measure, the more responsible for instances of poor reporting. But that’s when journalists make mistakes. When economy is mixed with ideology, the skewed or distorted reporting is purposeful. And as some good commentary I’ve read recently puts it, it’s not always about the science.
However, not to say journalists don’t have their days where they just plain fall asleep at the wheel. Ben, an example of what Nick Davies calls ‘churnalism’ in the reporting of climate change, if you’ve not picked it up yet. Lots of ink on the blogs about Dr. Tom Chalko’s recent momentous self-published paper on how climate change is linked to the fivefold increase in Earthquakes. What? But it got through via AP, was published by CBS, Yahoo, MSNBC etc… (who mainly quickly pulled it). You can read more over here.
2. I like this column.
simongates said,
June 21, 2008 at 10:31 am
Sorry, this is amazingly anal and pedantic, but officially I believe it’s The Cochrane Collaboration, not the Cochrane Collaboration (i.e. capital T).
TimW said,
June 21, 2008 at 10:34 am
The pure power of memes – when are people going to start taking us seriously?
elfy said,
June 21, 2008 at 12:58 pm
Management structures and time pressures and things like that certainly have their roles to play in blame, but a lot of the problems with science articles could be very quickly fixed by the journalist concerned.
I used to do a lot of student journalism, and wrote some basic advice for student journalists covering science stories – which, if followed, would probably mean they were better than a good 50% of the professional science journalists out there. A lot of it was very, very simple stuff.
Don’t just say “Scientists” as if they were a homogeneous group, instead say “Researchers at X” or whatever. Say what how the study was conducted and what the study actually showed, not just what the hypothesis was and that the evidence supported it. These things don’t take long, and don’t require chasing up new sources – just reading the study. If you can’t do that, you shouldn’t be writing science articles.
dbhb said,
June 22, 2008 at 12:12 pm
Hold on there Dudley. Yes it’s natural that people will buy newspapers with striking sensationalist headlines (sometimes to take umbrage with the story). In exactly the same way, it’s only natural that companies and corporations can sell more products if they’re liberal with the truth.
Shrugging and saying ‘that’s capitalism baby’ is a weird kind of excuse for blaming the customers and doing nothing about it. After all, we normally feel quite justified in legislating to regulate the worst excesses of capitalism.
deejay2257 said,
June 22, 2008 at 12:53 pm
In extraordinary language you worry that reading the Daily Mail “might infect” you. No doubt you are a picture of purity and health right now.
Is reading the DM to be considered the manifestation of a disease? You seem to believe that keeping away from it will keep you pure and healthy. Is reading only Polly Toynbee or nodding in agreement with every Guardian Leader a sign of health?
In scientific terms (as I’m sure you are aware) you should subject every theory to as much scrutiny from as many different angles as is possible. And those who refuse to examine their pet theories cannot be called good scientists. The same is true of nonscientific opinions. The Daily Mailers are not infected, Ben, they just hold different opinions to you.
I’m sure you are aware, Ben, that Guardianistas can be every bit as hysterical and self righteous when they want to be – as you yourself prove in the first para of this article.
Dudley said,
June 22, 2008 at 1:28 pm
dbhb –
Didn’t really mean to say that nothing should be done. But I am suggesting that the problem is maybe a little more deep-rooted than could be solved by journos doing their jobs better (but in a less interesting way. therefore probably less successfully). Intervention to help people understand the ludicrousness of certain ways of talking about science would probably need to come in the form of a massively reformed education system. Journalists are selling a product: to blame them and not the state for an increasingly ill-informed and logic-resistant public feels a bit too “shoot the messenger.”
Nelson said,
June 22, 2008 at 3:03 pm
The Daily Mail to the Guardian.. the whole gamut of opinion, from A to B 🙂
raygirvan said,
June 23, 2008 at 12:01 pm
known in scientific circles as “a joke”
Yep. However, there is something particularly objectionable about the DM. It’s not so much what it believes (though its repetitive “Asylum seekers laughed at dying Diana as house prices plummeted” line is pretty tedious) as, in contrast the Guardian, the authoritarian guarding of those beliefs. It has a cowardly resistance to open discussion, which is anathema to anyone from a science background (or anyone who doesn’t what to be told what to think without right of reply).
Compare, for instance, the Guardian forums, where you can say pretty much what you like, to the DM’s discussion threads, that are assiduously censored to remove comments on factual errors and expression of differing viewpoints.
Or, for instance, can you find a DM equivalent of the Guardian’s Corrections and clarifications with its policy “to correct significant errors as soon as possible”?
casper_gutman said,
June 23, 2008 at 12:16 pm
Why should the Mail need a corrections and clarifications section? Everyone knows it’s infallible.
Godwin’s law? What’s that?
deejay2257 said,
June 23, 2008 at 1:08 pm
Daibhid C: I never thought that the “might infect” comment was anything other than a throwaway remark but that’s why I though it was so telling.
raygirvan: Yes, I agree, you certainly can say pretty much anything on the Guardian comment boards as demonstrated by the preponderance of 9/11 “truth-seekers”.
Ben Goldacre said,
June 23, 2008 at 2:05 pm
ooh i’m always keen to get called out on something like this.
i think it would be a very fair criticism if we were talking about a risk or benefit issue where absolute figures are relevant to the readers health behaviour or reading interest, but the fact that breast cancer screening bookings “rose by 40%” is only interesting for the fact of a change related to media coverage, i’d say, while the absolute figures on breast cancer screening bookings per week are not very interesting or relevant themselves. all refs were above and a reader could have checked (although no weblinks as i was away from computer for a few days, also worth saying i do send these to the guardian too, usually, although they don’t use them). but yes, good call in some respects, at least a keepontoes one, and if anyone is interested the figures are: 10,513 screening bookings per week went up to 14,716 per week.
pob said,
June 23, 2008 at 2:21 pm
deejay2257,
With regard to your point about the Daily Mail, I think Dr Goldacre’s main gripe is that with that particular newspaper is its consistent disregard for checking facts, its inability to accurately present scientific reports or summaries, its hysterical delight in proclaiming the next epidemic and the lack of duty is exhibits in delivering such news to its readers. However, you can find within Dr Goldacre’s articles criticisms of scientific journalism that also cite poor standards within the Guardian.
I believe his viewpoint is that the Daily Mail is the worst proponent of such reporting whilst claiming to wear the clothes of a quality newspaper when it clearly makes no effort to attain even the basic level of such an ideal.
To address the wider discussion of journalistic standards I would level that the public (or educational standards) are guilty of failing to redress the balance of poor journalism. I suggest that this is due to readers themselves being unable to scrutinise articles with an objective, critical mind. One does not need a science degree in order to determine if the facts of a case appear to add up and that is where I think readers fail themselves.
I have the feeling that many simply wish to be spoon fed rather than have to think for themselves and hold science as a boring subject which only geeks and brainiacs can hope to understand. What a poor state of affairs.
raygirvan said,
June 23, 2008 at 4:05 pm
pob> due to readers themselves being unable to scrutinise articles with an objective, critical mind
which is particular poison when coupled with the DM’s censorship of dissenting views that might inform them of the results of such scrutiny.
healthydistrust said,
June 23, 2008 at 5:08 pm
If anyone’s interested, I’ve posted a review of the literature on journalists vs scientists, the influence of the media when it comes to health issues, et cet.
a-healthy-distrust.blogspot.com/
Henry said,
June 24, 2008 at 4:53 pm
Interestingly, the most recent analysis of the ‘Kylie Effect’ showed that, despite the increase in Australian screening rates, there weren’t any extra cases of cancer detected. (info.cancerresearchuk.org/news/archive/newsarchive/2008/june/18626580 & ije.oxfordjournals.org/cgi/content/abstract/dyn090v2)
The authors concluded that “High-publicised illnesses may affect both consumer and provider behaviour. Although they present opportunities to improve public health, they also have the potential to adversely impact the appropriateness and cost-effectiveness of service delivery.
Is it as simple as wishing for ‘better science journalists’? We should be so lucky…”
GH05T said,
July 17, 2008 at 6:43 pm
nekomatic said,
June 23, 2008 at 12:47 pm
“Richard, the point is that quoting an effect in relative terms is a good way of quantifying the effectiveness of an intervention in comparison with other interventions (or with doing nothing)”
Not true. Imagine a study of 100 people, 50 given treatment, 50 untreated, where 1 person on the treated side dies during the study period of 1 year, and 2 people die on the untreated side. Anyone familiar with the most basic tenants of proper scientific method could tell you that the study was meaningless. The sample was too small, the study period too short. Quality of life is not assessed. However, relative statistics allows us to report that New Miracle Cure X cut the death rate in half. A good scientist may use relative numbers in a brief summary or an abstract but would never neglect to include the real numbers in the actual paper. Real numbers aren’t just about interpreting results; they are also, and in my opinion much more importantly, about interpreting the validity of the research. That is why so many people don’t want them published.
HolfordWatch said,
September 16, 2009 at 9:11 pm
via @EvidenceMatters and @ivanoranksy
The issue of Kiernan’s findings about press increasing academic citations has been revisited in discussion.
Does press coverage of journal articles really matter?
Kiernan: Diffusion of News about Research.