Saturday June 21, 2008
The best thing about this job is you have an excuse to read the Daily Mail every day: but sometimes, out of the corner of my eye, I worry that it might infect me. We are all biased by the information we expose ourselves to, through our friends, our reading, and our choices in life. I think science coverage is pretty poor, and a lot of it is plainly wrong. Am I biased by my work? Do I see only the bad, in a very literal sense? Like many before it, this is a problem which can only be cracked with an ingenious idea from 20,000BC called “counting” (quibblers are welcome, my date is from the notched “Ishango bone“).
Gary Schwitzer used to be a journalist, but now he has turned to quantitative analyses of journalism, and this month he published an analysis of 500 health articles from mainstream media in the US. Theresults were dismal. Only 35% of stories were rated satisfactory for whether the journalist had “discussed the study methodology and the quality of the evidence”: because in the media, as you will have noticed, science is about absolute truth statements from arbitrary authority figures in white coats, rather than clear descriptions of studies, and the reasons why people draw conclusions from them. Only 28% adequately covered benefits, and only 33% adequately covered harms. Articles routinely failed to give any useful quantitative information in absolute terms, preferring unhelpful eye-catchers like “50% higher” instead.
Was this new? No. The same thing has been shown in Canada and Australia, and in the US almost a decade ago. Does it matter? Yes. Regardless of what they say in surveys about trusting doctors and priests, and despising hacks, in reality, people listen to journalists. This is not idle speculation.
A 2005 study in the Medical Journal of Australia looked at the impact of Kylie Minogue’s breasts on mammogram bookings, looking at appointments made before, during, and after the peak of publicity over her cancer. This period saw a 20-fold increase in news coverage of breast cancer and that, in turn, had a significant impact on health behaviour at a population level. Bookings rose by 40% during the 2 week peak, and six weeks later they were still up by a third. The increase among previously unscreened women in the 40-69 year age group was 101%. These surges were unprecedented.
Am I cherry picking? A systematic review from the Cochrane Collaboration found 5 studies looking at the use of specific health interventions before and after media coverage of specific stories, and each found that favorable publicity was associated with greater use, and unfavorable with lower.
And it’s not just the public: medical practice is influenced by the media too. Coverage of a flesh-eating bacteria outbreak led to a massive increase in group A streptococcus screening in one A&E department (from 55 to 103 per 1000 visits), despite no change in the number of children presenting with appropriate symptoms. Fair enough, you might say.
But even academics are influenced by media coverage: Phillips et al showed, in a seminal paper from the New England Journal of Medicine in 1991, that if a study was covered by The New York Times, it was significantly more likely to be cited by other academic papers. Was coverage in the NYT just a surrogate marker for the importance of the research? History provided the researchers with a control: for 3 months, large parts of the NYT went on strike, and while the journalists did produce an “edition of record”, this was never published. They wrote stories about academic research, using the same criteria of importance as ever: the research they wrote about, in articles which never saw the light of day, saw no increase in citations.
People read newspapers. Despite everything we think we know, their contents seep in, we believe them to be true, and we act upon them.
Schwitzer G PLoS Med 2008 5(5): e95
Chapman S et al. Med J Aust. 2005 Sep 5;183(5):247-50.
Grilli R et al. Cochrane Database of Systematic Reviews 2001, Issue 4: CD000389.
Sharma V et al Arch Pediatr Adolesc Med. 2003;157:257-60
Phillips DP et al. N Engl J Med. 1991;325:1180-3.