Comment
Risky business
Health-scare stories often arise because their authors simply don’t understand numbers
Ben Goldacre
Monday June 20, 2005
The Guardian
Competence always looks better from a distance, but I have a confession to make: I’m a doctor, and I just don’t understand most of the stories on health risks in the news. I don’t mean I can’t understand the fuss. I mean I literally can’t understand what they’re trying to communicate to me.
Last week, we were told that red meat causes bowel cancer, and Nurofen causes heart attacks, but I was no wiser. Try this, on bowel cancer, from the Today programme: “A bigger risk meaning what, Professor Bingham?” “A third higher risk.” “That sounds an awful lot, a third higher risk; what are we talking about in terms of numbers, here?” “A difference … of around about 20 people per year.” “So it’s still a small number?” “Umm … per 10,000 …”
Article continues
HG Wells, 150 years ago, said that statistical thinking would one day be as important as the ability to read and write in a modern technological society. I disagree; probabilistic reasoning is difficult for everyone, but everyone understands normal numbers. Which is why “natural frequencies” are the only sensible way to communicate risk.
Let’s say the risk of having a heart attack in your 50s is 50% higher if you have high cholesterol: that sounds pretty bad. Let’s say the extra risk of having a heart attack if you have high cholesterol is only 2%. That sounds OK to me. But they’re both talking about the same (hypothetical) figures. Out of a hundred men in their 50s with normal cholesterol, four will be expected to have a heart attack; whereas out of 100 men with high cholesterol, six will be expected to have a heart attack. That’s two extra heart attacks. Those are natural frequencies. Easy.
Natural frequencies are readily understandable, because instead of using probabilities, or percentages, they use concrete numbers, just like the ones you use every day to check if you’ve lost a kid on a coach trip, or got the right change in a shop. Lots of people have argued that we evolved to reason and do maths with concrete numbers like these, and not with probabilities, so we will find them more intuitive. I’ll start believing evolutionary psychologists on the day they start defecating in the back garden at dinner parties like the monkeys they extrapolate from, but the point stands. Simple numbers are simple.
I’m not alone in finding percentages unhelpful, incidentally. There are studies of doctors, and commissioning committees for local health authorities, and people from the legal profession, that show that even people who interpret and manage risk for a living are much more likely to make the wrong decision when information about risk is presented as probabilities or percentages, rather than as natural frequencies.
So let’s read about painkillers and heart attacks, another front-page story this month. It was a study over four years, and it suggested, using natural frequencies, that you would expect one extra heart attack for every 1,005 people taking ibuprofen. Or as the Daily Mail, in an article titled “How pills for your headache could kill”, reported: “British research revealed that patients taking ibuprofen to treat arthritis face a 24% increased risk of suffering a heart attack.”
Almost everyone reported the percentages: diclofenac increases the risk of heart attack by 55%, ibuprofen by 24%. Only the Daily Telegraph and the Evening Standard reported the natural frequencies, one extra heart attack in 1,005 people on ibuprofen. The Mirror, for example, reported that one in 1,005 people on ibuprofen “will suffer heart failure over the following year”. Several other papers repeated the mistake. No. It’s heart attacks, not heart failure, and it’s one extra person in 1,005, over the heart attacks you’d get anyway.
I could be a lot more forgiving if I believed that a nefarious, knowing, numerate media was choosing to report the higher, scarier percentage figures, to mislead and titillate an innumerate public. Actually, I think that they just don’t understand what they are reporting.
So if anyone is listening, this is the information I want from a newspaper, to help me make decisions about my health: I want to know who you’re talking about (eg men in their 50s); I want to know what the baseline risk is (eg four out of 100 will have a heart attack over 10 years); and I want to know what the increase in risk is, as a natural frequency (two extra men out of that 100 will have a heart attack over 10 years); and I want to know exactly what’s causing that increase in risk – an occasional headache pill or daily pain relief for arthritis. Health journalists are perfectly well paid, and the ones I know get paid more than the NHS pays me; it’s not too much to ask.
· Ben Goldacre is a medical doctor and writes the Bad Science column in the Guardian
Adrian Gaylard said,
August 26, 2005 at 12:01 pm
Excellent analysis, with which I agree whole heartedly. I would couch the comment in slightly different terms: relative risk (“percentages”) and absolute risk (“natural frequency”). Extra data that I would like to see are the “numbers neaded to treat” (NNT) or the “numbers needed to harm” (NNH). These measures are useful in terms of public health policy and personal concern. They address the question of how many people do you need to medicate/intervene to help one, or how many people need to take something before one suffers harm.
Relative risks are an excellent way of making something seem very bad that is actually not really worth worrying about. It may even be that the worry is worse for you than the putative problem! Just to put it into perspective, a smoker in the UK increases their risk of lung cancer by around 2400%. Now that’s a level of relative risk worth worrying about!
Mick James said,
September 7, 2005 at 9:37 pm
I hate to carp but the extra risk of having a heart attack in your example is not 2% but 2 percentage points. The increase in risk is still 50%.
anonymous said,
September 8, 2005 at 3:18 pm
You may be interested in this article from Science magazine.
——–
Statistics: What Seems Natural?
(Science, Vol 292, Issue 5518, 853-855 , 4 May 2001)
Which statistical data seem easier to understand, 10 cases in 100, or 10%? In their Policy Forum “Communicating statistical information” (Science’s Compass, 22 Dec., p. 2261), U. Hoffrage and colleagues offer persuasive evidence that both experts and novices find it to be the former. When prevalence, sensitivity, and false positive rates are given as probabilities (e.g., 10%), most physicians misinterpret the information in a way that could be potentially disastrous for the patient, but when they are presented as “natural frequencies” (e.g., 10 cases in 100), the physicians’ performance is dramatically better. The authors suggest ways to improve both communication of statistical information and medical education by using frequencies rather than probabilities.
The discussion by Hoffrage et al. leaves open the question as to why this is the case. Elsewhere, Gigerenzer and Hoffrage suggest that “humans seem to be developmentally and evolutionarily prepared to handle natural frequencies” (1, p. 430) by accumulating examples of the category in question. However, this would not, in itself, explain why this accumulation is preferentially represented as frequencies rather than being transformed into some other representation, such as rate or probability.
Frequencies (e.g., 10 cases in 100) can be thought of as a subcollection (with a numerosity of 10) in a collection (with a numerosity of 100). I have suggested that we are born with a specialized capacity for representing collections and their numerosities (2). The evidence for this comes from a range of studies showing that infants, even in the first week of life, are sensitive to changes in the numerosity of a collection of visual objects (3) and that, at 6 months, they are able to form arithmetical expectations on the basis of adding an object to a collection or taking it away (4). The almost universal use of fingers as the representative collection in counting and arithmetic suggests that collections and numerosities form the basis of later representations also (2). This suggestion has been supported by recent brain-imaging evidence showing that key number areas are closely connected to the finger circuit in the intraparietal sulci (5).
Of course, the big developmental gap between the capacities of young children and the performance of adult decision-makers is typically filled by an education system that teaches children about collections and numerosities far more than about probability. It is thus plausible that educational practices are, in part, responsible for the biases Hoffrage et al. report. However, there is indirect evidence that probability concepts are intrinsically difficult for humans. Although the computational techniques required by probabilities of the type described by the authors would have been available to the ancient Greeks, an understanding of the concepts began only with Girolamo Cardano’s Liber de ludo aleae (1525, published in 1663) and in the correspondence between Pascal and Fermat about games of chance in 1654.
Brian Butterworth
Institute of Cognitive Neuroscience,
University College London,
Alexandra House,
17 Queen Square,
London WC1N 3AR, UK;
e-mail: b.butterworth @ucl.ac.uk
References and Notes
G. Gigerenzer, U. Hoffrage, Psychol. Rev. 106, 425 (1999).
B. Butterworth, What Counts: How Every Brain Is Hardwired for Math (Free Press, New York, 1999) [in the UK, The Mathematical Brain (Macmillan, London, 1999)].
S. E. Antell, D. P. Keating, Child Devel. 54, 695 (1983); P. Starkey, R. G. Cooper Jr., Science 210, 1033 (1980); E. Van Loosbroek, A. W. Smitsman, Devel. Psychol. 26, 916 (1990).
K. Wynn, Nature 358, 749 (1992).
S. Dehaene, E. Spelke, P. Pinel, R. Stanescu, S. Tsivkin, Science 284, 970 (1999); B. Butterworth, Science 284, 928 (1999).
Trackback: That Science Coverage We All Hate | Cosmic Variance said,
September 8, 2005 at 10:54 pm
[…] He makes several observations that I’ve made in the past, and that I also constantly rant on about at dinner parties (which might explain why I have not been invited to any for a while), and so I’ll tease you with some extracts, which will hopefully encourage you to go and read the whole article, and then come back here and chat with us about it, ok? It is my hypothesis that in their choice of stories, and the way they cover them, the media create a parody of science, for their own means. They then attack this parody as if they were critiquing science. Science stories usually fall into three families: wacky stories, scare stories and “breakthrough” stories. Last year the Independent ran a wacky science story that generated an actual editorial: how many science stories get the lead editorial? It was on research by Dr Kevin Warwick, purporting to show that watching Richard and Judy improved IQ test performance (www.badscience.net/?p=84). Needless to say it was unpublished data, and highly questionable. Wacky stories don’t end there. They never end. Infidelity is genetic, say scientists. Electricity allergy real, says researcher. I’ve been collecting “scientists have found the formula for” stories since last summer, carefully pinning them into glass specimen cases, in preparation for my debut paper on the subject. So far I have captured the formulae for: the perfect way to eat ice cream […] the perfect TV sitcom […], the perfect boiled egg, love, the perfect joke, the most depressing day of the year […], and so many more. A close relative of the wacky story is the paradoxical health story. Every Christmas and Easter, regular as clockwork, you can read that chocolate is good for you (www.badscience.net/?p=67), just like red wine is, and with the same monotonous regularity…. These stories serve one purpose: they promote the reassuring idea that sensible health advice is outmoded and moralising, and that research on it is paradoxical and unreliable. At the other end of the spectrum, scare stories are – of course – a stalwart of media science. Based on minimal evidence and expanded with poor understanding of its significance, they help perform the most crucial function for the media, which is selling you, the reader, to their advertisers. […]
DensityDuck said,
September 12, 2005 at 3:32 pm
Mick James: Yes, see, you’re doing exactly the wrong thing, which is exactly what the author is railing about. “The number changed by 50% of its value” is not the same thing as “50% increase in risk”.
In this context, “risk” refers to the probability of an event.(*) If you have 100 men, and four have heart attacks, that’s a four percent chance of a heart attack. Now you “increase the risk by fifty percent”. That’s a six percent chance of a heart attack. Not a fifty-four percent chance.
The problem is that statistical data is meaningless when you only present one number. You present a percentage? Well, what’s the actual number? And conversely, just tossing out a number is useless without background. “It only caused two more heart attacks” is more meaningful if your sample size is ten people…
That said, the press is mostly an industry that’s based on the dissemination and reinforcement of urban legends. “Study: after smoking ban, heart attacks dropped from 14 in 125.000 to 8 in 125.000” is not scary. Saying “smokers have 50% higher heart attack risk, says doctor” is scary, which is what all urban legends need to be in order to propagate themselves.
Ben Goldacre said,
September 12, 2005 at 3:53 pm
Mick James: you’re wrong. The extra risk, or to use a more technical term, the “absolute risk increase” of having a heart attack in my example is indeed 2%. Using the phrase “percentage points” in certain specific situations to refer to percentages is a convention i’m not familiar with, and one I suspect you may have just made up.
Henning Schomerus said,
September 13, 2005 at 2:55 pm
Mick: In Ben’s phrase “the extra risk of having a heart attack if you have high cholesterol is only 2%” the “extra” relates the 2% to the total sample size, not the original risk (expressed in percentages) itself. Well, of course the “extra” could also just indicate that the probability increases (compared to “reduced risk”),
but I had to think hard to misrepresent the phrase in this way.
But bear with me:
Ben: While I find it hard to misunderstand the phrase in the way Mick did, the use of percentage points is still an accurate and well established terminology (en.wikipedia.org/wiki/Percentage_point). It is very frequently used, indeed (>7 Mio google hits), and it regularly comes across in articles about poll data. Certainly, pollsters don’t like to read in every other sentence that their sample size is just about 1000, typically – giving you a margin of sampling error of plus or minus three percentage points. Of course, you could say polls are not science (they can be input for science), but at the end it’s just statistics, and the Wikipedia entry indeed presents the term in a medical-risk context.
DensityDuck: In adressing Mick’s point, you say
> Now you “increase the risk by fifty percentâ€. That’s a six percent chance of a heart
> attack. Not a fifty-four percent chance.
The concern, of course, is not to end up with a fifty-four percent chance.
It is about ending up with a 4.08% chance (4*1.02 %), which is a bit too tiny to be newsworthy. That’s what you get when you (wrongly) relate the 2% in Ben’s phrase to the 4% risk in your example, not to the 100% of total sample size.
For me, all of this underscores Ben’s main point: probabilities are confusing, while natural frequencies are readily comprehensible, especially for communicating medical risks.
Ben Goldacre said,
September 13, 2005 at 3:21 pm
Hi Henning. I think you’re right, this all goes to illustrate the confusion of not using natural frequencies. Even the first line of that wikipedia entry is ambiguous: “Percentage points are the proper unit for the difference of two percentages.” Do they mean relative difference, or absolute difference? It’s the same ambiguity as the 50% or 2% increase. Using universal terminology like ARI or RRI (absolute risk increase/relative risk increase), or natural frequencies, is the only way, and only the latter requires no prior knowledge of stats terminology.
miasmic said,
September 13, 2005 at 4:57 pm
From what I understand, “percentage points” is a term commonly used in North America, but not, or less so in the UK. Certainly while I was living in Canada it was a term frequently used in the news media.
So I would expect that both conventions are correct, and this is a case of transatlantic misunderstanding.
Ben Goldacre said,
September 13, 2005 at 5:00 pm
Sure, but what’s the use of a convention that’s not universal? Especially when there’s a way of communicating risk that doesn’t rely on the ambiguous convention?
Jim Mack said,
September 16, 2005 at 4:45 pm
It never occurred to me that there could be a misunderstanding in the use of the term “percentage points” based on one’s location. Here in the US, it simply means the absolute difference between two percentage values, and it’s a useful way to talk about such differences.
For example, if there had been 4 cases of X in a population of 100, absent factor Y, but in the presence of Y the number of cases was 6, we would say that Y is associated either with a 50% increase in the number of cases, or an increase of 2 percentage points in their incidence. In no case would we say that there had been a 2% increase.
Very interesting, and it will be my new thing learned for today…
Magda said,
October 30, 2005 at 10:34 pm
‘Percentage points’ is certainly used in market/opinion research (and possibly in other statistically heavy social science disciplines) to avoid confusion between relative and absolute changes. I have learned this usage working as a pollster (with background in experimental psychology) in Poland and I found the differentiation very useful in avoiding confusion, along the lines of what Jim Mack said in his last comment. Though using natural frequencies would be perhaps even better.
Considering how bad human beings are in taking into account base rates when performing (or trying to) Bayesian reasoning, it’s remarkable that we seem to be much better at it when problems are presented in terms of natural frequencies rather than percentages, bit similar to the way we cope much better with a logical task of deciding how to test T/F of implication when it’s presented as a cheater-finding social problem rather than anything else.
Polina said,
February 25, 2006 at 5:59 pm
WOW ! very informative …and i like those pics of ur nephew , so cute !! can see that u put a lot of effort in it so
KEEP IT UP !
Jeniffer said,
February 27, 2006 at 2:37 pm
A great site where one can enjoy the thought of a great mind long departed. Cheers for the good work!
Mick James said,
February 28, 2006 at 12:50 pm
Ben I’m frankly amazed not only that is ‘the phrase “percentage points†… a convention i’m not familiar with’ but that you still find it ambiguous and confusing. This isn’t complex statistical jargon, but basic English.
Your financial colleagues could enlighten you here because they regularly have to talk about changes in indices which are themselves expressed in percentages. Far from being an “ambiguous convention” percentage points are precisely the way to avoid confusing “half as much again” with “a fiftieth more” as you did in your original piece. Suppose the Bank of England increased base rates from 4% to 6%. Is “interest rates up by 50%” a scaremongering headline? Would anyone really think that interest rates were now 54%. And would your mortgage payments go up by 2% or by a lot more?
I agree that it can seem like scaremongering if the papers talk about a risk “doubling” when its gone from one tiny figure to another–say from 0.0001% to 0.0002%. But it’s also perfectly accurate and meaningful: if that proportion represents all the people who are going to need a special drug, then you’ll need twice as much. To describe it as a 0.0001% increase in risk is not “another way of putting it”, it’s just wrong. This is not a matter of “convention”.
Use the “universal terminology” of ARI and RRI if you like , but if you do talk about percentages you really must get the percentage increase/percentage point increase right to avoid seriously misleading people.
Oh, and HG Wells was right.
Alex said,
March 23, 2006 at 6:16 am
Is more tremendous! I yet did not meet on the Internet of a drain of games and the information about texas holdem game! Very much it was pleasant. Thanks founders for a good site
Alex said,
April 2, 2006 at 2:11 am
Hi,
Everyone, who loves football, I invite on the site:
www.wmsever-2005.be
vacuumen said,
June 6, 2006 at 11:30 pm
Wells born in 1866, otherwise excellent.
wayscj said,
November 21, 2009 at 8:24 am
ed hardy ed hardy
ed hardy clothing ed hardy clothing
ed hardy shop ed hardy shop
christian audigier christian audigier
ed hardy cheap ed hardy cheap
ed hardy outlet ed hardy outlet
ed hardy sale ed hardy sale
ed hardy store ed hardy store
ed hardy mens ed hardy mens
ed hardy womens ed hardy womens
ed hardy kids ed hardy kids ed hardy kids
iphone revolution said,
December 30, 2009 at 8:33 am
iphone wireless
iphone
Apple iphone