You are 80% less likely to die from a meteor landing on your head if you wear a bicycle helmet all day.
We’re all suckers for a big number, and you’ll be delighted to hear that the Journal of Consumer Research has huge teams of scientists all eagerly writing up their sinister research on how to exploit us.
One excellent study this month looked at how people choose a digital camera. This will become relevant in three paragraphs’ time. The researchers took a single image, then processed it in Photoshop to make two copies: one where the colours were more vivid, and one where the image was sharper. They told participants that each image came from a different camera, and asked which they wanted to buy. About a quarter chose the one with the more colourful sharper image.
Then the researchers started to pile it on. Firstly they said that this camera had more pixels, using a figure derived from the diagonal width of the sensor: suddenly more than half picked it instead. Then, crucially, they told them that this camera had more pixels, but this time, they used the number of pixels as evidence: a figure measured, as you know, in millions. Suddenly, three quarters chose the supposedly better camera. Just a bigger number. Nothing more.
This week you’ll have noticed the news on rosuvastatin (or Crestor, since either through ignorance, or corporate whoredom, the media love to help drug companies by using their corporate brand names instead of the generic). The JUPITER trial on rosuvastatin has just reported, several months early, and most papers called it a “wonder drug”. The Express, bless them, thought it was an entirely new drug.
“Heart attacks were cut by 54 per cent, strokes by 48 per cent and the need for angioplasty or bypass by 46 per cent among the group on Crestor compared to those taking a placebo or dummy pill”, said the Daily Mail. Dramatic stuff. And in the Guardian, we said: “Researchers found that in the group taking the drug, heart attack risk was down by 54% and stroke by 48%”.
Is this true? Yes. Those are the figures on risk, expressed as something called the “Relative Risk Reduction“. It is the biggest possible number for expressing the change in risk. But 54% lower than what? This was a trial looking at whether it is worth taking a statin if you are at low risk of a heart attack (or a stroke), as a preventive measure: it is a huge market – normal people – but these are also people whose baseline risk is already very low.
If you express the exact same risks from the same trial as an “Absolute Risk Reduction“, suddenly they look a bit less exciting. On placebo, your risk of a heart attack in the trial was 0.37 events per 100 person years, and if you were taking rosuvastatin, it fell to 0.17 events per 100 person years. 0.37 to 0.17. Woohoo. And you have to take a pill every day. And it might have side effects.
And if you express the risk as “Numbers Needed To Treat“, probably the most intuitive and concrete way of expressing a benefit from an intervention, then I reckon, from the back of this envelope in front of me (they naughtily don’t even give the figure in the research paper), that a couple of hundred people need to take the pill to save one life.
Is it a good idea for you personally to take rosuvastatin? That’s not my job here - get over yourself, we’re allowed to talk about ideas - but the way figures are presented can have a huge impact on decisions everyone makes, and this is not idle speculation. In fact the phenomenon has been carefully studied, in many groups, and for many years.
In 1993 Malenka et al recruited 470 patients in a waiting room, and gave them details of a hypothetical disease, and a choice of two hypothetical treatments. In fact it was the same treatment, with the risk expressed in two different ways. 56.8% chose the medication whose benefit was expressed as a relative risk reduction, while only 14.7% chose the medication whose benefit was in absolute terms (15.5% were indifferent).
Are patients uniquely stupid? Joy, no. In fact the exact same result has been found repeatedly in experiments looking at doctors’ prescribing decisions, and even the purchasing decisions of health authorities.
We all love big numbers, and we’re all fooled by big numbers, because we’re all idiots. That’s why it’s important to think clearly, and ignore all newspapers.
Picklish said,
November 15, 2008 at 8:05 am
1 chance of a heart attack every 300 years you live versus 1 chance of a heart attack every 600 years you live??? but…are we talking older people, or those with high fatty diets? Or living in Scotland like me?
sure, the figures make you expect more of a difference than there actually is, in real terms, but there is still a difference, no?
but then, are you claiming that it is statistically significant? using a formula devised by…..?
when it comes to a whole population, is it not impossible to study every variable, so one has to take ‘educated guesses’?
i don’t know, these are just questions, i just enjoy this internet-response-board-thing to help me learn and understand
small increments in recovery/prevention/success rates? are they not important too?
Dr* T said,
November 15, 2008 at 9:11 am
From Science-Based Medicine:
“They estimated that 95 people would have to take rosuvastatin over 2 years or 25 people would have to take it over 5 years to prevent one new major cardiovascular event.”
‘Sceptical Rogue’ wrote a similar piece Thinking is Dangerous.
humber said,
November 15, 2008 at 10:34 am
Picklish,
At the risk of talking about statistics, “cuts risk by 50%” is meaningless unless you know the background risk.
Expressing the same as a ratio of deaths per year, gives some idea of the improvement offered over doing nothing, and a means of comparing that risk with others that are so often ignored.
In this case, the benefit must also be weighed against the long term use of the drug. Rather difficult to asses a risk that may take a lifetime to collate.
Cyclists often imagine that helmets are useful because they think only of head injuries, but in reality, you don’t get to choose the way you would be injured. A case of selective risk assessment.
jodyaberdein said,
November 15, 2008 at 10:39 am
Having yet to actually read the paper (it is early on Saturday after all), I’d be inclined to agree that the first point is pretty important. Often trials are done on fairly unrealistic patient groups. Presumably they excluded those with heart disease but otherwise took all comers.
I think the point is that there is a genuine difference, and that difference is being oversold by relying on inherent weaknesses in how people react to evidence.
Regarding populations and educated guesses however: the beauty of randomisation is that it evenly spreads out all the differences that you can’t even begin to guess about across both groups you are studying, so they no longer matter. Cool eh?
Jody
muscleman said,
November 15, 2008 at 11:07 am
I wondered on hearing about this research what the figures would be if in a separate arm of the study the people exercised, lost weight assessed by hip/waist ratio and improved their diets?
Sure there are side effects, most of them beneficial. Or you can take a pill a day for the rest of your life to give you a false sense of invulnerability. We know that safety measures in cars have not had the expected reductions in road deaths and injuries since it makes people feel safer and so they take more risks. So then how do we know that people on rosuvastatin will not, if they know they are on it, compensate by not exercising and/or saying yes to that pizza or fish supper?
drunkenoaf said,
November 15, 2008 at 11:27 am
Wait till you see one risk reduction value in a paper, then the p-value for the other. And a tiny asterisk directing you to a footnote to explain…
polly said,
November 15, 2008 at 11:59 am
‘ignore all newspapers’
Since (with a few honorable exceptions) they’re filled with lazy opinion by non-experts (not just science - but most subjects which deserve more than 10 minutes thought) sounds good to me.
briantist said,
November 15, 2008 at 12:15 pm
Did you hear this?
http://www.bbc.co.uk/iplayer/search/?q=feedback
Seems that “Dr Ben is working for the government!”
Also, did you see this one, the £15,000 fine for “potentially dangerous claims regarding the ability of his homeopathic treatments to cure cancer and other serious illnesses, such as diabetes and hepatitis”?
http://www.ofcom.org.uk/tv/obb/prog_cb/obb121/
briantist said,
November 15, 2008 at 12:50 pm
Also, any chance you can fix the date and times of postings back to GMT now BST is over? It’s 10:50am now but the clock on badscience.net seems to have gone the wrong way…
BryanKitts said,
November 15, 2008 at 2:25 pm
I’m intrigued by the headline statistic. Given the low absolute risk, the sample size would have to be huge. How do you find that many people who’d be willing to spend their lives wearing bicycle helmets? Or did the researchers just use meteor death statistics for groups that would naturally be wearing bicycle helmets all day: bicycle couriers, Tour de France competitors etc?
Sili said,
November 15, 2008 at 3:10 pm
Well, since dr Goldacre is an NHS doctor, duh! he works for the government!
Really. Some people.
twaza said,
November 15, 2008 at 4:59 pm
I calculated the NNTb for deaths from any cause, and used the data in the paper’s Table 3. There were 198/8901 deaths in the rosuvastatin group and 247/8901 deaths in the placebo group. The NNTb = 182 (95% CI 99 to 1088).
I am not sure what period the NNT relates to (the median follow up of 1.9 years?)
Whatever, the 95% confidence interval is so wide that one can’t be very confident in the NNT.
Tina Russell said,
November 15, 2008 at 8:58 pm
Oooh! You shoulda seen the Colbert Report on Thursday:
http://www.colbertnation.com/the-colbert-report-videos/210357/november-12-2008/cheating-death—women-s-health
At about 2:10 is when he talks about rosuvastatin and comes to about the same conclusion.
gazza said,
November 15, 2008 at 10:16 pm
By a bizarre coincidence my GP has asked me to take part in a trial looking at the effects of a low aspirin dose (1/6 of a tablet - more details to follow) on future incidences of stroke and heart attack. I would put myself in the relatively fit, ‘middle aged’ category. What’s the betting that this trial will end up the same as the rosuvastatin study that Ben refers to?
I surprised my GP at discussing with him how the outcome might be presented in terms of relative and absolute risk statistics!
From reading Ben’s column and book I look forward to discussing the marvels of the placebo effect with him, as well as considering trial protocols. Should be fun!
julie oakley said,
November 16, 2008 at 4:58 pm
As an older mother one of the kinds of statistics that I think is abused by the health service is the risk of having a Downs Syndrome child. Mothers are always informed that they will have a 1 in whatever chance of having a child with Downs Syndrome (in my case I think it was one in ten) which sounds pretty worrying to the average prospective parent. However they are never informed of the converse (ie in my case a 90% chance of not having a Downs Syndrome child)
mdimmick said,
November 17, 2008 at 12:23 pm
I thought you’d read “Flat Earth News”, Ben!
A large proportion of the ‘news’ is simply press releases - sometimes paraphrased or rewritten, but largely unresearched - and this is no exception. I’ve no doubt that the press release had these ‘facts’ in it, and of course the aim of the release is to promote their drug so they want to make the effectiveness look as large as possible. This is particularly the case where they’re proposing that people self-medicate with this drug every day.
njd said,
November 17, 2008 at 1:15 pm
90% fat-free sounds better than 10% fat.
That Uni. of Chicago study is behind a paywall, but one of the authors has it available at http://faculty.chicagogsb.edu/christopher.hsee/vita/Papers/SpecificationSeeking.pdf
Dr* T said,
November 17, 2008 at 6:27 pm
Mdimmick - true, but this is column is about statistics 101, any actual journalist (rather than just an admin-type PR rehasher) should be able to put a note in about absolute vs relative risk. It would take all of 5 mins.
beast9 said,
November 18, 2008 at 1:42 am
Is the post title about bike helmets a real statistic?
Ms Imelda said,
November 18, 2008 at 12:48 pm
humber wrote:
“Cyclists often imagine that helmets are useful because they think only of head injuries, but in reality, you don’t get to choose the way you would be injured. A case of selective risk assessment.”
But selective by whom? When making these judgments people rely on those we expect to know the overall risks such as A & E departments or organisations which compile accident statistics. It’s not just “imagination”.
heavens said,
November 19, 2008 at 12:18 am
Dr*T,
You seem to have forgotten people don’t go into journalism because they’re able to understand statistics. If they were good with math or science, they’d have chosen a different career.
Picklish said,
November 19, 2008 at 12:11 pm
Yeah, i agree heavens. Not even everyone who works in a scientific field understands statistics well. Although, it’s not necessary to understand how concepts such as multiple regression and chi squared and other exciting sounding statistical tricks work, but rather how and when to use them, which is completely different (and much easier). But as Dr G. says, we’re all idiots, and we should definately ignore all newspapers…
mikey baby said,
November 19, 2008 at 6:28 pm
Did anyone notice the baseline risk of the placebo group?
More men +0.6%
More smokers +0.3%
More metabolic syndrome +0.8%
Family history of premature CHD +0.6%
AZ were clearly paying attention when you lectured them on tricks big pharma employ.
This could explain the greater than expected benefit…..
cellocgw said,
November 19, 2008 at 8:37 pm
Re bicycle helmet:
You don’t get to choose your injury, but you do get to analyze the cost-benefit ratio.
A broken leg or collarbone (both common-ish in bike crashes) are painful but generally noncrippling.
It only takes a minor cranial injury to mess you up, for life, in ugly ways.
Dan Kimberg said,
November 22, 2008 at 9:51 am
Ben, in parts of this column you’re doing exactly the same thing as the people you’re criticizing. Recasting the 54% figure as 0.37 vs. 0.17 indeed makes it seem like a ridiculously tiny difference to people who aren’t used to working with numbers. You can’t look at a difference of 0.2 and call it small any more than you can look at a 54% reduction and call it large. In fact, the difference between 0.37 and 0.17 seems like a very enviable effect size to me, and could easily be recast as percentages that would seem so to anyone.
Appealing to statistics like “numbers needed to treat” is potentially very misleading as well. Both heart attack and stroke often have very severe consequences for quality of life in survivors. But certainly “saving” one life in 100 is a proportion that is either big or small, depending on the costs (money and side effects) of the treatment to the other 199 folks living on your envelope.
Anyway, playing a little fast and loose, let’s imagine that your risk of heart attack in your lifetime is 30% on placebo, 15% on the drug, and that the same numbers apply to stroke. Let’s also fantasize that they’re independent. That would mean you have a 51% chance of experiencing either a stroke or a heart attack or both on placebo, a 28% chance on the drug. Is that a big enough effect to be meaningful to you? Ultimately the answer doesn’t depend on what statistic you use to express exactly the same data. It depends on whether the cost (money, side effects) is worth the benefit. Which is the part of this question you sidestepped.
At the end of the day, these statistics are somewhat hard to interpret. I’d rather die from stroke or heart attack than in a skydiving accident, because the latter would be more likely to afflict me earlier in life. But if there’s a drug that will reduce my risk of heart attack and stroke, it probably won’t cause a skydiving accident. Unless it encourages me to continue skydiving later in life. At any rate, it’s never a great idea to read too much into absolute values, even if they’re small. No one dies from highly likely events.