Putting a number in its context

January 8th, 2011 by Ben Goldacre in bad science, numerical context, statistics | 28 Comments »

Ben Goldacre, The Guardian, Saturday 8th January 2011

600 pregnancies despite contraceptive implant” said the BBC.  “500 fall pregnant after having contraceptive implant” said the Express. “Contraceptive implant alert” said the Daily Mail: “Hundreds of women fall pregnant after birth control fails”.

The story first broke on Channel 4, and it’s still not entirely clear why it’s the biggest medical story so far this year. Some women have had some compensation: but a lot of people get a lot of payouts. There’s a law firm touting for more business, but that’s hardly news either, and the news story was: this contraceptive device has failed.

But is the failure rate exceptional? A figure means nothing if it has no context: 600 pregnancies sounds like a big number, but there is no way to know what it means unless we know how many women had Implanon, and for how long. The device was first launched in 1999, so that makes 60 pregnancies a year, which feels like a smaller number, but that’s still not enough information. The figure that epidemiologists use for context is “person-years-at-risk”.

I contacted the MHRA. They estimate that 1.355 million Implanon implants have been sold.  If we assume that each implant lasted 3 years, which they do, this gives a total exposure time of 4.06 million women-years at risk. 584 unplanned pregnancies in this exposed population means there were 1.4 unwanted pregnancies reported for every 10,000 women with Implanon implants per year. If you prefer, you can say that the failure rate is 0.014% per year. This is rather good: in fact, implants are still the most reliable form of contraception.

Back with our 584 unwanted pregnancies, meanwhile, we see the difference between individuals and statistics. For some of the people who got pregnant, from their end of the telescope, this is a disaster. Some cases may well have been avoidable. Some will want financial compensation, and you’ll have your own views on the state’s role in this more broadly. But for you as a potential user of the implant, or a news editor, looking at the whole population, even though this particular variety of implant had a problem with insertion – which has already been improved on – at its worst, it still seems to be one of the most effective forms of contraception available.

Meanwhile there were 1,669 reports of suspected adverse events associated with the device, including arm scarring and other injuries. This is a rate of 4 adverse events for every 10,000 women per year, or if we assume that the implant lasts for 3 years, around 1 in every 1,000 insertions results in a suspected adverse event report.

Every medical intervention can have unintended side effects. It’s worth noting that for the past 5 years, patients have been able to report adverse events for any drug directly to the MHRA online, just like doctors have always done, and I encourage you to do so, as it provides an important resource for the fascinating problem of catching adverse reactions before they affect too many people.

Spotting a true signal of a problem in this database, incidentally, presents an awesome methodological challenge, because any one of thousands of drugs could cause any one of thousands of types of adverse reactions or symptoms. What you get, as a result, is a giant spreadsheet with a number in each cell, and inevitably, with such a vast number of cells, by the simple play of chance, there will also be a vast number of cells with an unusually high number of event reports in them, simply through random variation. Geniuses who worship the Reverend Bayes, with his statistical methods that exploit prior probability, have enjoyed scratching their heads over this resource for many decades.

And lastly, just like a number deserves its context, so too does a scare. In the 1990s, a temporary concern about a modestly increased risk of blood clot, particularly in one type of oral contraceptive pill, resulted in a mass abandonment of oral contraceptives generally, around the world, including among low risk women, and the following years saw an increase in both pregnancies and abortions, with all that this entails. Words can do harm, just as surely as hormones can.

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

28 Responses

  1. penglish said,

    January 8, 2011 at 12:18 am

    Spot on as usual, Ben; it was crap journalism by C4. Culpably bad.

    There’s an interesting point here about the difference between less than ideal use when the patient has control, and is responsible for the imperfect use (e.g. of pills, condoms…), assuming that they’ve been correctly advised; and the improper insertion of Implanon.

    But this point fades into insignificance compared with the fact that real world Implanon use was in a different league of efficacy compared to other reversible methods of contraception.

  2. mem_somerville said,

    January 8, 2011 at 12:37 am

    Of course, there’s an xkcd for this….

    xkcd.com/558/ 1000 Times

  3. mrjohnc said,

    January 8, 2011 at 1:25 am

    thanks ben

    when twitter overthrows the government i hope you get to do lovely old fashioned public service announcements

  4. briantist said,

    January 8, 2011 at 6:37 am

    Tim Harford got there first (well, yesterday) with this story – BBC – BBC Radio 4 Programmes – More or Less, 07/01/2011.

  5. Seahorse said,

    January 8, 2011 at 8:19 am

    Nicely put, as ever. To be only a bit anal, I suspect three years (life of each Implanon) may be a bit long for your calculation. They are meant to be removed after three years. Some must end up being around for longer than this, but can’t really be blamed for failing after their ‘take me out’ date. But a fair few get removed earlier, because of side-effects (irregular bleeding, mainly), or because women want to get pregnant, or for other reasons. My guess is that the mean life of an Implanon is a bit short of three years.

    But even if it’s two years that would still be an annual failure rate of 0.022%.

  6. Indy said,

    January 8, 2011 at 8:44 am

    Great piece, Ben.

    News of the ‘Epic Failure’ rate reached us here in New Zealand too, and again the reportiing was equally dodgy. The only other number provided in the articles I saw was 1.355 million, which of course immediately put the failure rate into perspective, but it was certinly not the conclusion made in the story!

  7. dherron said,

    January 8, 2011 at 9:36 am

    The 600 pregnancies story needs to be set in context but so does the 1.3 million implants. Many are removed after less than 3 years due to irregular bleeding, often they are removed before 6 months. The implanon is widely quoted as having a failure rate of <0.07 per hundred women years. Or: less than 7 unwanted pregnancies reported for every 10,000 women with Implanon implants per year.

  8. IanHSci said,

    January 8, 2011 at 9:36 am

    Nicely done as usual. It’s also worth noting that many of the complaints made (the 1669 quoted) were made by the professionals involved in the implantation process – hence the changes made to turn Implanon into Nexplanon.

  9. QuietKnoll said,

    January 8, 2011 at 11:56 am

    Your mention of the blood clot link to Oral Contaceptives and consequent increase in (and associated risks from) both births and abortions might be worth an article in its own right.

    Stephen Dorrell was the Sec of State for Health at the time and his defence then was that he had to pass on the advise he received from “experts” once he was aware of the facts.

    I expect more from political leaders and other lay people involved in scientific areas. They have a responsibility to question the experts and make sure that they themselves understand the context of any advise. It is far too easy for political “decision makers” to dodge responsibility for their actions in this way.

    Stephen Dorell was at fault by failing to understand the advice he was given and in a knee jerk reaction passed on responsibility instead of trying to interpret and understand the relative risks involved.

    Why is this still important? Well the same Stephen Dorell is now Chair of the Commons Health Select Committee – the body that should hold ministers and officials to account for health policy (evidence based or not).

  10. suzwriter said,

    January 8, 2011 at 12:02 pm

    Great piece on the statistics of this on Radio 4’s most excellent More or Less this week at www.bbc.co.uk/programmes/b00x44l1 – plus Nigel Pargetter’s death yodel too!

  11. geodoc said,

    January 8, 2011 at 1:58 pm

    Thank you Ben. Suspected as much, but unlike you couldn’t be arsed to contact the MHRA and ferret out the numbers. Shocking that Channel 4 journalists can’t or won’t carry out this simple step.

    The current flu stories in the media are almost as dire in terms of missing context. Quick look at the weekly influenza reports on the HPA website will reveal that currently, in E&W:
    -the incidence of GP consultations for influenza-like illness is half that required for threshold for an epidemic to be declared (94.4/100,000, vs threshold of 200);
    -the incidence peaked a week ago (though admittedly there have been ‘double peaks’ in previous flu seasons)
    -remains below the peak when swine flu hit last summer;
    -is not very significant historically- while higher than the baseline of recent years, it remains substantially below the 1999/2000 flu season and notably the 90/91 season (when >500);
    -has had a minimal impact of outbreak on all-cause mortality (barely above upper limit for expected deaths for this time of yr, see graph page 4 of report)

    Caveats: the disease severity/mortality figures a week or two behind, and the deaths are indeed more in younger age groups compared to previous years (perhaps the only really newsworthy thing here, which we already knew from experience with H1N1 last summer). Also no info on flu related ICU occupancy.


  12. dilleight said,

    January 8, 2011 at 2:31 pm

    Sorry but I’m not sure about this. As Seahorse says some people will have the implant taken out early and anecdotally it could be quite a few as some women find that it has side effects they don’t like and choose to remove it after 3- 6 months. I don’t know what proportion this is but neither do you. Also just because the implants have been sold that doesn’t mean they’ve been used. I agree with the general message that 584 failures isn’t necessarily a lot but I don’t think your figures are accurate enough either.

  13. Curlylox said,

    January 8, 2011 at 4:02 pm

    It frustrates me no end when the media makes these outrageous claims and then doesn’t print the statistical information to back them up (or the report they are based on).

    Apart from all the hoohar about this. Every woman should know that no contraception is 100% reliable, (excepting hysterectomy which can hardly be classed as contraception in any case), and whilst my sympathies go out to those who have fallen pregnant, if you really didn’t want kids, you should have crossed your legs.

  14. mikewhit said,

    January 8, 2011 at 6:48 pm

    However, there is a qualitative difference in the “failure rate” between for example, condoms, and female hormonal contraceptives, in that condom use pretty much implies intercourse, whereas being on Implanon does not imply any sex took place at all during the presence of the implant.

    OK yes, and condom use may have occurred during the partner’s infertile period …

    So if you are really interested in effectiveness as a contraceptive you should be recording number of “events” during fertile period.

    And I’d also be looking at “age” of implant at time of conception.

  15. naath said,

    January 8, 2011 at 10:58 pm

    Are they really counting “a small scar” as a “side effect” of note? (I have a small scar from my first implanon removal, and whilst I’m not ecstatic about that I’m not about to demand compensation over it). It’s worth noting that ALL hormonal contraceptives have some side effects in some women, and those side effects range can be seriously awful (like “feeling suicidal” or “constant, painful, cramping”) – a broader range of available contraceptives increases any given woman’s chance of finding one that doesn’t have negative side effects she can’t cope with.

    However I don’t think “sold” equals “used” (my GP has never had to order one in for me, so I assume they have some sitting around waiting for people to ask for them), nor does it equal “used for 3 years” since a)some people decide they want a baby now, please and b)some people experience bad side effects and have them out early on that basis.

  16. jack21222 said,

    January 9, 2011 at 6:47 pm

    Dr. Goldacre:

    This isn’t directly related, but I couldn’t figure out another way to contact you to express my thanks for writing Bad Science.

    I’m currently reading through the book (almost finished!), and the information in the book helped me see right through that recent psychology paper showing evidence for ESP, of which I’m sure you’ve heard. That paper uses almost all of the statistical errors that you outline in the “Bad Stats” chapter.

    Again, thanks for writing quality information.

  17. baratron said,

    January 9, 2011 at 10:26 pm

    Excellent report. Like you, I noticed the total lack of information about the number of Implanons used and the timescale, but unlike you, I didn’t have the energy to check it out. Thank you for doing so!

  18. apothecary said,

    January 10, 2011 at 9:05 am

    I heard the solicitor for (some of) the women making a group claim, being interviewed on R4 last week. She seemed quite sensible, actually – the point did not seem to be that correctly inserted Implanon devices had failed (she accepted that all contraceptives have a failure rate), but that some had not been inserted correctly. Now IANAL, but it seems to me that if someone has been fully informed that the failure rate of a correctly-administered Implanon is around 1 in 10,000 women years (or whatever it is, accepting the discussion above about Ben’s figures), she doesn’t really have a legal claim if she is that unfortunate one, however catastrophic for her that might be, and however much we might sympathise with her as a fellow human being. However, if someone accepts this risk and chooses an implant but the person adminstering it doesn’t do so properly and the device fails as a result, I should have thought she has a claim, just as much as if a surgeon botches an operation, which perhaps only becomes apparent later. I think the solicitor was claiming that in some cases the implant hadn’t actually been inserted at all. But why that should be such a big story I don’t know, unless editors and journalists can’t see that distinction.

  19. CampFreddie said,

    January 10, 2011 at 1:23 pm

    Great article. When I heard the story, the focus was on the ‘devastating’ impact of unplanned pregnancy on 600 people.

    At the time I thought that 600 would be pretty good, assuming >100,000 people used it. I remember being told that the pill was about 99% effective based on a year of ‘correct use’. I think condoms were about 98%. Yet we don’t get stories about the thousands of pregancies that occur to women on the pill or men who always used a condom.

    The only real story was about the complications in the implant/removal procedures, and the story on C4 reduced this to an anectodal interview. The fact that the implant/removal procedures seem to have been fixed wasn’t mentioned, nor was the likelihood of anyone having a complication.

  20. squitchtweak said,

    January 10, 2011 at 10:10 pm

    I thought these stories were a bit silly when reading them. No method of contraception is foolproof. That isn’t news.

  21. barefoothippy said,

    January 11, 2011 at 10:02 pm

    So the story here was that most of the failures were due to improperly placed implants. When you get your implant you’re supposed to have a good feel to check it’s there, go back to your gp/nurse a couple of months later so they can check it’s still there and all ok, and keep checking it occasionally to make sure it hasn’t moved. So these implants didn’t fail (as there wasn’t an implant *to* fail!) – the health professional placing them did. In other words: People Sometimes Make Mistakes… but that’s not as exciting as a paradigm-shifting something-that-was-good-is-now-bad-shock! story :S

  22. fontwell said,

    January 12, 2011 at 5:28 pm

    I’m glad to say, thanks to regular doses of Bad Science, I realised quite early on that this was most likely being reported is the most misleading way possible. Then one report gave it away by revealing the huge number of women that would “be worried” by this news.

    The other thing I get irritated by is when they say that with the pill, you still have a 1% chance of getting pregnant. For a start, this 1% is over a year and for another thing its per 100 women. So 1% means one woman in 100 gets pregnant per year. So actually the failure per shag is nothing like 1%. And it also seems a bit unfair that most pregnancies with the pill are due to failure to take it

  23. Paddy_Irish_Girl said,

    January 12, 2011 at 11:32 pm

    Ha…this made the papers in Australia too. In fact I was quite disappointed that my usual paper, normally rational and intelligent, ran with it without going into any detail on the numbers.
    God the media is so frustratingly gullible at times.

  24. 4tytwo said,

    January 14, 2011 at 9:13 pm

    interresting to actually do the statistic as failure per intercourse or per shag as fontwell so nicely said. This would really put the rankings in perspective. However most studies give overinflated numbers so perhaps Ben can advise were to get reality checked data from

  25. Sili said,

    January 21, 2011 at 10:52 pm

    Any chance you’ll suck Cox on stage?

    I’ll pay.

  26. Sili said,

    January 21, 2011 at 10:53 pm

    The Hell?!

    This is not where I logged in.

  27. mikewhit said,

    January 22, 2011 at 7:20 am

    @fontwell: Indeed.

  28. wordsbymt said,

    December 13, 2011 at 1:42 pm

    There was a new one on Sky news last night, talking about the devastating consequences of (and here’s a British favorite) inoculation.
    turns out they interviewed the parent of a child who had developed narcolepsy three months after receiving an inoculation for something or other (i was throwing things at the TV by then).
    they looked at studies in finland, too. the upshot was that there had been seven recorded cases of narcolepsy after six million inoculations.
    there was no mention of outside factors, or whatever other medicines people were taking at the time, nor was there a mention of what the rate of narcolepsy development was in the general public.
    more scaremongering rubbish?