Just looked this up myself and saw that for some reason it never got posted on the blog, so here it is.
Ben Goldacre
The Guardian
Saturday 10th October 2008
What I like about Bad Science is that it’s a game the whole family can play. This month “Lloydspharmacy”, as Lloyds Pharmacy insist on being called, is trying to flog carbon monoxide detectors (for only £12.99). It is a noble calling, so it decided to follow industry protocol for getting its product and brand into the media: it produced a misleading set of superficially plausible survey figures to massage our prejudices, which journalists obediently copied and pasted out of the Lloyds press release email and into their word processors, to make a “news” article.
“More than a quarter (28%) of council tenants said their local authority landlord had not serviced their boiler in the last 12 months, according to the ICM Research poll for Lloyds Pharmacy,” the Evening Standard said. “It also found more than half of council properties (59%) do not have carbon monoxide (CO) monitors – the lowest penetration of any type of housing tenure.” After being picked up by the Press Association this frightening ICM data found its way into Metro, the Express and, more importantly, about 150 local newspapers.
But it was nonsense. Richard Stokoe from the Local Government Association press office got onto Lloyd’s award winning PR company, Citigate Dewe Rogerson, and after a struggle he got the survey data. He was kind enough to send it on to me.
Let’s go through Lloyd’s press release together. “The UK’s biggest landlord is failing to protect vulnerable tenants according to a new ICM research released today from Lloydspharmacy. The research found that almost a third (28%) of council residents claim that their local authority landlord has not serviced their boiler in the last 12 months.” I have their figures here. The first thing to notice is that although everyone hates councils – so this is clearly a great way to get your story into the reactionary rightwing press – in this survey, councils are actually better at regularly servicing boilers than any other group in the country, including homeowners, private landlords and housing associations.
But what about the numbers, and the 28% who “claim that their local authority has not serviced their boiler in the last 12 months”? Seventy-two per cent said their boiler had been serviced in the past year; 14% said their boiler had been serviced, but at some time between now and “more than 10 years ago”, which is to say “ever” (and 10% were serviced in the last two years, making 82% out of the 86% who actually knew when their boiler was last serviced). Meanwhile 7% said “not applicable/not serviced” (which sounds a tiny bit like it might often be “I don’t know” or maybe even “er, my boiler is electric”) and 7% said “don’t know”.
It seems Citigate Dewe Robertson decided to parcel all these figures up with the 14% of people who actively said (or “claimed”) their boiler hadn’t been serviced in the past year in order to create its much more impressive figure of 28% (or “almost a third”). I contacted Citigate and it clarified that this was its error, and not ICM’s. We might want to be a bit speculative about using these figures at all. It told the Press Association that 1,003 people were surveyed, but only 99 of those were council tenants, which is 0.000022% of the 4.5 million people who live in council housing. This will become amusingly relevant.
The press release goes on. “Furthermore, an estimated 59% of these properties are not fitted with a CO monitor – the lowest penetration of any housing tenure.” This figure was reported everywhere. Come into my anal world and see once again how this is simply untrue. By its own figures, 84% of people who live “rent free” have no CO monitor fitted, and 62% of people who “rent from housing association” have no CO monitor fitted. So council tenants do not have the lowest penetration of CO monitors of any housing tenure (and it failed to mention that 17% of council homes probably won’t need a CO monitor, because they only use electricity, and have no gas boiler).
Citigate explains that it ignored those figures because the number of respondents for each of “rent free” and “rent from housing association” was below its cut-off. It has decided this is 100 people. It won’t tell me how it decided this, except to say it spoke to three experts. It won’t tell me who. However I note with some pleasure that the number of respondents in the category “council tenant” is also beneath its arbitrary cut-off, and it happily used that data.
It’s not exactly good information. It’s not exactly “news”. Carbon monoxide poisoning is pretty scary: it’s hard to spot the symptoms, and you can’t smell the gas; a lot of people probably aren’t careful enough about it; and a lot of councils probably are pretty rubbish to their tenants. But that doesn’t mean you can just massage some pleasing figures and bung them in 150 newspapers. There was a time – before they started flogging homeopathy sugar pills, perhaps – when people thought they could rely on what they were told by large respected pharmacies.
Please send your bad science to bad.science@guardian.co.uk
James G said,
March 17, 2009 at 5:12 pm
Come to think of it, I have no idea when our boiler was last serviced, and the carbon monoxide detector was one of those little spots, and was long past its sell by date. I really should get a new one, but I’ll specifically try and avoid Lloyds.
Oh, and privately rented accommodation for what its worth.
Kates48 said,
March 17, 2009 at 5:43 pm
I am so glad you are here to do all the sums – saves me the trouble as I’m crap at maths, like so many other people. And that of course is what naughty bad science people rely on…..
hat_eater said,
March 17, 2009 at 6:33 pm
Im sure I read this already, and I’m pretty sure I read it here.
But it is nevertheless a delicious dissection of a quite well camouflaged bad science specimen. If I may say so.
The Biologista said,
March 17, 2009 at 6:45 pm
Pretty sure I read this one online too…
mchem said,
March 17, 2009 at 8:00 pm
There’s a link to it from the “The Year in Bad Science” post, which points to the article on the guardian website.
www.badscience.net/2008/12/the-year-in-bad-science-2/
End of the third paragraph.
TimW said,
March 17, 2009 at 9:18 pm
It just goes to show you can’t be too careful.
mrsniddynoddy said,
March 17, 2009 at 9:36 pm
Hi – just saw you at the Babbage – great talk, thank you – v accessible – now to continue trying to persuade my students that statistics is interesting and valuable.
Toenex said,
March 17, 2009 at 11:10 pm
@Kates48 Somehow it has become socially acceptable to claim being inadequate at maths in exactly the same way it isn’t acceptable to claim an inability to point up at the sky.
Dr Jim said,
March 18, 2009 at 1:12 am
I spent over an hour with a student the other day trying to explain that whilst his experimental numbers certainly could be fudged to fit what he wanted to say, wouldn’t it in fact be nicer if he just didn’t pre-judge the outcome and try and decide what the numbers actually showed.
I have my doubts that he really was interested in anything other than the nice little narrative he had planned. I will admit it sounded good, were it not speculative hand-waving, oh, and it being BAD SCIENCE. I will be sure to send him the email of the job desk at Lloyds Pharmacy.
Godliness said,
March 18, 2009 at 1:15 am
My boiler is located next to my kitchen sink. Whenever the small child that I keep tethered to the sink expires, I get the boiler serviced. Does this count as a carbon monoxide detector?
johnrs said,
March 18, 2009 at 3:09 am
@Godliness – only if you feed them.
Pete Beaudro said,
March 18, 2009 at 12:55 pm
Hey, Toenex
Given that we’re constantly being reminded (often by scientists) that sciences are much harder than other subjects, I’d suggest that ‘being crap at maths’ is not uncommon
As Kate48 points out, it is a fact frequently exploited by those people who are good at maths, but who use their skills to make maths ‘tell fibs’.
Adequate numeracy is a pretty moveable feast: I’m pretty numerate to a basic (GCSE) level, I’m happy to say (and my sky-pointing is outstanding), but it would take me a heck of a long time to do certain proportionate comparisons (ie practical statistics) these days.
The basic functionally numerate are the most succulent fodder for these dodgy PR sorts – they understand percentages, fractions etc. to a degree in certan contexts, but struggle to spot the flaws when the context changes.
Having said that, I don’t feel people should take pride in poor numeracy (and I don’t think you did, Kate48), particularly when they accompany this pride with snide remarks about other people’s use of the apostrophe. I’ve been that man – I am no longer!
mch said,
March 18, 2009 at 1:47 pm
Boilers need servicing?!
WigglePig said,
March 18, 2009 at 1:55 pm
What’s a boiler?
Synchronium said,
March 18, 2009 at 2:07 pm
WigglePig: It’s English for “woman”.
lasker said,
March 18, 2009 at 4:04 pm
Its easy to get statistics wrong but surely 99/4,500,000 =0.000022 so to get 99 as a percentage of the number of council tenants you would have to multiply by 100, giving 0.0022%.
OK Nitpicking, but hey scientific accuracy and all that…
mikewhit said,
March 18, 2009 at 6:30 pm
Adequate numeracy is a pretty moveable feast – what, like Easter ?
marcusavgreen said,
March 19, 2009 at 1:19 pm
I used this article as a teaching aid to illustrate how important it is to question statistics and the motivation of the people who produce them.
dizzydi said,
March 20, 2009 at 11:24 am
I temped in a council office that managed all the testing of boilers for Brighton & Hove’s council properties. Apart from the usual problems of using outside contractors, basically we did a pretty good job and it was a ‘Key Perfomance Indicator’ so very well monitored. The main reason we couldn’t get to 100% each year was that a small proportion of council tenants wouldn’t let our engineers into their home (and were often pretty abusive about it!).
Plato-says said,
March 20, 2009 at 4:27 pm
I’m sure I read this HERE – a few months back.
Can it have self-destructed?
Robert Carnegie said,
March 21, 2009 at 11:51 pm
Poundstretcher/Instore has a relatively cheap carbon monoxide alarm, probably equivalent to Lloyds if it’s any good, which I don’t know. It’s electronic and supposedly bleeps if CO exceeds a certain level and also bleeps in a different way when the battery is running out. My own CO spot is overdue for a change anyway, but it crosses my mind that arranging not to have a poison gas generating machine in your home in the first place is yet another approach.
If you heat by electricity, do you have a boiler?
elspeth2009 said,
March 22, 2009 at 11:42 pm
Hi,
I also saw you at the Babbage laboratory, I just wanted to let you know how much I appreciated what you said about risk assessment and psychaiatric patients. I think as many people as possible should say this as often as possible as loudly as possible.
They probably all went mental from carbon monoxide poisoning from private landlords anyway.
elspeth2009 said,
March 23, 2009 at 12:11 am
Oh, and I’ve never been under the care of a community mental health team but I have seen a few psychaiatrists (for anyone that hasn’t the procedure is, they flick through your notes, say, “tell me about your last episode of terrifying psychosis”, listen, ask “did you find you were eating less than usual at this time”, wonder whether this merits/ the NHS can afford a follow-up appointment)I’m amazed it’s only 5%.
essjaydee88 said,
March 27, 2009 at 1:56 pm
Sorry to be puerile, but look at the third line from the third-from-bottom paragraph. Really quite a hilarious turn of phrase. I’ve never been asked to do that by a blog before.
RexT said,
March 27, 2009 at 10:50 pm
The turn of phrase is apposite in view of the proported “low penetration rate”
achieved by councils. We are all supposed to be in favour of high pentration rates…especially by Lloyds.
peningda said,
April 3, 2009 at 6:12 am
I think they picked council tennants as their highlighted group because they are the group least able to act to get their furnaces serviced in response to the fear campaign, or even to be able to find out when the furnace was last serviced, and so most likely to be persuaded to buy Carbon Monoxide detectors.
dvavasour said,
April 12, 2009 at 8:26 pm
Just found this article.
The other thing which isn’t considered is how many of the properties have sealed boilers: that is, boilers where both the incoming air and outgoing combustion products go via a concentric dual direction flue straight to outside, and the boiler is totally sealed to inside the house.
The possibility of such a boiler leaking CO into the inside depends not only on the combustion process being misadjusted (relatively possible), but also on the seal between the boiler and inside the house being compromised (remote possibility).
Putting a CO detector next to a sealed boiler is a waste of money.
abdou said,
April 16, 2009 at 11:04 pm
Excellent stuff Ben, especially getting the phrase ‘Come into my anal world’ into the national press
wayscj said,
November 21, 2009 at 6:26 am
ed hardy ed hardy
ed hardy clothing ed hardy clothing
ed hardy shop ed hardy shop
christian audigier christian audigier
ed hardy cheap ed hardy cheap
ed hardy outlet ed hardy outlet
ed hardy sale ed hardy sale
ed hardy store ed hardy store
ed hardy mens ed hardy mens
ed hardy womens ed hardy womens
ed hardy kids ed hardy kids ed hardy kids