Just looked this up myself and saw that for some reason it never got posted on the blog, so here it is.
Saturday 10th October 2008
What I like about Bad Science is that it’s a game the whole family can play. This month “Lloydspharmacy”, as Lloyds Pharmacy insist on being called, is trying to flog carbon monoxide detectors (for only £12.99). It is a noble calling, so it decided to follow industry protocol for getting its product and brand into the media: it produced a misleading set of superficially plausible survey figures to massage our prejudices, which journalists obediently copied and pasted out of the Lloyds press release email and into their word processors, to make a “news” article.
“More than a quarter (28%) of council tenants said their local authority landlord had not serviced their boiler in the last 12 months, according to the ICM Research poll for Lloyds Pharmacy,” the Evening Standard said. “It also found more than half of council properties (59%) do not have carbon monoxide (CO) monitors – the lowest penetration of any type of housing tenure.” After being picked up by the Press Association this frightening ICM data found its way into Metro, the Express and, more importantly, about 150 local newspapers.
But it was nonsense. Richard Stokoe from the Local Government Association press office got onto Lloyd’s award winning PR company, Citigate Dewe Rogerson, and after a struggle he got the survey data. He was kind enough to send it on to me.
Let’s go through Lloyd’s press release together. “The UK’s biggest landlord is failing to protect vulnerable tenants according to a new ICM research released today from Lloydspharmacy. The research found that almost a third (28%) of council residents claim that their local authority landlord has not serviced their boiler in the last 12 months.” I have their figures here. The first thing to notice is that although everyone hates councils – so this is clearly a great way to get your story into the reactionary rightwing press – in this survey, councils are actually better at regularly servicing boilers than any other group in the country, including homeowners, private landlords and housing associations.
But what about the numbers, and the 28% who “claim that their local authority has not serviced their boiler in the last 12 months”? Seventy-two per cent said their boiler had been serviced in the past year; 14% said their boiler had been serviced, but at some time between now and “more than 10 years ago”, which is to say “ever” (and 10% were serviced in the last two years, making 82% out of the 86% who actually knew when their boiler was last serviced). Meanwhile 7% said “not applicable/not serviced” (which sounds a tiny bit like it might often be “I don’t know” or maybe even “er, my boiler is electric”) and 7% said “don’t know”.
It seems Citigate Dewe Robertson decided to parcel all these figures up with the 14% of people who actively said (or “claimed”) their boiler hadn’t been serviced in the past year in order to create its much more impressive figure of 28% (or “almost a third”). I contacted Citigate and it clarified that this was its error, and not ICM’s. We might want to be a bit speculative about using these figures at all. It told the Press Association that 1,003 people were surveyed, but only 99 of those were council tenants, which is 0.000022% of the 4.5 million people who live in council housing. This will become amusingly relevant.
The press release goes on. “Furthermore, an estimated 59% of these properties are not fitted with a CO monitor – the lowest penetration of any housing tenure.” This figure was reported everywhere. Come into my anal world and see once again how this is simply untrue. By its own figures, 84% of people who live “rent free” have no CO monitor fitted, and 62% of people who “rent from housing association” have no CO monitor fitted. So council tenants do not have the lowest penetration of CO monitors of any housing tenure (and it failed to mention that 17% of council homes probably won’t need a CO monitor, because they only use electricity, and have no gas boiler).
Citigate explains that it ignored those figures because the number of respondents for each of “rent free” and “rent from housing association” was below its cut-off. It has decided this is 100 people. It won’t tell me how it decided this, except to say it spoke to three experts. It won’t tell me who. However I note with some pleasure that the number of respondents in the category “council tenant” is also beneath its arbitrary cut-off, and it happily used that data.
It’s not exactly good information. It’s not exactly “news”. Carbon monoxide poisoning is pretty scary: it’s hard to spot the symptoms, and you can’t smell the gas; a lot of people probably aren’t careful enough about it; and a lot of councils probably are pretty rubbish to their tenants. But that doesn’t mean you can just massage some pleasing figures and bung them in 150 newspapers. There was a time – before they started flogging homeopathy sugar pills, perhaps – when people thought they could rely on what they were told by large respected pharmacies.
Please send your bad science to firstname.lastname@example.org