Ben Goldacre, The Guardian, Saturday 12 February 2011
One thing you hope for, with politicians, is that they won’t make the same mistakes over and over again.
Last week we saw that the government has overstated the problems in the NHS by using dodgy figures (to be precise, they used misleading static figures instead of time trends). We saw that Andrew Lansley’s repeated claim that his reforms are justified by evidence was untrue: the evidence doesn’t show that his price-based competition improves outcomes (if anything it makes things worse); and the evidence also doesn’t show that GP consortia improve outcomes (unless you cherry pick only the positive findings). It’s okay if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.
Now Lansley’s junior minister Paul Burstow MP has kindly responded, repeating the exact same mistakes again, only more clumsily. I find this, in all seriousness, genuinely frightening from a minister, so I’ll explain how he does it.
The government initially claimed that UK heart attack death rates were twice as bad as France. This was an overstatement: they are, but following recent interventions the gap is closing so rapidly that on current trends it will have disappeared entirely by 2012. In response, Burstow cites a 2008 paper by McKee and Nolte which he says “concluded that the UK had one of the worst rates of mortality amenable to healthcare among rich nations”.
Burstow either misunderstands or misrepresents this very simple and brief paper. It is a study explicitly looking at time trends, not static figures, and it once again finds that comparing 2003 with 1998, the UK still had fairly high rates of avoidable mortality, but these were falling faster than in all but one of the other 18 industrialised countries they examined (meanwhile in the US, avoidable mortality improved at a disastrously slow pace, although they spent more money).
This is a paper showing the success of the NHS, and the fact that we are discussing such a massive improvement in avoidable mortality from Labour’s first term in government is not my choosing: this is the paper that was cited by the Tory minister as evidence, bizarrely, of the NHS’s recent failures.
Next Burstow says I “overlooked the impact assessment we published alongside the health and social care bill, where we present a thorough analysis of the evidence for and against our plans… studies show that GP fundholding and practice-based commissioning delivered shorter waits and fewer referrals to hospitals for patients.”
In its section on GP fundholding, this “thorough analysis” ignores the four peer-reviewed academic papers I described last week, which sadly found no evidence of an overall benefit from GP fundholding. It makes a series of 5 assertions about outcomes, though these are unreferenced to any paper at all.
I contacted the Department of Health, who ferreted out the sources: there was just one, a document from the King’s Fund. It’s not a peer-reviewed academic journal article, but the King’s Fund are pretty good, in my view. If you read this document, it too finds that the results of GP Fundholding were mixed: some things got better, some things got worse.
So the Minister has cherry picked only the good findings, from only one report, while ignoring the peer-reviewed literature. Most crucially, he cherry-picks findings he likes whilst explicitly claiming that he is fairly citing the totality of the evidence from a thorough analysis. I can produce good evidence that I have a magical two-headed coin, if I simply disregard all the throws where it comes out tails.
Here is what politicians apparently cannot understand: it’s fine to make policy based on ideology, whim, faith, principles, and all the other things we’re used to. It’s also fine for evidence to be mixed. And it’s absolutely fine if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.
irishaxeman said,
February 16, 2011 at 2:02 pm
emen said the introduction of competition improves services and lowers prices. British Rail anyone? Bus services anywhere in Britain anyone? Or do emen and others not actually use public services in the main parts of the UK? There is no evidence in the world in the last forty years that neoliberal freemarket competition – the ultimate in competition – did anything but impoverish the majority while lining the pockets of business and politicians.
Guy said,
February 17, 2011 at 9:47 am
I think we need to remember that Ben’s title is slightly ironic. Those of us who work in evidence based disciplines look for evidence to improve what we do. If fresh evidence comes up then we re-appraise and hopefully adapt what we do to fit the evidence.
Then there is politics. Politicians aren’t stupid and understand evidence perfectly well. They just aren’t terribly honest!
So yes there is no evidence. They know that. We know that. So they are just trying to hoodwink those who they can take in through this pretence. I largely blame the lazy media who allow them to get away with it.
So lets not pretend they are stupid or don’tknow what they are doing. They are driven by dogma and as we know that disease is immune to evidence.
I agree with Ben that if they were honest about that, this debate would be a lot simpler. Back in pre-Maggie NHS days they used to pilot big changes and actually wait for the answer with interest. Oh those were the days….
nathaniel said,
February 17, 2011 at 11:09 pm
There’s another reason why evidence is hard for politicians, though perhaps it’s not directly relevant here: if a politician changes her mind on the basis of evidence then she will be accused of “doing a U-turn,” which is seen as a sign of weakness. I despise that phrase and long for a day when a politician can say “I’ve changed my mind on the basis of new evidence” and have that be seen as the positive thing it obviously is.
msjhaffey said,
February 18, 2011 at 8:04 am
Someone identifying himself as Paul Epstein has contacted me about this blog entry and the comments. He is for an unknown reason (he is apparently registered) unable to post comments and has asked me to post the following two comments.
—
@msjhaffey
Interesting question about the strong linearity of the graphs. As Ben says, John Appleby’s article was an “instant” response to government claims, and indeed he seems to have simply copied the graphs from the original OECD source, without bothering to make additional interpretative comments and explanations (such as providing figures as well as graphs, and such as explaining why he bases his analysis on a 30 year trend rather than some other more relevant time period).
In other words, the BMJ author may know the answer to your question, but it’s also plausible that he does not know. Try asking on a newsgroup to people who are likely to know about the methodology by which the OECD collect statistics. Or ask the OECD for the figures on which the graph is based.
John Appleby’s data does seem to match the OECD data so your question is properly addressed to the OECD.
—
@Youdell,
I read the BMJ paper too. From a statistical point of view, the paper simply doesn’t present any evidence of the long-term trend you refer to. The paper may have gained plaudits, but I don’t think the crossover point around 2012 will have convinced many experienced statisticians.
The graphs go back over 30 years, but it’s important to observe (in the spirit of Ben’s “be sceptical” advice) that no figures are provided in the BMJ paper. The BMJ paper doesn’t say how often the sampling was carried out. If the statistics were collected every four years (a reasonable hypothesis but correct me if you know how often they were collected) then a 30-year graph has only seven data points! Not nearly enough for any type of linear-trend hypothesis. If you look at the original OECD graphs for other countries not referred to in the BMJ paper, they do not show linear-trending effects.
Graphs of coronary heart disease rates (not the same as AMI) don’t show linear-trending either.
@emen was 100% correct to say “2. “the gap is closing so rapidly” – that often happens when the start is very poor: countries with lower GDP, higher inflation or huge fiscal deficit can happily point out: oh yes, but our fiscal deficit is being reduced at a quicker rate than in some richer countries. It is, until it reaches their level, then the speed will most probably slow down.”
This effect, so well explained by @emen does seem to happen with regard to mortality rates too. It’s basically a learning-curve effect. It is true that the BMJ paper doesn’t illustrate the learning-curve effect, but it is apparent elsewhere in the literature. For example, the Mckee and Nolte paper on 1998 to 2003 trends specifically says that improvements during this 5 year period were particularly marked in nations with poor records at the start of the period.
msjhaffey said,
February 18, 2011 at 8:12 am
@guy
Unfortunately, your post is exactly the sort of post that gives science a bad name. Most of the sentences contain unproven generalisations and several are demonstrably false. It is difficult to be overtly prejudicial on one hand and then to say “Listen to me because I have evidence that you really need to know.”
BenG succeeds, despite IMHO skating some times on thin ice, because in his swashbuckling way he assembles evidence to support his points. This is why I think Bad Science should be mandatory reading for all newly-elected politicians.
Guy said,
February 18, 2011 at 10:29 am
“Most of the sentences contain unproven generalisations and several are demonstrably false”
Possibly, but if you want to engage in anything other than a slanging match you have to provide examples! Your only example is a misrepresentation. I don’t say listen to me I have evidence because as Ben so clearly demonstrated, there isn’t evidence for what they are doing. I wish there was evidence that they “really need to know”.
jodyaberdein said,
February 18, 2011 at 1:16 pm
Regarding one unproven generalisation –
…’They just aren’t terribly honest’..
NHS operating framework 2011/2012
5.43 The flexibilities set out in the 2010/11 NHS Operating Framework will remain largely in place for 2011/12. One new flexibility being introduced in 2011/12
is the opportunity for providers to offer services to commissioners at less than the published mandatory tariff price, where both commissioner and provider agree. Commissioners will want to be sure that there is no detrimental impact on quality, choice or competition as a result of any such agreement.
DOH 24/01/11
‘Health Secretary Andrew Lansley today outlined how the NHS must embrace value-based competition if it is to meet the future needs of the public it serves….competition must be based on the quality of results for patients and not cost alone…They [Monitor] will also oversee the process of price competition, which is to be allowed only where it is deemed appropriate and where it will not harm quality of service.
DOH 18/02/11
There have been reports in the media that the Department of Health has changed its direction on the issue of price competition, based on David Nicholson’s letter published yesterday, in which he highlighted that ‘there will be no question of introducing price competition’.
In response, a Department of Health spokesperson said:
“There is no u-turn because we never intended to introduce price competition. There has been incorrect reporting around this and Sir David’s letter simply sets that straight.
“There is no change to the policy set out in the 2011-12 Operating Framework and first set out under the previous Government in December 2009. Given the potential risk to NHS business planning of the incorrect speculation that this policy amounts to price competition, Sir David’s letter serves to reaffirm this policy – that we want to see competition on quality, not price. Anyone can see that it is difficult for the government to u-turn on a policy which it inherited from the previous administration and which it has not amended since coming into office.
“The NHS Operating Framework published last December remains unamended.”
msjhaffey said,
February 18, 2011 at 2:13 pm
@guy ok here we go
“Politicians aren’t stupid”
Demonstrably, many are. The basis for being elected has little correlation with IQ. Do you have any supporting evidence for your claim?
[Politicians] “understand evidence perfectly well.” Proof? In the absence of evidence to support this statement, I’d suggest that some do and some don’t.
“They just aren’t terribly honest!” I know several politicians of various stripes who are scrupulously honest. Again, can you prove your claim?
“So yes there is no evidence. They know that.”
Proof?
“So they are just trying to hoodwink those who they can take in through this pretence.” Even if they did know there is no evidence, what proof do you have of their motives?
“They are driven by dogma” Proof? I know again a good many who are driven but not much more than wanting to do a good job for their community.
I don’t claim that politicians are all pure of heart. That wouldn’t be credible. Nor is it credible to claim the opposite, as you do. There are people who are dishonest, malign or deluded in all professions and it is lazy thinking to hurl generalisations in this fashion.
Cold fusion, anyone?
Guy said,
February 18, 2011 at 3:25 pm
Most of us who have moved on beyond playground bickering can tell the difference between
1 evidence based statements eg drug XXX showed a 0.88 (CI 0.85-0.93) risk of death compared to placebo, and
2 rhetorical statements like “Politicians aren’t stupid”.
Bob Calder said,
March 3, 2011 at 2:30 pm
I attended a symposium where Gavin Schmidt spoke about the same problem. He refers to it as “scientifying” or applying scientific terminology in a political argument. A political argument is one where all knowledge is on the table and is traded from one player to another in a game process. When politicians see everything as a political asset, the values tend to be equalized so that “global warming” can be traded for a barrel of oil for instance. This is what I got out of the talk at any rate. He undoubtedly has a more nuanced view.