Why is evidence so hard for politicians?

February 12th, 2011 by Ben Goldacre in bad science, politics | 60 Comments »

Ben Goldacre, The Guardian, Saturday 12 February 2011

One thing you hope for, with politicians, is that they won’t make the same mistakes over and over again.

Last week we saw that the government has overstated the problems in the NHS by using dodgy figures (to be precise, they used misleading static figures instead of time trends). We saw that Andrew Lansley’s repeated claim that his reforms are justified by evidence was untrue: the evidence doesn’t show that his price-based competition improves outcomes (if anything it makes things worse); and the evidence also doesn’t show that GP consortia improve outcomes (unless you cherry pick only the positive findings). It’s okay if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.

Now Lansley’s junior minister Paul Burstow MP has kindly responded, repeating the exact same mistakes again, only more clumsily. I find this, in all seriousness, genuinely frightening from a minister, so I’ll explain how he does it.

The government initially claimed that UK heart attack death rates were twice as bad as France. This was an overstatement: they are, but following recent interventions the gap is closing so rapidly that on current trends it will have disappeared entirely by 2012. In response, Burstow cites a 2008 paper by McKee and Nolte which he says “concluded that the UK had one of the worst rates of mortality amenable to healthcare among rich nations”.

Burstow either misunderstands or misrepresents this very simple and brief paper. It is a study explicitly looking at time trends, not static figures, and it once again finds that comparing 2003 with 1998, the UK still had fairly high rates of avoidable mortality, but these were falling faster than in all but one of the other 18 industrialised countries they examined (meanwhile in the US, avoidable mortality improved at a disastrously slow pace, although they spent more money).

This is a paper showing the success of the NHS, and the fact that we are discussing such a massive improvement in avoidable mortality from Labour’s first term in government is not my choosing: this is the paper that was cited by the Tory minister as evidence, bizarrely, of the NHS’s recent failures.

Next Burstow says I “overlooked the impact assessment we published alongside the health and social care bill, where we present a thorough analysis of the evidence for and against our plans… studies show that GP fundholding and practice-based commissioning delivered shorter waits and fewer referrals to hospitals for patients.”

In its section on GP fundholding, this “thorough analysis” ignores the four peer-reviewed academic papers I described last week, which sadly found no evidence of an overall benefit from GP fundholding. It makes a series of 5 assertions about outcomes, though these are unreferenced to any paper at all.

I contacted the Department of Health, who ferreted out the sources: there was just one, a document from the King’s Fund. It’s not a peer-reviewed academic journal article, but the King’s Fund are pretty good, in my view. If you read this document, it too finds that the results of GP Fundholding were mixed: some things got better, some things got worse.

So the Minister has cherry picked only the good findings, from only one report, while ignoring the peer-reviewed literature. Most crucially, he cherry-picks findings he likes whilst explicitly claiming that he is fairly citing the totality of the evidence from a thorough analysis. I can produce good evidence that I have a magical two-headed coin, if I simply disregard all the throws where it comes out tails.

Here is what politicians apparently cannot understand: it’s fine to make policy based on ideology, whim, faith, principles, and all the other things we’re used to. It’s also fine for evidence to be mixed. And it’s absolutely fine if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

60 Responses



  1. Niall said,

    February 12, 2011 at 12:26 am

    It’s disturbing that this misleading of the public will probably not be considered a resigning matter.

  2. penglish said,

    February 12, 2011 at 12:28 am

    I can’t say I’m surprised.

    A while ago a minister said that there was evidence that school uniforms improved educational outcomes. When I wrote to my MP to ask for the evidence it turned out that the evidence was… a survey of parental views about school uniform. No outcome measures whatsoever…

    People say that it’s sad that people are so cynical about politics and politicians – but they don’t do themselves any favours, do they.

  3. jodyaberdein said,

    February 12, 2011 at 12:54 am

    It’s fine..

    .. so long as you don’t mind if your policies don’t actually work, so long as you don’t mind repeating mistakes over and over and over. Assuming your stated aim is your actual one of course.

  4. Riffler said,

    February 12, 2011 at 1:07 am

    “We don’t have anything you’d call evidence, but we made some stuff up, and we’re going to call it evidence.”

    Has anyone got to the bottom of what the christian ACMD appointee-then-sacked means when he advocates “total abstinence” from drugs – is that all drugs or just illegal drugs? Does he think a drug suddenly, magically becomes harmful the day it’s made illegal? Or does he advocate totally abstaining from aspirin and caffeine too? What would partial abstinence consist of?

  5. slartibartfast said,

    February 12, 2011 at 1:29 am

    The fundamental mistake the NHS made was the introduction of QOF and the payments associated with it. QOF is fundamentally associated with the biomedical model and wedded to EBM. EBM is a very useful tool but it is not a recipe for good health. GP Consortia and commissioning will further divide GPs from their patients and potentially alienate their hospital-based colleagues. Why was it necessary to change a world-class, cost-effective health system in 2004? Was it, as Lord Darzi later said “to break the power of the doctor-patient relationship”? Seems to have worked – if you believe what you read in the Daily Mail!

  6. maurizio said,

    February 12, 2011 at 2:39 am

    Great article Ben, thanks!

    The fundamental issue is not that politicians select and make up evidence to support their agenda, but that people do not sanction them for it. What people do not demand in honesty politicians will hardly give. Take it from an Italian, the problem has farcical proportions where I come from.

  7. jasonw said,

    February 12, 2011 at 7:24 am

    I think you’re being a bit optimistic, sadly.

    There’s no evidence (hah!) that they’re misunderstanding anything or making any mistakes. It’s at least as likely that they understand the evidence perfectly well and know that misusing it will support their case.

    The real question should be, why can they get away with it so easily?

  8. kimaldis said,

    February 12, 2011 at 7:28 am

    The disturbing thing for me is that we don’t see clear, well reasoned explanations like this appearing on newspaper front pages.This is a big deal and yet the press are all but ignoring it.

  9. skyesteve said,

    February 12, 2011 at 7:48 am

    @ slartibartfast (what does that mean by the way?) – QOF is not perfect. And you are right, it does shift the care model from a patient-centered model to one that is “medicalised”. But I don’t think QOF is all bad. Prior to 2004 there was, at least here in Scotland, a wide variation in the quality of primary care on offer in some areas. There were not uniform expectations on the quality of care. QOF has helped to narrow that gap. It’s probably not made good practices much better but it has helped poorer practices improve in my view. I have seen that as someone who has visited a large number of practices in a former QOF role.
    The other thing with QOF was that the changes brought about by it were always likely to be a bit soft and take a long time to come to fruition and even 6 years may not be enough. And QOF must not be seen in isolation. One also has to look at Enhanced Services which run hand-in-hand with QOF and which, again, I think have resulted in a more level playing field when it comes to the care on offer. Of course, this is largely gut feeling so may be not acceptable to some on this forum!
    As for the 2004 GP contract – the only reason this was voted for by the profession was because of the out-of-hours opt out. The Government at the time said they would impose the contract without the OOH opt out if GPs didn’t vote for it.
    Since the NHS was founded successive Governments have had it in for GPs. They could not stand the fact that, in a publically-funded National Health Service, most GPs were self-employed independent contractors beholden to no-one in the Dept of Health. There have been repeated attempts to bring GPs to heal and I think 2004 was another attempt. They promised a high-trust, bureaucracy-light contract and gave us quite the opposite!
    I have heard repeated references to the Government’s proposals on commissioning that it is some kind of “Fundholding Plus”. In my view Fundholding first time around sucked. It benefited some practices at the expense of others and, by definition, some patients at the expense of others – truly a post-code lottery of health care and the complete antithesis of the founding principle of the NHS.

  10. twaza said,

    February 12, 2011 at 7:55 am

    spot on as usual – but “the King’s Fund are pretty good, in my view” ????

    Not when it comes to reviewing the science required to assess complementary and alternative medicine:

    www.kingsfund.org.uk/press/press_releases/complementary_meds.html

    The report shows no understanding of the placebo effect, and a wilful denial of risks of bias in results from unblinded trials.

  11. jodyaberdein said,

    February 12, 2011 at 9:17 am

    slartibartfast (what does that mean by the way?) shame on you lol

    en.wikipedia.org/wiki/Slartibartfast

  12. misterjohn said,

    February 12, 2011 at 10:36 am

    skyesteve suggests that, “There have been repeated attempts to bring GPs to heal”.
    A nice suggestion.

  13. mesmer said,

    February 12, 2011 at 11:06 am

    @ twaza; yes, I remember when I first read this report from KF it seemed to be an open invitation to ‘find means of evidence’ to support interventions which had already been challenged by scientific method.

  14. obearap said,

    February 12, 2011 at 11:36 am

    Fancy that. A politician being selective in the “evidence” he cites to support his case. Keep up the good work Ben. It might help people to decide whether this is how we want decisions made when the next election comes around.

  15. jenniferbridge said,

    February 12, 2011 at 11:41 am

    If I was David Cameron I wouldn’t be very happy that a Minister was making such errors. However, I don’t imagine any of the Gov’t are Guardian readers so how are you to get the message across?

  16. QuietKnoll said,

    February 12, 2011 at 11:50 am

    Not only have governments of the past 2 decades pursued faith based rather than evidence based health policies they have also relied heavily on voodoo economics and continue to confuse efficiency with effectiveness.

    As a result readmissions and hospital acquired infections are not penalised in the pricing tariffs that underpin the approach to ersatz markets. The continuing fragmentation of the service further frustrates effective health provision such as continuity of care and collaborative responses to the needs of an ageing population with multiple co-morbidities and poly pharmacy.

    The relay baton of joined up health care will be dropped on an increasing basis without the evidence based approach that Ben is advocating.

  17. reprehensible said,

    February 12, 2011 at 12:18 pm

    I would suggest that it is only ok to create policy based on ideology if you state this explicitely and then detail how you will measure the effects (properly) in advance… not like it’s their money.

  18. emen said,

    February 12, 2011 at 12:25 pm

    Ben, this was stupid last week, and it is still stupid today:

    “The government initially claimed that UK heart attack death rates were twice as bad as France. This was an overstatement: they are, but following recent interventions the gap is closing so rapidly that on current trends it will have disappeared entirely by 2012.”

    1. first of all, as you know, you can cherry-pick anything to support your argument, is the gap closing so rapidly in every other areas where UK performs lower than average in Europe?

    2. “the gap is closing so rapidly”
    that often happens when the start is very poor: countries with lower GDP, higher inflation or huge fiscal deficit can happily point out: oh yes, but our fiscal deficit is being reduced at a quicker rate than in some richer countries. It is, until it reaches their level, then the speed will most probably slow down.

    3. also don’t forget, that more than 10% of UK population is privately insured and a lot more people pay for private care occasionally, so you can’t attribute any improving trends to interventions within the NHS only (which, the NHS is what the currents reforms are trying to improve.)

    4. but since you have cherry-picked a “trend”, let me cherry-pick another one:

    In most European countries (including Eastern-Europe) antenatal care includes 4 ultra sound scans.

    -one in week 7-8
    -one in week 11-13
    -one in week 19-20
    -one in week 34-36

    all evidence-based and performed in countries with state-run healthcare systems (so no private vultures after your money).
    In fact from Germany to Hungary, they do a lot more than just these 4.

    Up until a few years ago, the NHS provided only the two middle ones, and there was evidence that leaving out the last one is responsible for causing stillbirths which would have been avoidable.

    Now the NHS is only providing the 19-20 week scan in most areas of Britain. That (among other things) means that a a lot of women (unless actually miscarry) who used to be able to find out in week 12 that the baby they are carrying has been dead for 3 weeks, will now have to wait until week 17 when the midwife can’t detect a heartbeat, and wait for a bit longer because “baby might just be a little smaller than we thought”.

    “ON CURRENT TRENDS ANTENATAL ULTRASOUND SCANS WILL HAVE TOTALLY DISAPPEARED BY 2012, THE NHS HAPPILY PROVIDING THIRD-WORLD ANTENATAL CARE.”

    You see what I mean, I guess.

    Look, you can criticize the current proposed changes on any basis: ideological, professional, financial etc.

    But don’t do it by claiming that the NHS is a good service.

    Don’t make the same mistake over and over again. :-))

  19. jodyaberdein said,

    February 12, 2011 at 12:49 pm

    ‘As a result readmissions and hospital acquired infections are not penalised in the pricing tariffs that underpin the approach to ersatz markets’

    If a certain large NHS trust misses its target (a moving and ever decreasing one) for c diff infection this year it will be fined 2.5% of its entire operating budget. I think this is the case for all acute trusts actually.

  20. emen said,

    February 12, 2011 at 1:03 pm

    Incidentally, I agree that there are risks involved with the current proposals, the sheer speed of the changes, the radical increase of GPs’ responsibilities without training them etc.

    But introducing competition in almost any economic situation will almost immediately increase quality and reduce prices.

    One big mistake that the founders of the NHS made in 1948 was that they thought that in order to achieve universal access you have to make the system a complete state monopoly. By exluding the beneficial factors of private provision, such as competition, market, customers’ choice, customers’ rights, incentive to innovate, they created a system where low quality and high prices were inevitable. That is why most developed countries (not the USA), although provide universal access to healthcare, use private provision and private payer systems to achieve a quicker and better quality service with a wider range of treatments available.

  21. emen said,

    February 12, 2011 at 2:13 pm

    (somebody called “peitha” wrote this on the Guardian’s website)

    “The government initially claimed that UK heart attack death rates were twice as bad as France. This was an overstatement: they are but …”

    So actually what he said was right and not an overstatement after all. The ‘bad science’ is in your reading Ben, because you want to extrapolate a point made about a static snapshot into a claim about a trend, which he didn’t make.

    This is a paper showing the success of the NHS, and the fact that we are discussing such a huge improvement in avoidable mortality from Labour’s first term in government is not my choosing: this is the paper that was cited by the Tory minister as evidence, bizarrely, of the NHS’s recent failures.

    Absolute twaddle! IF the NHS had ben established only in 1997 THEN you might have a point, but after 40 years of the NHS we had amongst the highest death rates in the study. And in terms of policy, that other countries managed to achieve comparable reductions in mortality over the period of the study WITHOUT the vast extra expenditure can be construed as evidence that we got a pathetic bang for our taxpayers buck from the vast extra resources spent on healthcare.

  22. tig said,

    February 12, 2011 at 2:20 pm

    I am concerned that the UK Statistics Authority has not intervened in this, or in the debate on the misleading analyses used in the DLA consultation paper. Have you heard from them?

  23. DisagreeableWeasel said,

    February 12, 2011 at 3:06 pm

    The DLA consultation paper, as Tig says above, is full of similar assertions backed up with dubious ‘evidence’. It seems to be endemic in this crop of politicians.

    I’d love it, Ben, if you could find some time to expose the nonsense peddled as justification for reforms to disability benefit.
    Meanwhile, campaign group The Broken of Britain, have tried to take apart the DLA consultation paper’s twisted logic:
    bit.ly/gtkVy0

  24. NorthernBoy said,

    February 12, 2011 at 6:25 pm

    Hang on, back up a minute.

    Did someone up there just ask what “Slartibartfast” “means”?

    Has the world of Internet-based statistics discussion been brought so low that Adams’ writings are no longer known by all?

  25. Jonarific said,

    February 12, 2011 at 10:01 pm

    My knowledge of the inner workings of whitehall isn’t great, but from my understanding I think people are misconstruing how policy generation works in the realm of politics.

    The fundamental point here is that the Tories came in with an ideological position on how the NHS should be run and are trying to implement it. As part of the practice of generating policy Civil Servants have to conduct cost-benefit analyses, consultations and so on. However, this is a farce as at best it simply ends in tweaking a pre-determined policy direction.

    Here’s an example from my line of work: planningblog.wordpress.com/2011/02/01/best-guestimates/

    You also can refer to the need to carry out reasonable consultation and approach things with an open mind as a legal requirement in the Building Schools for the Future programme recently.

    I assume it was thought to be a good idea as it would help hold policies to account for the public or help generate good policy, but in reality it probably leaves some poor civil servants using all the textbook bad science tricks to justify their minister’s decision.

  26. robzrob said,

    February 12, 2011 at 10:14 pm

    Because evidence has nothing to do with politics.

  27. msjhaffey said,

    February 12, 2011 at 10:49 pm

    Following this link

    www.bmj.com/content/342/bmj.d566.full

    in Ben’s article, you can see that “standardised death rates” quoted are decreasing roughly linearly over 30 years in France and Britain, although at different rates. This seems extraordinarily unlikely.

    Can anyone explain?

  28. juantootree said,

    February 12, 2011 at 11:18 pm

    To emen, as the NHS had never been adequately funded since its founding, especially from the mid 70s to 1997, then looking at the NHS from 1997 seems pretty fair. NHS funding only started to more nearer western averages with Blair and Brown.
    That’s not a political point by the way, as none of the previous Labour governments (not even Atlee’s that created it) funded health properly.

    Regarding the relative mortality rates between the UK and France:

    On Radio 4’s “More or Less” the other week a guy stated that in the UK, as in most countries, where someone with a history of heart problem dies and no absolute cause can be identified the cause of death recorded will be coronary related. In France the cause of death is only recorded as heart related if it can be proven, otherwise it’s recorded as other.
    Podcast available here (downloads.bbc.co.uk/podcasts/radio4/moreorless/moreorless_20110121-1400a.mp3)

    Given this, the fact that we’re closing the gap with France is hugely impressive.

  29. slartibartfast said,

    February 13, 2011 at 1:52 am

    @skyesteve: QOF was associated with payments – hence it introduced a “fee-for-service” component to the NHS. I do agree that it impacted adversely on “patient-centred care” and the doctor-patient relationship. The therapeutic benefits of the relationship and patient-centredness are not easily understood by politicians – for that matter, they are not easily understood by GPs or other doctors. I also think that with QOF came a greater awareness of EBM and utilization of EBM in practice, but QOF over-road the “person of the patient” aspect in the utilization of EBM to the point where the “medicalised outcome” was more important than the impact on the person of the patient. I believe that many of the perceived benefits of QOF merely came from the acceptance and use of EBM as a tool and not from QOF itself. EBM = “educate”, QOF = “regulate”. Better information = better outcomes.
    As for the after-hours issue, well that is very complex. Prior to the early 90’s, after-hours services were effectively voluntary. GPs did this work as part of their commitment to their patients and to “continuity of care”. In the early 90’s legislation changed all that, it made 24/7 provision of care mandatory, a “requirement”. Not only did this occur in the UK, it occured in Canada, Australia, New Zealand and other countries. Compulsion changes attitudes, and attitudes changed. The 2004 contract merely removed that compulsion. But now we have after-hours without continuity of care, and the cracks are beginning to show.

    Now for a comment on some of the other posts – with particular regard to insurance/ user-pays systems: I whole heartedly agree! An editorial in a recent edition of the Annals of Family Medicine summarised care rather well. It referred to two axes: a longitudinal axis of “continuity of care” heavily predicated on patient-centredness; and a vertical axis of specialist intervention. Specialist intervention has been shown to be more effective at achieving QOF-type targets but considerably less effective at achieving improved long-term outcomes and longevity! We need a balance. Specialist interventions should be timeous and brief. Under a waiting-list system, they are neither. The NHS favours the longitudinal axis, but distorts the vertical – and so outcomes are not what they should be. The US system favours the vertical axis over the longitudinal – and makes outcomes VERY expensive (more than 4x the per capita expenditure) for outcomes that are, in fact, worse than in the UK when considered over time. We need an appropriate blend of the two: improved access to primary care with better information and greater patient-centredness, and better access to secondary care with reduced waiting times (not waiting times that are politically adjusted to look good, but REAL reductions in waiting times). And a lot less money wasted on unnecessary (doctor-centred) investigations.

  30. chris lawson said,

    February 13, 2011 at 8:11 am

    Is this the same Richard Horton who thought it was a good idea to publish Andrew Wakefield’s autism paper in The Lancet?

    @slartibartfast: you seem to be using a definition of EBM that is not used in actual practise.

    @emen: what evidence is there that doing 4 or more routine ultrasounds makes a positive difference to antenatal care? In Australia we do one routine ultrasound at 18-20 weeks and there is an optional one at 11-13 weeks for nuchal fold thickness (not paid for by the govt. but most couples choose to pay for it even if they are at low risk of trisomy), and otherwise only if indicated by clinical scenarios. We seem to have excellent outcomes. I would also caution against blanket statements that state-run health systems are automatically evidence-based and free from interference by private monetary interests. And are you really saying that ultrasounds will have disappeared entirely from the NHS by next year? Because that sounds to me like hyperbolic bullshit you just made up on the spot.

  31. chris lawson said,

    February 13, 2011 at 8:37 am

    Actually, on re-reading emen’s post I am now wondering he his comment about ultrasounds in the UK was meant as a satirical take on the closing gap between UK and French mortality rates post-MI.

    The problem is that the paper Ben linked to shows a steady and reliable trend rate that has persisted from 1979 to 2007, that is nearly 30 years of consistent progress in both the UK and French trend lines. It is hardly stupid to extrapolate these trends for another 5 years. It is hardly stupid to criticise Lansley’s claim that the NHS is failing when he compared the UK to the best result in the EU (cherry-picking) *and* does not mention that a large proportion of France’s better mortality rate is due to miscoding *and* leaves out the fact that most of France’s improvement in the mortality has been among its immigrant population rather than across all demographics *and* does not mention the improving trend.

    When a politician leaves out four important caveats, that to me is strong evidence of misleading the public.

  32. jimjim237 said,

    February 13, 2011 at 11:27 am

    Evidence seems likely to be hard for politicans for two reasons.

    1.
    Many politicians probably don’t do numbers.

    2.
    They are held to account primarily by the national press and journalists (with honourable exceptions:) don’t do numbers either. Journalists don’t need numbers since their readers don’t care about numbers other than of course The Lottery Numbers.

    On the proposed NHS reforms more generally, I have not read much about these reforms but my first take on the TV News presentations on it is that GP’s are going to want something for becoming the heads of multi-million pound business, presumably responsible for huge budgets. No specific mention seems to have been made of this but I would imagine that quite a few GPs will be going to get properly stinking rich out of it.

    So how much NHS money will a consortium manage? Let’s try to estimate it from some published figures.

    www.guardian.co.uk/society/2007/oct/09/publicservices
    “NHS spending will rise from £90bn this year to £110bn in 2010″

    That’s say

    100,000,000,000 – pounds spending
    50,000,000 – population – yes I know its nearer 60M
    but this is good enuf.

    10,000
    5

    £2000 a head per year.

    Each GP I seem to recall has about 2000 patients so the budget each has to manage will be £4M.

    There was I seem to recall talk of consortia of say 50 GPs so that’s a £200M slush fund. I would imaging that the head of that (presumably a GP) would be taking home a fair wedge on top of her NHS GP income. I presume there will be some mechanism to stop the unscrupulous from simply pocketing the money? Are there limits as to the amounts that can be skimmed off as profits? Are there limits as to the amounts that these GPs can pay themselves?

    Memo from Cammo to GPs:-

    Here’s £200M a year, hire yourself, your spouse and some old uni friends and PARTY like you had won the lottery.

    Please drink responsibly,
    David.

    ###
    Here is another crack at the likely budget of the consortia from another tack.

    www.guardian.co.uk/politics/2011/jan/17/david-cameron-nhs-reforms-speech
    “141 groups of GP practices, caring for half of the population of England”
    “80% of the NHS’s £100bn budget”

    So we have:-

    40,000,000,000 – pounds – half of 80% of 100M
    141 – consortia

    Calculator for this one:-

    £283,687,943 per consortium. Well bugger me sideways, same as guessed at above. Well £200M is near enough £283M for me.

    ###
    How many patients will a consortium have?

    www.guardian.co.uk/politics/2011/jan/17/david-cameron-nhs-reforms-speech
    “141 the number of such “pathfinder” groups who provide healthcare to 28.6 million people in England”

    So that’s

    29M / 141

    205,673 patients per consortium

  33. Youdell said,

    February 13, 2011 at 11:59 am

    @emen – interesting points raised but I have a couple of comments.

    “1. first of all, as you know, you can cherry-pick anything to support your argument, is the gap closing so rapidly in every other areas where UK performs lower than average in Europe?”

    I think this misses the point, Ben makes it pretty clear in his original article he is not interested in making any claims regarding how the NHS compares to other health systems, but as a critique of how evidence is used by the politicians in question. I’m not sure if it is appropriate to therefore accuse him of cherry picking as he is debunking specific claims made – not making conclusions of his own. I am pretty sure no one would be happier to see multiple and appropriate outcomes to be used to measure the workings of the NHS with which to govern policy.

    “2. “the gap is closing so rapidly” – that often happens when the start is very poor: countries with lower GDP, higher inflation or huge fiscal deficit can happily point out: oh yes, but our fiscal deficit is being reduced at a quicker rate than in some richer countries. It is, until it reaches their level, then the speed will most probably slow down.”

    If you go back to the original BMJ debunking paper referenced (www.bmj.com/content/342/bmj.d566.full#ref-2), you’ll see that the numbers involved and the time over which these studies have been carried out mean that the argument you put forward does not really apply. The trend is long term and continues over range of relative differences thus I think that analogy you make regarding GDP is inappropriate.

    “3. also don’t forget, that more than 10% of UK population is privately insured and a lot more people pay for private care occasionally, so you can’t attribute any improving trends to interventions within the NHS only (which, the NHS is what the currents reforms are trying to improve.)”

    Again – go look at the BMJ data – 10% is an insignificant amount in the context of the long term changes.

    And the guy you are quoting from the guardian comments needs to check his facts – if you want to talk about “bang for your buck” factor in to all comparisons that we spend less than France and most other western European countries on healthcare.

    “…stupid……still stupid”? Seems a little harsh?

  34. SamBC said,

    February 13, 2011 at 12:44 pm

    If you think this is all bad, you should look over the recent press releases and statements about disability benefits. Heck, the whole consultation paper. They’ll happily cite sources, but actually trying to find justification in the sources for what the government says they say seems fruitless.

  35. slartibartfast said,

    February 13, 2011 at 8:59 pm

    @Chris Lawson: EBM = Evidence Based Medicine. How does my definition differ from what is used in actual practise? Am I missing something?

  36. chris lawson said,

    February 13, 2011 at 10:07 pm

    @slarti…

    When you say, “EBM is a very useful tool but it is not a recipe for good health” it seems to me that you are using the common strawman version of EBM. The best definition of EBM is David Sackett’s: “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research.”

    When put like it, you can see that EBM is all about devising the best recipes for good health based on the best available evidence.

  37. slartibartfast said,

    February 13, 2011 at 11:18 pm

    @Chris Lawson:
    EBM is often used to descibe the “current best evidence” information, rather than the wider description and application thereof. I take your point, however, and perhaps I should refer to “Evidence Based Guidelines” rather than EBM in order to be semantically correct.
    It is not the semantics that are my concern, it is the issue of the use of disease-specific information in a patient-centred way, especially when the patient may have several comorbid diseases. David Sackett’s definition is the ideal, but, I fear, not the “standard practice”. The “best practice” ideal for each condition when applied collectively may not be the “best practice” for the individual patient, and so net outcome is worse – despite so-called “best practice”. QOF incentivises the “best practice” for the individual condition, not the individual patient. Hence EBM IS only a tool, because there IS NO EBM model specific to the individual patient and thus there is only an individual recipe and not a recipe for all. This is why, I believe, medicine is an art as well as a science.

  38. slartibartfast said,

    February 14, 2011 at 4:13 am

    @emen; @jimjim237:
    Per capita health spending (standardised for USD)per annum (OECD figures – 2007 comparison – have not included all OECD countries – so sorry,”cherry picked”)
    1)USA: $7 258; 2)Norway: $4 791; 3)Canada: $3 867;
    4)Netherlands: $3 853; 5)Germany: $3 619;
    6)France: $3 593; 7)Denmark: $3 540;
    8)Australia: $3 353; 9) Sweden: $3 349;
    10)United Kingdom: $2 990; 11) Japan $2 701;
    12)New Zealand: $2 471
    Interestingly New Zealand has average life expectancies almost identical to Norway – but spends just over half of what Norway does. Of all the countries compared, the USA does the worst – despite spending the most. The UK spends less than Germany, France and Denmark – for comparable outcomes in life expectancy.
    Just for comparison really – the figures don’t really lend themselves to any major conclusions.

  39. skyesteve said,

    February 14, 2011 at 8:47 am

    @11 jodyaberdein – I read the Hitchhiker’s Guide when I was 13 which is over 30 years ago. I don’t recall all the characters and I haven’t read it since as I belive (and at the risk of heresy here) that 13 is about the appropriate reading age for it!

    @12 misterjohn – :-))

    @29 slartibartfast – I think it’s worth pointing out that UK NHS general practice has always had an element of “item-of-service” payments about it. Pre-2004 it was colloquially referred to as “the red book”. Even out-of-hours work carried with it a small annual retainer fee plus a paultry fee paid for visits made during the night.

  40. pauldepstein said,

    February 14, 2011 at 9:05 pm

    @msjhaffey

    Interesting question about the strong linearity of the graphs. As Ben says, John Appleby’s article was an “instant” response to government claims, and indeed he seems to have simply copied the graphs from the original OECD source, without bothering to make additional interpretative comments and explanations (such as providing figures as well as graphs, and such as explaining why he bases his analysis on a 30 year trend rather than some other more relevant time period).

    In other words, the BMJ author may know the answer to your question, but it’s also plausible that he does not know. Try asking on a newsgroup to people who are likely to know about the methodology by which the OECD collect statistics. Or ask the OECD for the figures on which the graph is based.

    John Appleby’s data does seem to match the OECD data so your question is properly addressed to the OECD.

  41. chris lawson said,

    February 14, 2011 at 10:07 pm

    @slarti…

    I think we’re both on the same page. EBM is important. It is not always implemented well. There are complex issues in how one implements it. Medicine is an art as well as a science. I think we agree on those points.

    My only quibble would be with your version of EBM as disease-specific when in fact good EBM includes all evidence, including evidence involving co-morbidities (and the literature is *full* of such evidence, by the way). I agree that narrow application of one or two studies can result in poor outcomes (e.g. treating someone who had a stroke with warfarin without taking a good enough history to exclude their bleeding peptic ulcer), but my answer to that is that that is not a failure of EBM but of clinical practise. And given the choice of criticising EBM itself because someone didn’t implement it well, or trying to get health practitioners to understand the principles of EBM, I would go with the latter. Especially seeing as there are plenty of people out there who would like us to scuttle EBM completely.

  42. chris lawson said,

    February 14, 2011 at 10:08 pm

    Errrgh, there’s an awful sentence in that previous post…

  43. slartibartfast said,

    February 15, 2011 at 1:45 am

    @Chris lawson
    I’m not critical of EBM, I believe it is extrodinarily useful – especially when, as you said, it is practiced properly. I would apologise to any who thought I was critical. I am critical, however, of QOF, particularly the implementation of QOF! Although not part of the NHS, I have more than a passing interest in it – especially as we seem to “inherit” bureaucrats from it (@skyesteve – just in case you wondered where your former head of the Scottish NHS went – we’ve got him. Do you want him back?)

    Evidence-based guidelines form the “bio” part of Engel’s Biopsychosocial model (overly simplistic perhaps) – and if, as you correctly assert, we use these guidelines together with the other information (“patient as person” – psychosocial), we will practice true Evidence Based Medicine. Patients would be happier – and better for it (not to mention the doctors). Those who would like us to scuttle it are either ignorant, or worse. We have a fair few of them across the ditch too.

  44. Robert Carnegie said,

    February 15, 2011 at 2:15 am

    I don’t see that a two-headed coin would have to be magical. A coin with the head on both faces would come up heads every time naturally. However, it’s the sort of trick device that a stage magician might use.

    You would ascribe genuine magical properties to a coin with a heads face and a tails face that always came up heads – although you might refer them to James Randi and his million dollar prize first, and general careful scrutiny. Or, rather, he would; it’s a lot of -his- money.

    I think the point youdwanted to make may have been the one in my first paragraph.

    By the way, to what extent are there trends in health outcomes? There are more and more tubbos today, and fewer and fewer smokers, so I suppose those are trends and relevant factors.

  45. Jeffreysnj said,

    February 15, 2011 at 5:07 am

    Hi Just saw a bit of Lansley on Hardtalk spouting the same canards

    www.bbc.co.uk/programmes/b00ymycb

    Was it my imagination but was Sarah Montague spectacularly bad?

  46. CampFreddie said,

    February 15, 2011 at 12:07 pm

    The problem is that politicians don’t start with a blank slate and devise policies based on evidence. They start with ideas and look for evidence that backs them up. It’s simply not a politicians job to devise policy based on evidence. Their job is to do it the other way round!

    In this case, they government have started with a more or less fully-formed ideological policy of free market reform with ‘decentralised’ GP-focused responsibility. They’ve then scoured the data for evidence that supports this position. They wouldn’t know a student-t test from a student grant, and have no intention of supporting either.

    I’ve done this in the past, searching for evidence that backs up the argument instead of searching for an argument that meets the evidence. It’s an easy and tempting mistake.

  47. IMC said,

    February 15, 2011 at 4:33 pm

    @slartibartfast (good choice of name, btw)

    Interested in looking up a couple of things you quoted: the Annals of Family Medicine editorial you mention (I had a look on PubMed, but couldn’t find anything that looked right); and the healthcare spending figures. Or maybe the healthcare spending figures came from the Annals editorial? Anyway, I’d be interested to read them over…

  48. slartibartfast said,

    February 15, 2011 at 9:18 pm

    @IMC:
    Perhaps the axes are not as explicitly stated as I have made them, but they are there (under “Implications”).
    “The Paradox of Primary Care” Stange, Ferrer.
    AnnFamMed. 7:293-299(2009)
    As for the spending figures, these are straight off the OECD website Healthcare spreadsheet.
    stats.oecd.org/Index.aspx?DataSetCode=HEALTH

  49. secondfresh said,

    February 16, 2011 at 3:06 am

    I’m starting to get disheartened and think that evidence is hard for a whole bunch of people. I’ve been on youtube the last couple of days looking at people’s comments on the vaccination program that Bill Gate’s is involved in to help distribute vaccines in the developing world. I’ve sent people lists of peer-reviewed articles supporting the use of vaccinations, but to no avail! People send me back internet links to random sites that protest the use of vaccines on hearsay. If anyone is interested in the evidence, or would like to be slightly entertained by seeing how frustrated I am, check out our latest post “Vaccinations needed in the Developing World” on:

    secondfresh.com

    Comments appreciated :)

    Cheers,

    Lauren K

  50. pauldepstein said,

    February 16, 2011 at 9:33 am

    @Youdell,

    I read the BMJ paper too. From a statistical point of view, the paper simply doesn’t present any evidence of the long-term trend you refer to. The paper may have gained plaudits, but I don’t think the crossover point around 2012 will have convinced many experienced statisticians.

    The graphs go back over 30 years, but it’s important to observe (in the spirit of Ben’s “be sceptical” advice) that no figures are provided in the BMJ paper. The BMJ paper doesn’t say how often the sampling was carried out. If the statistics were collected every four years (a reasonable hypothesis but correct me if you know how often they were collected) then a 30-year graph has only seven data points! Not nearly enough for any type of linear-trend hypothesis. If you look at the original OECD graphs for other countries not referred to in the BMJ paper, they do not show linear-trending effects. Graphs of coronary heart disease rates (not the same as AMI) don’t show linear-trending either.

    @emen was 100% correct to say “2. “the gap is closing so rapidly” – that often happens when the start is very poor: countries with lower GDP, higher inflation or huge fiscal deficit can happily point out: oh yes, but our fiscal deficit is being reduced at a quicker rate than in some richer countries. It is, until it reaches their level, then the speed will most probably slow down.”

    This effect, so well explained by @emen does seem to happen with regard to mortality rates too. It’s basically a learning-curve effect. It is true that the BMJ paper doesn’t illustrate the learning-curve effect, but it is apparent elsewhere in the literature. For example, the Mckee and Nolte paper on 1998 to 2003 trends specifically says that improvements during this 5 year period were particularly marked in nations with poor records at the start of the period.

  51. irishaxeman said,

    February 16, 2011 at 2:02 pm

    emen said the introduction of competition improves services and lowers prices. British Rail anyone? Bus services anywhere in Britain anyone? Or do emen and others not actually use public services in the main parts of the UK? There is no evidence in the world in the last forty years that neoliberal freemarket competition – the ultimate in competition – did anything but impoverish the majority while lining the pockets of business and politicians.

  52. Guy said,

    February 17, 2011 at 9:47 am

    I think we need to remember that Ben’s title is slightly ironic. Those of us who work in evidence based disciplines look for evidence to improve what we do. If fresh evidence comes up then we re-appraise and hopefully adapt what we do to fit the evidence.

    Then there is politics. Politicians aren’t stupid and understand evidence perfectly well. They just aren’t terribly honest!

    So yes there is no evidence. They know that. We know that. So they are just trying to hoodwink those who they can take in through this pretence. I largely blame the lazy media who allow them to get away with it.

    So lets not pretend they are stupid or don’tknow what they are doing. They are driven by dogma and as we know that disease is immune to evidence.

    I agree with Ben that if they were honest about that, this debate would be a lot simpler. Back in pre-Maggie NHS days they used to pilot big changes and actually wait for the answer with interest. Oh those were the days….

  53. nathaniel said,

    February 17, 2011 at 11:09 pm

    There’s another reason why evidence is hard for politicians, though perhaps it’s not directly relevant here: if a politician changes her mind on the basis of evidence then she will be accused of “doing a U-turn,” which is seen as a sign of weakness. I despise that phrase and long for a day when a politician can say “I’ve changed my mind on the basis of new evidence” and have that be seen as the positive thing it obviously is.

  54. msjhaffey said,

    February 18, 2011 at 8:04 am

    Someone identifying himself as Paul Epstein has contacted me about this blog entry and the comments. He is for an unknown reason (he is apparently registered) unable to post comments and has asked me to post the following two comments.

    @msjhaffey

    Interesting question about the strong linearity of the graphs. As Ben says, John Appleby’s article was an “instant” response to government claims, and indeed he seems to have simply copied the graphs from the original OECD source, without bothering to make additional interpretative comments and explanations (such as providing figures as well as graphs, and such as explaining why he bases his analysis on a 30 year trend rather than some other more relevant time period).

    In other words, the BMJ author may know the answer to your question, but it’s also plausible that he does not know. Try asking on a newsgroup to people who are likely to know about the methodology by which the OECD collect statistics. Or ask the OECD for the figures on which the graph is based.

    John Appleby’s data does seem to match the OECD data so your question is properly addressed to the OECD.

    @Youdell,

    I read the BMJ paper too. From a statistical point of view, the paper simply doesn’t present any evidence of the long-term trend you refer to. The paper may have gained plaudits, but I don’t think the crossover point around 2012 will have convinced many experienced statisticians.

    The graphs go back over 30 years, but it’s important to observe (in the spirit of Ben’s “be sceptical” advice) that no figures are provided in the BMJ paper. The BMJ paper doesn’t say how often the sampling was carried out. If the statistics were collected every four years (a reasonable hypothesis but correct me if you know how often they were collected) then a 30-year graph has only seven data points! Not nearly enough for any type of linear-trend hypothesis. If you look at the original OECD graphs for other countries not referred to in the BMJ paper, they do not show linear-trending effects.

    Graphs of coronary heart disease rates (not the same as AMI) don’t show linear-trending either.

    @emen was 100% correct to say “2. “the gap is closing so rapidly” – that often happens when the start is very poor: countries with lower GDP, higher inflation or huge fiscal deficit can happily point out: oh yes, but our fiscal deficit is being reduced at a quicker rate than in some richer countries. It is, until it reaches their level, then the speed will most probably slow down.”

    This effect, so well explained by @emen does seem to happen with regard to mortality rates too. It’s basically a learning-curve effect. It is true that the BMJ paper doesn’t illustrate the learning-curve effect, but it is apparent elsewhere in the literature. For example, the Mckee and Nolte paper on 1998 to 2003 trends specifically says that improvements during this 5 year period were particularly marked in nations with poor records at the start of the period.

  55. msjhaffey said,

    February 18, 2011 at 8:12 am

    @guy

    Unfortunately, your post is exactly the sort of post that gives science a bad name. Most of the sentences contain unproven generalisations and several are demonstrably false. It is difficult to be overtly prejudicial on one hand and then to say “Listen to me because I have evidence that you really need to know.”

    BenG succeeds, despite IMHO skating some times on thin ice, because in his swashbuckling way he assembles evidence to support his points. This is why I think Bad Science should be mandatory reading for all newly-elected politicians.

  56. Guy said,

    February 18, 2011 at 10:29 am

    “Most of the sentences contain unproven generalisations and several are demonstrably false”

    Possibly, but if you want to engage in anything other than a slanging match you have to provide examples! Your only example is a misrepresentation. I don’t say listen to me I have evidence because as Ben so clearly demonstrated, there isn’t evidence for what they are doing. I wish there was evidence that they “really need to know”.

  57. jodyaberdein said,

    February 18, 2011 at 1:16 pm

    Regarding one unproven generalisation –

    …’They just aren’t terribly honest’..

    NHS operating framework 2011/2012

    5.43 The flexibilities set out in the 2010/11 NHS Operating Framework will remain largely in place for 2011/12. One new flexibility being introduced in 2011/12
    is the opportunity for providers to offer services to commissioners at less than the published mandatory tariff price, where both commissioner and provider agree. Commissioners will want to be sure that there is no detrimental impact on quality, choice or competition as a result of any such agreement.

    DOH 24/01/11

    ‘Health Secretary Andrew Lansley today outlined how the NHS must embrace value-based competition if it is to meet the future needs of the public it serves….competition must be based on the quality of results for patients and not cost alone…They [Monitor] will also oversee the process of price competition, which is to be allowed only where it is deemed appropriate and where it will not harm quality of service.

    DOH 18/02/11

    There have been reports in the media that the Department of Health has changed its direction on the issue of price competition, based on David Nicholson’s letter published yesterday, in which he highlighted that ‘there will be no question of introducing price competition’.

    In response, a Department of Health spokesperson said:

    “There is no u-turn because we never intended to introduce price competition. There has been incorrect reporting around this and Sir David’s letter simply sets that straight.

    “There is no change to the policy set out in the 2011-12 Operating Framework and first set out under the previous Government in December 2009. Given the potential risk to NHS business planning of the incorrect speculation that this policy amounts to price competition, Sir David’s letter serves to reaffirm this policy – that we want to see competition on quality, not price. Anyone can see that it is difficult for the government to u-turn on a policy which it inherited from the previous administration and which it has not amended since coming into office.

    “The NHS Operating Framework published last December remains unamended.”

  58. msjhaffey said,

    February 18, 2011 at 2:13 pm

    @guy ok here we go

    “Politicians aren’t stupid”

    Demonstrably, many are. The basis for being elected has little correlation with IQ. Do you have any supporting evidence for your claim?

    [Politicians] “understand evidence perfectly well.” Proof? In the absence of evidence to support this statement, I’d suggest that some do and some don’t.

    “They just aren’t terribly honest!” I know several politicians of various stripes who are scrupulously honest. Again, can you prove your claim?

    “So yes there is no evidence. They know that.”
    Proof?

    “So they are just trying to hoodwink those who they can take in through this pretence.” Even if they did know there is no evidence, what proof do you have of their motives?

    “They are driven by dogma” Proof? I know again a good many who are driven but not much more than wanting to do a good job for their community.

    I don’t claim that politicians are all pure of heart. That wouldn’t be credible. Nor is it credible to claim the opposite, as you do. There are people who are dishonest, malign or deluded in all professions and it is lazy thinking to hurl generalisations in this fashion.

    Cold fusion, anyone?

  59. Guy said,

    February 18, 2011 at 3:25 pm

    Most of us who have moved on beyond playground bickering can tell the difference between

    1 evidence based statements eg drug XXX showed a 0.88 (CI 0.85-0.93) risk of death compared to placebo, and

    2 rhetorical statements like “Politicians aren’t stupid”.

  60. Bob Calder said,

    March 3, 2011 at 2:30 pm

    I attended a symposium where Gavin Schmidt spoke about the same problem. He refers to it as “scientifying” or applying scientific terminology in a political argument. A political argument is one where all knowledge is on the table and is traded from one player to another in a game process. When politicians see everything as a political asset, the values tend to be equalized so that “global warming” can be traded for a barrel of oil for instance. This is what I got out of the talk at any rate. He undoubtedly has a more nuanced view.

You must be logged in to post a comment.