There’s something magical about watching patterns emerge from data

June 23rd, 2011 by Ben Goldacre in bad science | 26 Comments »

Ben Goldacre
The Guardian
Saturday 11 June 2011

We all know one atom of experience isn’t enough to spot a pattern: but when you put lots of experiences together and process that data, you get new knowledge. This might sound obvious, but following it through – watching patterns emerge from the noise – still gives me a sense of beauty and awe.

A paper in the British Medical Journal this week is a perfect example. Medicine is an imperfect art, so it’s inevitable healthcare workers will make some suboptimal decisions: not so much the dramatic stuff, injecting people with the wrong drug, but more the marginal decisions, at the edges of the tweaks in a patient’s journey, affecting outcomes in ways that are harder to predict.

These kinds of complex decisions will inevitably be affected by context, and one example of that context is the franticness of A&E. Waiting times are a problem in a lot of countries. In the UK we introduced a four-hour ceiling as our target, and most hospitals reached it. Abolishing that four-hour target was one of the coalition government’s first NHS reforms. But do waiting times matter?

Some researchers in Canada decided to find out. They gathered data from all the people who visited any A&E department in Ontario over a five-year period: this gave them data on a dizzying 22 million visits. Of these, 14 million resulted in the patient being seen and then sent home. Then they followed these patients up to see what happened, and specifically, to see if they died.

But they also had another piece of information: for each patient, they knew, from internal hospital data, what the average waiting time in A&E was when they arrived. This means that they were able to compare the odds of death for patients discharged when the average wait in A&E was less than four hours (or more), against the odds of death for patients discharged when the wait was less than one hour. Remember, this isn’t the time that patient waited, it’s the average wait in the department, as a proxy for how frantic things were.

The results were as you might fear. For patients sent home who attended an A&E department when the average wait there was more than six hours, their odds of death were almost twice that of patients sent home when the wait was less than one hour. This odds ratio was similar for patients measured as high or low urgency at triage, so it’s true for patients with both serious and less serious presentations.

And even more starkly, there’s a very clear trend in the data, where each step up in waiting time results in a higher risk of death. This becomes statistically significant when average waits reach just three hours. For those who care about saving money, the odds of being admitted – and so using an expensive hospital bed – also rose dramatically as average wait time increased.

However important you might think those results are, I think some of the methodological issues are even more interesting, and they all arise because of the big numbers. Large datasets were vital, because these outcomes were rare: you only see a handful of deaths in every 10,000 patients sent home.

What’s more, because they had so many patients’ worth of data, the researchers were able to see an effect even within hospitals: so it wasn’t just that crap hospitals had longer waits, and higher death rates. What’s more, amazingly, they didn’t lose a single patient during followup: the death – or otherwise – of every single patient who was sent home from A&E could be tracked through their notes.

No individual patient or doctor could possibly have shown, with any certainty, from their own personal experience of one adverse outcome, that long waiting times in A&E are dangerous. This study is a remarkable testament to the power of good quality computerised health records, and the kinds of new knowledge you can generate from interrogating them. It’s also, I’ll agree, a pretty frightening result.

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

26 Responses

  1. RichJay said,

    June 23, 2011 at 7:31 pm

    I’ve recently been doing some work using all hospital admissions in England over the last 5 years. That’s 92 million records!

  2. iainfletcher said,

    June 23, 2011 at 9:16 pm

    As an “IT guy” this is partly what frustrates me about the blase attitude to “getting rid of wasteful IT projects” that seems to get unanimous support on both sides of the political spectrum. Only with computerised records can this kind of study be done, and easily.

    Also, I’ve seen first hand the damage done when patient information is not readily available. My fiancee was recently in and out of hospital with debilitating pain from gall stones. She could not take any painkillers by mouth and only intravenous morphine would give her comfort.

    Almost every time she was admitted through A&E with abdominal pain (at her worst on a daily basis, once within 3 hours of being discharged) she had to tell her story again, get the nurses to believe she wasn’t just being a drama queen again, wait for a doctor to proscribe proper medication when he turned up again, explain that she couldn’t eat any fat again, etc. etc.

    When this happened after only having left 3 hours previously, I was told by the nurse “oh, the records are sent for storage in another city as soon as the patient is discharged, we can’t access them”.

    Computerised records are vital for a whole host of reasons, and worth whatever cost it takes to get them implemented.

    (Sorry for the OT rant)

  3. davehodg said,

    June 23, 2011 at 10:30 pm

    Anyone tracking A&E wait times under the Tories?

  4. mikewhit said,

    June 23, 2011 at 11:30 pm

    Well, this Govt. did seem keen on crowdsourcing at one point, perhaps an ‘open source’ data collection system (like TheyWorkForYou) could be considered.

  5. boogywonderland said,

    June 24, 2011 at 3:53 am

    What a wonderful example of how a poorly chosen and overly simplistic proxy can lead to a completely fallacious conclusion. Even worse, the conclusion being drawn that ‘long waiting times in A&E are dangerous’ completely ignores the fact that waiting time was at best a proxy for how busy the department was. It’s crap research like this that leads to ridiculous strategies such as the 4 hour wait limit in the UK which was at best an expensive exercise in administrative obfuscation.

  6. AllanW said,

    June 24, 2011 at 8:48 am


    It would be irrational NOT to curtail wasteful projects (IT or otherwise) in any walk of life, wouldn’t it? The clue is in the adjective used. I’ve been a fierce proponent of information-based solutions to NHS organisation since Stafford Beers’ work and I don’t see or hear many people disagreeing with that path forward but projects that consistently deliver less than promised, that take longer than expected to arrive and/or fail to perform at all do not endear thinking people to support the next one coming down the ‘pike.

    Maybe it is the simple lack of comprehension displayed by many IT personnel of these facts and even of simple English phrases (such as focussing on the ‘IT projects’ part of a sentence rather than the ‘wasteful’ part) that makes us throw our hands up in the air in despair.

  7. Condorson said,

    June 24, 2011 at 9:43 am

    Crap proxy? Have you worked in A&E? I imagine Goldacre has, and his experience may be helpful in this regard.

    The research has shown a clear proportionate positive correlation b/t average waiting time at admission and mortality post-discharge.

    There are no doubt other factors at work, but the data has shown something which happens when waiting times change affects clinical judgement. It doesn’t say how.

    Perhaps the A&E staff were under pressure to keep the average time below a certain level, so when the patient flow was low, they exercised purely clinical concern, and fewer at-risk patients were booted out, but when waiting times increased due to greater patient flow, the staff were more inclined to either turf people upstairs to the wards or back home.

  8. Polaris said,

    June 24, 2011 at 10:58 am

    Ben –

    Does the analysis only show an association between waiting times and outcome or is there anything to suggest causality?

    (you were the first person to teach me the difference).

  9. Chris Neville-Smith said,

    June 24, 2011 at 12:00 pm

    Hmm … I’m been a bit sceptical about reliance on correlations myself. A correlation alone between waiting times and deaths doesn’t necessarily prove anything other than some hospitals being poor at both keeping waiting times down and keeping patients alive. I know that these kinds of surveys try to take this into account and only compare like-for-like data, but I’ve never been convinced you can account for all the variables – there’s always the possibility that there’s a factor skewing the results that you haven’t thought of, or can’t be found in the data.

    The problems with targets-based performance assessment is that it can underestimate the amount of abuse of the system. Believe me, there are ways of re-interpreting targets and policy and laws to fit what they’re doing instead of actually doing something about it (the case of that silly hospital who took the wheels off trolleys in order to count them as beds being one of the more piss-taking examples). I don’t have an answer for whether a waiting times target was a good idea – just that an over-reliance on this target might not give us that benefit we’d expect.

  10. boogywonderland said,

    June 24, 2011 at 5:53 pm


    Having worked in healthcare informatics for the last 12 years I have spent a huge amount of time working with exactly this type of data. I have further worked in large A&E departments and am very familiar not only with how such departments function but also how difficult it is to draw meaniningful conclusions from the resulting data. The issue is not the volume of data available but the insane number of confounding variables to control and the complexity of the data. The study attempted to control some of these variables but even attempting to do so was bold, nevermind imply causal relationships from the results.

    The data may demonstrate a significant correlation, it does not however demonstrate causality as is implied by the statement ‘long waiting times in A&E are dangerous’.

  11. finnfann said,

    June 24, 2011 at 6:33 pm

    One of the neat things about this study was that people who left without being seen were NOT at increased risk, but rather those who were seen and discharged during high crowing WERE at higher risk of mortality. Not sure what that means, but definitely an interesting, counter-intuitive finding.

    I think it is unfair that this study be labelled “crap” just because they failed to suggest causation. The fact that longer wait times are correlated to adverse outcomes is interesting in itself, and suggests a fruitful area for future research (ie. to try to pin-point the lurking variables that may be affecting increased risk).

    Also, the fact that this trend likely would not be visible without the incredible statistical power made possible by electronic records keeping is a testament to their usefulness. In Canada, believe it or not, there is significant resistance to properly developing a cohesive, nation-wide records system, and I think studies like this are a great case-in-point for why we need to do a better a job of implementing them. Although you’d think the abundance of stories like iainfletcher’s would be all the evidence they need…

  12. Big M said,

    June 25, 2011 at 10:38 am

    I have a problem with your conclusion.

    “when the average wait there was more than six hours, their odds of death were almost twice that of patients sent home when the wait was less than one hour.”

    “Large datasets were vital, because these outcomes were rare: you only see a handful of deaths in every 10,000 patients sent home.”

    “long waiting times in A&E are dangerous.”

    Increased chances of death are of course a bad thing, but if you compare survival rates instead of mortality rages the numbers will paint a different picture.

    You’re talking roughly a 0.05% mortality rate among those people sent home from A&E. That means that the survival rate is 99.95%. I don’t know how many hospitals had certain waiting times, but even assuming that they all currently had 1 hour waiting times, and then suddenly all changed to have 6 hour waiting times, the mortality rate would only increase to 0.1%, leaving a survival rate of 99.9%.

    Yes, Increased waiting times kill people, but the chance that increasied waiting times will kill any given individual is extrememly low. Given finite resources, I think it’s likely that those resources should be directed to other areas of healthcare where they can save more lives than are lost through increased waiting times.

  13. WilliamJay said,

    June 25, 2011 at 8:00 pm

    @boogywonderland said,
    June 24, 2011 at 3:53 am
    “Even worse, the conclusion being drawn that ‘long waiting times in A&E are dangerous’ completely ignores the fact that waiting time was at best a proxy for how busy the department was. It’s crap research like this that leads to ridiculous strategies such as the 4 hour wait limit in the UK which was at best an expensive exercise in administrative obfuscation.”

    Apparently the noble art of thinking is following the ancient art of conversation into the infirmary!

    If you think about it you’ll see that your comments are totally irrelevant. It’s irrelevant whether the patients died because the hospital was too busy to deal with them or because it didn’t bother.

    You’re right that waiting times are a proxy for how busy the department was.
    But a passing seven-year-old child should have been able to tell you that a busy department is a proxy for the priorities of a hospitals management.

    Casualty departments are as busy as management is willing to let them be.

    If hospitals have no target waiting time they’ll pull staff off the A&E, to do more lucrative things like hip-replacements, while the patients in A&E will be left for hours. If they have a target they won’t, and they’ll be pulling extra nurses and doctors in on overtime.

  14. zilbermann said,

    June 26, 2011 at 3:50 am

    In New York City wait times increase on Saturday nights when a lot of people get drunk, increasing crime and accidents. Thus the medical problems are different at the times when the wait times are longer. If fatalities are worse for people injured on Saturday nights one could not infer that this results from inferior medical care.

  15. Terry Collmann said,

    June 26, 2011 at 10:36 pm

    To put what Big M says another way, how much would have to be spent on extra resources to cut waiting times in order to save one extra life?

    And Ben: “their odds of death were almost twice that of patients sent home when the wait was less than one hour.” You’re doing the old “relative risk without giving actual risk” trick there – twice a really tiny risk is still a really tiny risk. What was the absolute risk of death, please?

  16. Big M said,

    June 27, 2011 at 9:38 am

    Just to add to what I said, ‘average waiting times’ were being used as a proxy for how busy the department was. It doesn’t automatically follow that arbitrarily reducing waiting times will reduce the rate of send-home deaths. It may even increase it if each patient isn’t getting as thorough an examination as they otherwise would have done.

  17. RogerMexico said,

    June 27, 2011 at 12:51 pm

    In the way that ‘average waiting time’ is a proxy for a busy A&E, might not a busy A&E be an indicator of a busy hospital? And in that case might staff be less willing to admit marginal cases for admission? And wouldn’t those marginal cases have a (slightly) higher risk of death that those sent home without question? And the longer the average wait, presumably the more likely marginal cases go home – there may be no bed to admit them to.

    After all, we are after all talking about fairly small differences in absolute probability here – if a very large number of cases is needed to detect the differences.

    Similarly the increased likelihood of admission when waits are long could be due to a number of things such as less serious cases getting fed up and going home or elsewhere before they were seen.

    When you get down to it, most studies tell us (and put numbers on) what we know already. If they didn’t something would be really wrong with our understanding of the world.

    What I do find implausible though is “they didn’t lose a single patient during followup”. No tourists, vagrants, illegal immigrants, people about to move abroad? I know this is Canada, but still. I suspect either these cases were omitted or patients without the full paper trail never made it on the system in the first place. Presumably they would be treated though, which puts a bit a query over the figures. Perfection is always highly suspicious.

  18. skyesteve said,

    June 28, 2011 at 10:34 am

    I fear there are too many variables at play here to make any meaningful conclusions. I say this as someone who, a number of years ago, worked in one of the busiest A&E departments in the country with well over 100,000 attendances per annum.
    Not all A&E departments are the same – some are found in local community hospitals and have very low waiting times but also seldom see seriously ill people; some are in district generals and some are in tertiary centres – and again there are differences in the case mix between DGHs and tertiary centres. Where I worked, for example, the most serious trauma was automatically taken to us, the tertiary centre.
    Moreover, large urban A&E departments tend to serve areas of higher socio-economic deprivation which is a clearly recognised cause of increased morbidity and mortality. They also have to deal with higher levels of things like alcohol and drug abuse and violence.
    There’s also a high dependence on the quality of staff that deals with you in A&E. Some staff have high levels of experience and acumen, others do not and that can have an effect on outcomes too, especially in busier departments. It’s the old “do you really want to be admitted in August when all the newly qualified doctors start their first proper job?” question.
    I could go on but the point is that, unless you have a way of eliminating all these other variables, I struggle to see how it is possible to make meaningful conclusions from waiting time data alone.

  19. chris lawson said,

    June 28, 2011 at 3:23 pm

    A lot of the questions raised in these comments are answered in the paper itself (see Ben’s link). This paper does establish a fairly convincing association between long ED waiting time and 7-day mortality. As a retrospective cohort study, it cannot demonstrate causality and it cannot exclude all possible confounders, but the researchers did try to account for them. For instance, they checked inter- and intra-hospital rates, and the overall odds ratio was adjusted for “triage level, age group, sex, calendar month, income fifth, urban/rural community, No of visits to emergency department in previous year, chief complaint, time/day of shift.” If I have a complaint, it is in the arbitrary conglomeration of 6 recorded triage levels into 2 strata, thus losing valuable information to an unnecessary compression, but the rest of the paper seems quite reasonable to me.

    I’m not sure why boogywonderland thinks waiting time is a bad proxy for how busy the department is. It strikes me as an excellent measure. I’d be interested to know of a better choice.

    The finding that puzzles me most is that the correlation seems just as strong for the low acuity group as the high-acuity. I would have expected the odds of death and admission to be considerably lower in the low-acuity group. This suggests to me that the triage system in Ontario does not measure patient risk all that well.

  20. jamesey said,

    June 29, 2011 at 4:23 pm

    Did the study examine real waiting times and real waiting conditions, or the fake waiting times as made-up by hospital administrators: the ones that lead to people spending up to 6 hours in a queue of ambulances outside A&E because the staff refuse to accept them onto the premises and start the 4hr clock running?

  21. db68 said,

    June 29, 2011 at 11:15 pm

    It seems the study shows that shorter wait times are better, but not necessarily that eliminating a hard humber for a target “maximum wait time” is a good idea.

    Consider a situation with a target wait time of “no more than 4h” and a hospital with acurrent average wait time record at 4h20. It seems that the likely response by management might be to push medical staff to reduce the time taken to deal with a patient rather than to hire the extra medical staff required, leading to slopier treatment and likely to increased mortality despite reduced waiting time.

    Don’t get me wrong: I believe we should work hard to reduce waiting times. I just think that removing the hard target may in some cases not be all bad…

  22. darzil said,

    June 30, 2011 at 7:03 pm


    One of the reasons there are so many IT projects coming under fire is that too often they are portrayed as IT projects, rather than what they usually are, which are major Business projects facilitated by IT.

    What should happen is that people decide they need information, work out who needs what information, work out who needs to input what information to provide that, get everyone agreed, then design the IT to meet the needs of the Business project.

    What happens it that people decide that IT will fix the issue, so commission a database/infrastructure, and then the design commences, trying to get time from business people to help with the ‘IT project’.

    I’ve never been involved in an IT project that has gone far over budget or time, but that’s because IT projects tend to involve system upgrades or replacements for stuff like network or EMail servers. Changing Business procedures using IT is NOT an IT project, or at least shouldn’t be! Just giving it that label is a recipe for disaster.

  23. @peterbrennen said,

    July 11, 2011 at 10:23 pm

    Whilst researching a job I applied for this week, I read this years NHS Operating Framework and noticed the newly added “A&E Quality Indicators (5 measures)”. Here is a link to the DoH documents about these,

    Going ahead during the 2011/12 financial year will be in the implementation and monitoring of these KPIs. It also clearly states that these replace the abolished 4-hour waiting time standard and need to be implemented from July 2011 onwards.

    This information is to be gathered over a couple of years as indicated in the letter frm DoH to All NHS Chief Execs (see link above). Perhaps these new measures could be used to create an improved proxy to address the concerns raised about the simplicity of the Canadian study’s proxy.

    Unfortunately I do not know at present how to go about legitimately gaining access to this data flow for the purpose of a nationwide study as I’d be very interested to organise one. As Ben rightly says, the Canadian study clearly indicates a statistically significant correlation and if we can look to improve on the perceived, or otherwise issues with that study, we could provide something from the UK health sector to compare it to. I could do some enquiring if anyone would be interested in such an idea?

    Keep up the good work Ben – enjoyed your book.


  24. thom said,

    July 24, 2011 at 7:51 pm

    Inferring that longer waiting times increase risk of mortality seems to be an example of the ecological fallacy (e.g., Robinson, 1950). It is perfectly possible for average waiting times (e.g., busy departments) to increase mortality risk but the waiting time of individual patients (within that average) to be uncorrelated (or negatively correlated) with mortality risk. This doesn’t make it a crap study – but more information is needed to understand this pattern.

  25. pjie2 said,

    August 1, 2011 at 10:59 pm

    No individual patient or doctor could possibly have shown, with any certainty, from their own personal experience of one adverse outcome, that long waiting times in A&E are dangerous.

    This study doesn’t show that either. The causality could be the other way round – when a pulse of high-risk patients comes in (the aftermath of a car smash, say), then waiting times will go up.

    One of the neat things about this study was that people who left without being seen were NOT at increased risk, but rather those who were seen and discharged during high crowing WERE at higher risk of mortality. Not sure what that means, but definitely an interesting, counter-intuitive finding.

    And how is that counter-intuitive? The A&E department will triage their intake and give beds to the most severely injured. When they’re under pressure, e.g. after an influx of severely injured patients, then some people who might otherwise have been admitted will end up getting sent home. I’d expect death rates to go up across the board when an A&E department is busy, because that’s what a A&E department does – it looks after people in danger of death. A busy A&E by definition indicates lots of people in danger of death!

    The only surprise is that there isn’t an elevated risk for the people that leave without being seen. One possibility is that these are people with minor injuries that really didn’t need to go to A&E in the first place.

  26. Medical Man said,

    August 26, 2011 at 2:06 am

    This is one reason I was upset to see Google Health fade away –

    The healthcare system is so broken in America that hospitals can sometimes have as many billing clerks as hospital beds. We need to find a way to not only speed up the patients processing BUT also make the ENTIRE system more efficient.

    I really believed Google health was going to make a difference and the system is so screwed up that they pretty much gave up.

    My solution is to avoid the hospital entirely if I can 🙂