Teachers! What would evidence based practice look like?

March 15th, 2013 by Ben Goldacre in evidence based policy | 53 Comments »

I was asked by Michael Gove (Secretary of State for Education) and the Department for Education to look at how to improve the use of evidence in schools. I think there are huge, positive opportunities for teachers here, that go way beyond just doing a few more trials: there is a need for a coherent “information architecture” that supports evidence based practice. I was asked to write something that explains what this would look like, specifically for teachers. Pasted below is the briefing note from DfE press office, and then the text of what I wrote for them, which came out this week. You can also download a PDF from the DfE website here.

If you’re interested, there’s more on evidence based policy in this BBC Radio 4 documentary I did here, and in this Cabinet Office paper on trials in government that I co-authored here, as well as zillions more posts.

There’s a response to my DfE paper from the Education Endowment Foundation here (they’re running over 50 trials in 1400 schools), and a blog post from the Institute of Education here, I’ll post up more when I get a chance.

Hope you like it!

Building evidence into education

Dr Ben Goldacre will set out today how teachers in England have the chance to make teaching a truly evidence-based profession.

Education Secretary Michael Gove asked Dr Goldacre to examine the role of evidence in the education sector.

In a paper to be presented at Bethnal Green Academy, Dr Goldacre will say today that research into “which approaches work best” should be embedded as seamlessly as possible into everyday activity in education.

High-quality research into what works best can improve outcomes, benefitting pupils and increasing teachers’ independence. But Dr Goldacre’s recommendations go beyond simply running more “randomised trials”, or individual research projects. Drawing on comparisons between education and medicine, he said medicine had “leapt forward” by creating a simple infrastructure that supports evidence-based practice, making it easy and commonplace.

Dr Goldacre says that:

– research on what works best should be a routine part of life in education
– teachers should be empowered to participate in research
– myths about randomised trials in education should be addressed, removing barriers to research
– the results of research should be disseminated more efficiently
– resources on research should be available to teachers, enabling them to be critical and thoughtful consumers of evidence
– barriers between teachers and researchers should be removed
– teachers should be driving the research agenda, by identifying questions that need to be answered.

In some of the highest performing education jurisdictions, including Singapore, he explained: “it is almost impossible to rise up the career ladder of teaching, without also doing some work on research in education.”

Dr Goldacre said:

“This is not about telling teachers what to do. It is in fact quite the opposite. This is about empowering teachers to make independent, informed decisions about what works, by generating good quality evidence, and using it thoughtfully.”

“The gains here are potentially huge. Medicine has leapt forward with evidence-based practice. Teachers have the same opportunity to leap forwards and become a truly evidence-based profession. This is a huge prize, waiting to be claimed by teachers.”

Background:

Ben Goldacre is a doctor, academic and writer who focuses on problems in science, statistics, and evidence based practice. His first book “Bad Science” sold half a million copies. He is currently a Research Fellow in Epidemiology at London School of Hygiene and Tropical Medicine.

 

And here’s the paper…

 

Building evidence into education

 

I think there is a huge prize waiting to be claimed by teachers. By collecting better evidence about what works best, and establishing a culture where this evidence is used as a matter of routine, we can improve outcomes for children, and increase professional independence.

This is not an unusual idea. Medicine has leapt forward with evidence based practice, because it’s only by conducting “randomised trials” – fair tests, comparing one treatment against another – that we’ve been able to find out what works best. Outcomes for patients have improved as a result, through thousands of tiny steps forward. But these gains haven’t been won simply by doing a few individual trials, on a few single topics, in a few hospitals here and there. A change of culture was also required, with more education about evidence for medics, and whole new systems to run trials as a matter of routine, to identify questions that matter to practitioners, to gather evidence on what works best, and then, crucially, to get it read, understood, and put into practice.

I want to persuade you that this revolution could – and should – happen in education. There are many differences between medicine and teaching, but they also have a lot in common. Both involve craft and personal expertise, learnt over years of experience. Both work best when we learn from the experiences of others, and what worked best for them. Every child is different, of course, and every patient is different too; but we are all similar enough that research can help find out which interventions will work best overall, and which strategies should be tried first, second or third, to help everyone achieve the best outcome.

Before we get that far, though, there is a caveat: I’m a doctor. I know that outsiders often try to tell teachers what they should do, and I’m aware this often ends badly. Because of that, there are two things we should be clear on.

Firstly, evidence based practice isn’t about telling teachers what to do: in fact, quite the opposite. This is about empowering teachers, and setting a profession free from governments, ministers and civil servants who are often overly keen on sending out edicts, insisting that their new idea is the best in town. Nobody in government would tell a doctor what to prescribe, but we all expect doctors to be able to make informed decisions about which treatment is best, using the best currently available evidence. I think teachers could one day be in the same position.

Secondly, doctors didn’t invent evidence based medicine. In fact, quite the opposite is true: just a few decades ago, best medical practice was driven by things like eminence, charisma, and personal experience. We needed the help of statisticians, epidemiologists, information librarians, and experts in trial design to move forwards. Many doctors – especially the most senior ones – fought hard against this, regarding “evidence based medicine” as a challenge to their authority.

In retrospect, we’ve seen that these doctors were wrong. The opportunity to make informed decisions about what works best, using good quality evidence, represents a truer form of professional independence than any senior figure barking out their opinions. A coherent set of systems for evidence based practice listens to people on the front line, to find out where the uncertainties are, and decide which ideas are worth testing. Lastly, crucially, individual judgement isn’t undermined by evidence: if anything, informed judgement is back in the foreground, and hugely improved.

This is the opportunity that I think teachers might want to take up. Because some of these ideas might be new to some readers, I’ll describe the basics of a randomised trial, but after that, I’ll describe the systems and structures that exist to support evidence based practice, which are in many ways more important. There is no need for a world where everyone is suddenly an expert on research, running trials in their classroom tomorrow: what matters is that most people understand the ideas, that we remove the barriers to “fair tests” of what works, and that evidence can be used to improve outcomes.

How randomised trials work.

Where they are feasible, randomised trials are generally the most reliable tool we have for finding out which of two interventions works best. We simply take a group of children, or schools (or patients, or people); we split them into two groups at random; we give one intervention to one group, and the other intervention to the other group; then we measure how each group is doing, to see if one intervention achieved its supposed outcome any better.

This is how medicines are tested, and in most circumstances it would be regarded as dangerous for anyone to use a treatment today, without ensuring that it had been shown to work well in a randomised trial. Trials are not only used in medicine, however, and it is common to find them being used in fields as diverse as web design, retail, government, and development work around the world.

For example, there was a longstanding debate about which of two competing models of “microfinance” schemes was best at getting people out of poverty in India, whilst ensuring that the money was paid back, so it could be re-used in other villages: a randomised trial compared the two models, and established which was best.

At the top of the page at Wikipedia, when they are having a funding drive, you can see the smiling face of Jimmy Wales, the founder, on a fundraising advert. He’s a fairly shy person, and didn’t want his face to be on these banners. But Wikipedia ran a randomised trial, assigning visitors to different adverts: some saw an advert with a child from the developing world (“she could have access to all of human knowledge if you donate…”); some saw an attractive young intern; some saw Jimmy Wales. The adverts with Wales got more clicks and more donations than the rest, so they were used universally.

It’s easy to imagine that there are ways around the inconvenience of randomly assigning people, or schools, to one intervention or another: surely, you might think, we could just look at the people who are already getting one intervention, or another, and simply monitor their outcomes to find out which is the best. But this approach suffers from a serious problem. If you don’t randomise, and just observe what’s happening in classrooms already, then the people getting different interventions might be very different from each other, in ways that are hard to measure.

For example, when you look across the country, children who are taught to read in one particularly strict and specific way at school may perform better on a reading test at age 7, but that doesn’t necessarily mean that the strict, specific reading method was responsible for their better performance. It may just be that schools with more affluent children, or fewer social problems, are more able to get away with using this (imaginary) strict reading method, and their pupils were always going to perform better on reading tests at age 7.

This is also a problem when you are rolling out a new policy, and hoping to find out whether it works better than what’s already in place. It is tempting to look at results before and after a new intervention is rolled out, but this can be very misleading, as other factors may have changed at the same time. For example, if you have a “back to work” scheme that is supposed to get people on benefits back into employment, it might get implemented across the country at a time when the economy is picking up anyway, so more people will be finding jobs, and you might be misled into believing that it was your “back to work” scheme that did the job (at best, you’ll be tangled up in some very complex and arbitrary mathematical modelling, trying to discount for the effects of the economy picking up).

Sometimes people hope that running a pilot is a way around this, but this is also a mistake. Pilots are very informative about the practicalities of whether your new intervention can be implemented, but they can be very misleading on the benefits or harms, because the centres that participate in pilots are often different to the centres that don’t. For example, job centres participating in a “back to work” pilot might be less busy, or have more highly motivated staff: their clients were always going to do better, so a pilot in those centres will make the new jobs scheme look better than it really is. Similarly, running a pilot of a fashionable new educational intervention in schools that are already performing well might make the new idea look fantastic, when in reality, the good results have nothing to do with the new intervention.

This is why randomised trials are the best way to find out how well a new intervention works: they ensure that the pupils or schools getting a new intervention are the same as the pupils and schools still getting the old one, because they are all randomly selected from the same pool.

At around this point, most people start to become nervous: surely it’s wrong, for example, to decide what kind of education a child gets, simply at random? This cuts to the core of why we do trials, and why we gather evidence on what works best.

Myths about randomised trials

While there are some situations where trials aren’t appropriate – and where we need to be cautious in interpreting the results – there are also several myths about trials. These myths are sometimes used to prevent trials being done, which slows down progress, and creates harm, by preventing us from finding out what works best. Some people even claim that trials are undesirable, and even completely impossible, in schools: this is a peculiarly local idea, and there have been huge numbers of trials in education in other countries, such as the US. However, the specific myths are worth discussing.

Firstly, people sometimes worry that it is unethical to randomly assign children to one educational intervention or another. Often this is driven by an implicit belief that a new or expensive intervention is always necessarily better. When people believe this, they also worry that it’s wrong to deprive people of the new intervention. It’s important to be clear, before we get to the detail, that a trial doesn’t necessarily involve depriving people of anything, since we can often run a trial where people are randomly assigned to receive the new intervention now, or after a six month wait. But there is a more important reason why trials are ethically acceptable: in reality, before we do a trial, we generally have no idea which of two interventions is best. Furthermore, new things that many people believe in can sometimes turn out, in reality, to be very harmful.

Medicine is littered with examples of this, and it is a frightening reality. For many years, it was common to treat everyone who had a serious head injury with steroids. This made perfect sense on paper: head injuries cause the brain to swell up, which can cause important structures to be crushed inside our rigid skulls; but steroids reduce swelling (this is why you have steroid injections for a swollen knee), so they should improve survival. Nobody ran a trial on this for many years. In fact, it was widely argued that randomising unconscious patients in A&E to have steroids or not would be unethical and unfair, so trials were actively blocked. When a trial was finally conducted, it turned out that steroids actually increased the chances of dying, after a head injury. The new intervention, that made perfect sense on paper, that everyone believed in, was killing people: not in large enough numbers to be immediately obvious, but when the trial was finally done, an extra two people died out of every hundred people given steroids.

There are similar cases from the world of education. The “Scared Straight” programme also made sense on paper: young children were taken into prisons and shown the consequences of a life of crime, in the hope that they would be more law abiding in their own lives. Following the children who participated in this programme into adult life, it seemed they were less likely to commit crimes, when compared with other children. But here, researchers were caught out by the same problem discussed above: the schools – and so the children – who went on the Scared Straight course were different to the children who didn’t. When a randomised trial was finally done, where this error could be accounted for, we found out that the Scared Straight programme – rolled out at great expense, with great enthusiasm, good intentions, and huge optimism – was actively harmful, making children more likely to go to prison in later life.

So we must always be cautious about assuming that things which are new, or expensive, are necessarily always better. But this is just one special case of a broader issue: we should always be clear when we are uncertain about which intervention is best. Right now, there are huge numbers of different interventions used throughout the country – different strategies to reduce absenteeism, or teach arithmetic, or reduce teenage pregnancies, or any number of other things – where there is no evidence to say which of the currently used methods is best. There is arbitrary variation, across the country, across a town, in what strategies and methods are used, and nobody worries that there is an ethical problem with this.

Randomisation, in a trial, adds one simple extra chink to this existing variation: we need a group of schools, teachers, pupils, or parents, who are able to honestly say: “we don’t know which of these two strategies is best, so we don’t mind which we use. We want to find out which is best, and we know it won’t harm us.”

This is a good example of how gathering good evidence requires a culture shift, extending beyond a few individual randomised trials. It requires everyone involved in education to recognise when it’s time to honestly say “we don’t know what’s best here”. This isn’t a counsel of despair: in medicine, and in teaching, we know that most of what we do does some good (if we’re not better than nothing, then we’re all in big trouble!). The real challenge is in identifying what works the best, because when people are deprived of the best, they are harmed too. But this is also a reminder of how inappropriate certainty can be a barrier to progress, especially when there are charismatic people, who claim they know what’s best, even without good evidence.

Medicine suffered hugely with this problem, and as late as the 1970s there were infamous confrontations between people who thought it was important to run fair tests, and “experts”, who were angry at the thought of their expertise being challenged, and their favourite practices being tested. Archie Cochrane was one of the pioneers of evidence based medicine, and in his autobiography, he describes many battles he had with senior doctors, in glorious detail. In 1971, Cochrane was concerned that Coronary Care Units in hospitals might be no better than home care, which was the standard care for a heart attack at the time (we should remember that this was the early days of managing heart attacks, and the results from this study wouldn’t be applicable today). In fact, he was worried that hospital care might involve a lot of risky procedures that could even, conceivably, make outcomes worse for patients overall.

Because of this, Cochrane tried to set up a randomised trial comparing home care against hospital care, against great resistance from the cardiologists. In fact, the doctors running the new specialist units were so vicious about the very notion of running a trial that when one was finally set up, and the first results were collected, Cochrane decided to play a practical joke. These initial results showed that patients in Coronary Care Units did worse than patients sent home; but Cochrane switched the numbers around, to make it look like patients on CCUs did better. He showed the cardiologists these results, which reinforced their belief that it was wrong of Cochrane to even dare to try running a randomised trial of whether their specialist units were helpful. The room erupted:

“They were vociferous in their abuse: “Archie,” they said “we always thought you were unethical. You must stop this trial at once.” … I let them have their say for some time, then apologized and gave them the true results, challenging them to say as vehemently, that coronary care units should be stopped immediately. There was dead silence and I felt rather sick because they were, after all, my medical colleagues.

Similar confrontations are reported in many new fields, when people try subjecting ideas and practices to fair tests, in randomised trials. But being open and clear about the need for research – when there is no good evidence to help us choose between interventions – is also important because it helps make sure that research is done on relevant questions, meeting the needs of teachers, pupils and parents. When everyone involved in teaching knows a little about how research is done – and what previous research has found – then we can all have a better idea of what questions need to be asked next.

But before we get on to how this can happen, we should first finish the myths about trials. From now on, these are all cases where people overstate the benefits of trials.

For example, sometimes people think that trials can answer everything, or that they are the only form of evidence. This isn’t true, and different methods are useful for answering different questions. Randomised trials are very good at showing that something works; they’re not always so helpful for understanding why it worked (although there are often clues when we can see that an intervention worked well in children with certain characteristics, but not so well in others). “Qualitative” research – such as asking people open questions about their experiences – can help give a better understanding of how and why things worked, or failed, on the ground. This kind of research can also be useful for generating new questions about what works best, to be answered with trials. But qualitative research is very bad for finding out whether an intervention has worked. Sometimes researchers who lack the skills needed to conduct or even understand trials can feel threatened, and campaign hard against them, much like the experts in Archie Cochrane’s story. I think this is a mistake. The trick is to ensure that the right method is used to answer the right questions.

A related issue involves choosing the right outcome to measure. Sometimes people say that trials are impossible, because we can’t capture the intangible benefits that come from education, like making someone a well rounded member of society. It’s true that this outcome can be hard to measure, although that is an argument against any kind of measurement of attainment, and against any kind of quantitative research, not just trials. It’s also, I think, a little far-fetched: there are lots of things we try to improve that are easy to measure, like attendance rates, teenage pregnancy, amount of exercise, performance on specific academic or performance tests, and so on.

However, we should return to the overly exaggerated claims sometimes made in favour of trials, and the need to be a critical consumer of evidence. A further common mistake is to assume that, once an intervention has been shown to be effective in a single trial, then it definitely works, and we should use it everywhere. Again, this isn’t necessarily true. Firstly, all trials need to be run properly: if there are flaws in a trial’s design, then it stops being a fair test of the treatments. But more importantly, we need to think carefully about whether the people in a trial of an intervention are the same as the people we are thinking of using the intervention on.

The Family Nurse Partnership is a programme that is well funded and popular around the world. It was first shown to be effective in a randomised trial in 1977. The trial participants were white mothers in a semirural setting upstate from New York, and people worried at the time that the positive results might have been exceptional, and occurred simply because the specific programme of social support that was offered had suited this population unusually well. In 1988, to check that the findings really were applicable to other settings, the same programme was assessed using a randomised trial in African-American mothers in inner city Memphis, and again found to be effective. In 1994, a third trial was conducted in a large population of Hispanic, African-American, and Caucasian mothers from Denver. After this trial also showed a benefit, people in the US were fairly certain that the programme worked, with fewer childhood injuries, increased maternal employment, improved “school readiness”, and more.

Now, the Family Nurse Partnership programme is being brought to the UK, but the people who originally designed the intervention have insisted that a randomised trial should be run here, to see if it really is effective in the very different setting of the UK. They have specifically stated that they expect to see less dramatic benefits here, because the basic level of support for young families in the UK is much better than that in the US: this means that the difference between people getting the FNP programme, and people getting the normal level of help from society, will be much smaller.

This is just one example of why we need to be thoughtful about whether the results of a trial in one population really are applicable to our own patients or pupils. It’s also an illustration of why we need to make trials part of the everyday routine, so that we can replicate trials, in different settings, instead of blindly assuming we can use results from other countries (or even other schools, if they have radically different populations). It doesn’t mean, however, that we can never trust the results of a trial. This is just another example of why it’s useful to know more about how trials work, and to be a thoughtful consumer of evidence.

Lastly, people sometimes worry that trials are expensive and complicated. This isn’t necessarily true, and it’s important to be clear what the costs of a trial are being compared against. For example, if the choice is between running a trial, and simply charging ahead, implementing an idea that hasn’t been shown to work – one that might be ineffective, wasteful, or even harmful – then it’s clearly worth investing some time and effort in assessing its true impact. If the alternative is doing an “observational” study, which has all the shortcomings described above, then the analysis can be so expensive and complex – not to mention unreliable – that it would have been easier to randomise participants to one intervention or the other in the first place.

But the mechanics and administrative processes for running a trial can also be kept to a minimum with thoughtful design, for example by measuring outcomes using routine classroom data, that was being collected anyway, rather than running a special set of tests. More than anything, though, for trials to be run efficiently, they need to be part of the culture of teaching.

Making evidence part of everyday life.

I’m struck by how much enthusiasm there is for trials and evidence based practice in some parts of teaching: but I’m also struck that much of this enthusiasm dies out before it gets to do good, because the basic structures needed to support evidence based practice are lacking. As a result, a small number of trials are done, but these exist as isolated islands, without enough bridges joining the people and strands of work together. This is nobody’s fault: creating an “information architecture” out of thin air is a big job, and it might take decades. The benefits, though, are potentially huge. Some individual randomised trials from the UK have produced informative results, for example, but these results are then poorly communicated, so they don’t inform and change practice as well as they might.

Because of this, I’ve sketched out the basics of what education would need, as a sector, to embrace evidence based practice in a serious way. The aim – which I hope everyone would share – is to get more research done, involving as many teachers as possible; and to get the results of good quality research disseminated and put into practice. It’s worth being clear, though, that this is a first sketch, and a call to arms. I hope that others will pull it apart and add to it. But I also hope that people will be able to act on it, because structures like these in medicine help capture the best value from the good work – and hard work – that is done all around the country.

Firstly – and most simply – it’s clear that we need better systems for disseminating the findings of research to teachers on the ground. While individual studies are written up in very technical documents, in obscure academic journals, these are rarely read by teachers. And rightly so: most doctors rarely bother to read technical academic journals either. The British Medical Journal has brief summaries of important new research from around the world; and there is a thriving market of people offering accessible summary information on new “what works” research to doctors, nurses, and other healthcare professionals. The US government has spent vast sums of money on two similar websites for teachers: “Doing What Works”, and the “What Works Clearing House”. These are large, with good quality resources, and they are written to be relevant to teachers needs, rather than dry academic games. While there are some similar resources in the UK, these are often short-lived, and on a smaller scale.

For these kinds of resources to be useful at all, they then need to land with teachers who know the basics of “how we know” what works. While much teacher training has reflected the results of research, this evidence has often been presented as a completed canon of answers. It’s much rarer to find all young teachers being taught the basics of how different types of research are done, and the strengths and weaknesses of each approach on different types of question (although some individual teachers have taught themselves on this topic, to a very high level). Learning the basics of how research works is important, not because every teacher should be a researcher, but because it allows teachers to be critical consumers of the new research findings that will come out during the many decades of their career. It also means that some of the barreirs to research, that arise from myths and misunderstandings, can be overcome. In an ideal world, teachers would be taught this in basic teacher training, and it would be reinforced in Continuing Professional Development, alongside summaries of research.

In some parts of the world, it is impossible to rise up the career ladder of teaching without understanding how research can improve practice, and publishing articles in teaching journals. Teachers in Shanghai and Singapore participate in regular “Journal Clubs”, where they discuss a new piece of research, and its strengths and weaknesses, before considering whether they would apply its findings in their own practice. If the answer is no, they share the shortcomings in the study design that they’ve identified, and then describe any better research that they think should be done, on the same question.

This is an important quirk: understanding how research is done also enables teachers to generate new research questions. This, in turn, ensures that the research which gets done addresses the needs of everyday teachers. In medicine, any doctor can feed up a research suggestion to NIHR (the National Institute for Health Research), and there are organisations that maintain lists of what we don’t yet know, fed by clinicians who’ve had to make decisions, without good quality evidence to guide them. But there are also less tangible ways that this feedback can take place.

Familiarity with the basics of how research works also helps teachers get involved in research, and to see through the dangerous myths about trials being actively undesirable, or even “impossible” in education. Here, there is a striking difference with medicine. Many teachers pour their heart and soul into research projects which are supposed to find out whether something worked; but in reality the projects often turn out to be too small, being run by one person in isolation, in only one classroom, and lack the expert support necessary to ensure a robust design. Very few doctors would try and run a quantitative research project alone in their own single practice, without expert support from a statistician, and without help from someone experienced in research design.

In fact, most doctors participate in research by playing a small role in a larger research project which is coordinated, for example, through a research network. Many GPs are happy to help out on a research: they recruit participants from among their patients; they deliver whichever of two commonly used treatments has been randomly assigned to their patient; and they share medical information for follow-up data. But they get involved by putting their name down with the Primary Care Research Network covering their area. Researchers interested in running a randomised trial in GP patients then go to the Research Network, and find GPs to work with.

This system represents a kind of “dating service” for practitioners and researchers. Creating similar networks in education would help join up the enthusiasm that many teachers have – for research that improves practice – with researchers, who can sometimes struggle to find schools willing to participate in good quality research. This kind of two-way exchange between researchers and teachers also helps the teacher-researchers of the future to learn more about the nuts and bolts of running a trial; and it helps to keep researchers out of their ivory towers, focusing more on what matters most to teachers.

In the background, for academics, there is much more to be said on details. We need, I think, academic funders who listen to teachers, and focus on commissioning research that helps us learn what works best, to improve outcomes. We need academics with quantitative research skills from outside traditional academic education departments – economists, demographers, and more – to come in and share their skills more often, in a multidisciplinary fashion. We need more expert collaboration with Clinical Trials Units, to ensure that common pitfalls in randomised trial design are avoided; we may also need – eventually − Education Trials Units, helping to support good quality research throughout the country.

But just as this issue stretches way beyond a few individual research projects, it also goes way beyond anything that one single player can achieve. We are describing the creation of a whole ecosystem from nothing. Whether or not it happens depends on individual teachers, researchers, heads, politicians, pupils, parents and more. It will take mischievous leaders, unafraid to question orthodoxies by producing good quality evidence; and it will need to land with a community that – at the very least – doesn’t misunderstand evidence based practice, or reject randomised trials out of hand.

If this all sounds like a lot of work, then it should do: it will take a long time. But the gains are huge, and not just in terms of better evidence, and better outcomes for pupils. Right now, there is a wave of enthusiasm for good quality evidence, passing through all corners of government at the moment. This is the time to act. Teachers have the opportunity, I believe, to become an evidence based profession, in just one generation: embedding research into everyday practice; making informed decisions independently; and fighting off the odd spectacle of governments telling teachers how to teach, because teachers can use the good quality evidence that they have helped to create, to make their own informed judgements.

There is also a roadmap. While evidence based medicine seems like an obvious idea today – and we would be horrified to hear of doctors using treatments without gathering and using evidence on which works best – in reality these battles were only won in very recent decades. Many eminent doctors fought viciously, as recently as the 1970s, against the very idea of evidence based medicine, seeing it as a challenge to their expertise. The case for for change was made by optimistic young practitioners like Archie Cochrane, who saw that good evidence on what works best was worth fighting for.

Now we recognise that being a good doctor, or teacher, or manager, isn’t about robotically following the numerical output of randomised trials; nor is it about ignoring the evidence, and following your hunches and personal experiences instead. We do best, by using the right combination of skills to get the best job done.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

53 Responses



  1. alistaircunningham said,

    March 15, 2013 at 5:46 pm

    An excellent suggestion and a thorough paper. In my world the drum has been heavily banged for Financial Education in Schools and whilst logical (how can education be bad?) I’ve yet to see any relevant, and more importantly scientifically conducted, evidence of its benefit.

    Wiley Online has articles with some empirical evidence both for and against, but nothing of this level of rigour. Committing any resource to a cash-strapped public sector should be carefully planned, with the relevant evidence supporting (or not) any interventions.

    For Financial Education to succeed something else will need to be axed for the curriculum.

  2. Xobbo said,

    March 16, 2013 at 5:28 am

    It’s probably a bit late now, but “evidence based” always needs a hyphen when used as an adjective.

    I would be surprised to see Michael Gove taking any action on this issue, but then I was also surprised that he requested the report in the first place, so fingers crossed. At the very least it should help to raise awareness.

  3. Amorvincit said,

    March 16, 2013 at 11:04 am

    Ben, you are a hero of mine and I agree with everything you say about evidence based practice, but I have a queasy feeling you have been had. You speak about “dry academic games” but there are slimy political games too and I think in this instance you are the one that has been gamed. In a rational world it wouldn’t matter where you say something and at whose behest. However, in this world, when you stand in an academy school and say you are doing what Mr Gove asked and that “right now, there is a wave of enthusiasm for good quality evidence, passing through all corners of government at the moment”. You are standing in a monument to one of the biggest untested interventions in schools, wresting them from local authority control and making them independent. By implication you are supporting academies and suggesting that any improvement (or otherwise) in those schools is attributable not to the money spent, new facilities and influx of staff but their independence of local authority control. What is the likelihood that teachers will have a better research grounding in academies when they now no longer need to have any teacher training to work in them (www.bbc.co.uk/news/education-19017544)? What is the likelihood that there will be participation in trials (which might reveal current practice to be harmful) and openness about the results (a matter rightly dear to your heart) when local authorities, an accountable body that with a statutory duty to care for many schools, are wholly replaced in England by competing schools as sensitive to their media profile as any drug company. Is it a cat in hell’s chance? When the loathsome Gove issues diktats about teaching kings and queens is that evidence based?

    By “all corners of government” are you suggesting that the massive changes to the NHS are evidence-based too? You don’t really believe that do you?

  4. sam.hindes said,

    March 16, 2013 at 11:34 am

    The Wikipedia example is an interesting one and one that brings to mind a recent resignation at Google where a lead designer complained that Google was testing 41 shades of blue to see which one was more popular. His complaint was not that this would lead to the wrong decision, but that this kind of decision making overall paralysed any individual designer’s creative input.

    An equivalent in a school might be in introducing more testing, in order to get more data, in order to more accurately judge the effectiveness of various approaches. Whilst this might be great for any individual test and for increasing the accuracy of our evaluations of practice, a few years down the line we might look back and find that there are cumulative effects of educational environments with greatly increased student testing.

    It is all very well to say that there are some outcomes which are easily measured (e.g. teenage pregnancy rates) but this also has real implications. If certain outcomes are easily measured, this in itself becomes a bias with practitioners gravitating towards practice which can be easily tracked against clear, specific and short term targets. Who is going to do the research into whether a medical model does something to alienate teachers from their practice, or students from their own learning? What if the ‘medical model’ isn’t just about effectiveness, but also about how teachers and parents understand students, and how students understand themselves?

    I have been following Bad Science for years, and always feel wiser for coming to the site. In this case, however, I feel that there is a worrying reductionism. Even if the research/teaching utopia you paint is the ideal, what we should be considering is what sort of dangerous compromises we might end up with in trying to get there.

  5. nikhalton said,

    March 16, 2013 at 12:16 pm

    So a difference between medicine and education that I’d like to explore here is that in medicine there is a pretty clear idea of what “good” looks like. We mostly know what evidence denotes a positive outcome (ability to remove symptoms or underlying causes better than placebo, no significant side-effects) and we know what evidence denotes a negative outcome (no improvement over placebo and/or significant side-effects).

    Is that the same in education?

    At a simplistic level, one can argue that an improvement in scores at targets at the end of an academic year/Key stage is a sign of positive evidence.

    However, there is then the issue of whether these measures are the right ones. Are we testing for the right things? Are we testing over the right time-scale?

    Are we even sure that the curriculum objectives from Early Years to tertiary education are right? How do we identify this?

    We can run any number of randomised trials to prove which of various methods produces the best results, but if the measures themselves are not a reliable of a “good education” then it’s all somewhat meaningless.

  6. Lindash said,

    March 16, 2013 at 7:04 pm

    What’s interesting is what’s been left out of this paper. Evidence-based teaching and learning has been a concern to many teachers for years yet there’s an implication in the paper that there’s been no attempt to back practice with evidence. I was surprised to find no references. John Hattie (New Zealand) and Robert Marzano (USA) have carried out meta-analyses of hundreds of research findings and the cause has been supported by people like Martin Hammersley and Geoff Petty in the UK.
    Teachers often don’t show any interest in finding evidence as to what works, but that’s because they’re too busy dancing to OFSTED’s tune and OFSTED changes its goalposts annually, so teachers are continually trying to catch up with the latest fad, whether it be flipped classrooms or teaching by standing on your head.
    If various governments had put an emphasis on evidence-based practice, and followed it up with money, it would have a higher profile. Instead they’ve thrown money at a succession of projects that clearly weren’t evidence-based. Take a look, for example, at Subject Learning Coaching (disappeared without trace) or SEAL (because every child in the country must have esteem issues and every teacher must be the best person to sort them out.)
    Anyway, moocs will change the world – lots of research going on there, so lots of evidence for us to peruse.

  7. AnthonyEcc said,

    March 17, 2013 at 11:50 am

    This is an interesting article, Ben. Thanks for de-bunking what I thought was a pointless restriction on educational trials.
    I am a Trainee teacher and as part of the course we have to produce an open study on any topic we wish but…. we were told by the University lecturers that the implementation of an intervention e.g. applying a particular teaching method to one group and not another is ethically questionable. I did think at the time that this restriction was unsound. I for one believe that it is more ethically unsound to be prevented from trying something that might improve education. Paradoxically, it is for this reason that it was deemed inappropriate to perform such trails – because other classes wouldn’t be receiving the same treatment!! I despair.

  8. Workducker said,

    March 17, 2013 at 4:49 pm

    But this approach has no foundation as there is no consensus as to the desirable outcomes from our education system!

  9. JulianWells said,

    March 18, 2013 at 6:57 pm

    Ben’s overlooked the fact that he is proposing experiments with *classes*, not with individuals, and that the guardians of each child will have to give consent to their participation.

    In England there’s a legal limit of 30 on infant class sizes, and comparing two treatments to be compared requires (at least) two classes; presumably maximum size classes are best to account for random variation among the children.

    Suppose the probability that guardians refuse consent for the trial is 0.1; then the probability of all consenting is

    0.9^(30+30) = 0.00179701

    On the (highly-optimistic?) assumption that the refusal rate is 0.01, then chance of all consenting = 0.5471566

    Or to look at it another way 0.5^(1/60) = 0.988514 …

    Conclusion: (ethical) educational RCT will never happen.

  10. Ben Goldacre said,

    March 18, 2013 at 7:54 pm

    Hi JulianWells.

    “educational RCT will never happen”

    As I mention above, the Educational Endowment Foundation are now running 50 randomised trials, in 1,400 schools, covering over 200,000 pupils.

    They do happen, in the UK, and in the US, and elsewhere.

  11. JulianWells said,

    March 18, 2013 at 10:16 pm

    If Ben would take the trouble to quote me correctly, he would have to give a different answer: “*ethical* educational RCT will never happen”.

    I don’t doubt that educational RCT is happening: what I doubt is whether these experiments have the same ethical foundation as medical RCT — namely informed consent by the subjects (or, presumably in this case, their parents/guardians).

    A Google search for “success for all” + “randomised trials” + “consent” gets 20 distinct hits. So far as I can see none of them refer to the EEF experiments, or to their U.S. counterparts.

  12. Carloper said,

    March 19, 2013 at 10:04 am

    Apologies about the lengthy response, but this is a topic I care very much about.

    I respect and share Ben Goldacre’s passion for evidence-based research, but I find myself in disagreement with much of what he says. For the most part, my argument draws on the critique articulated by Kenneth Howe a few years ago. I strongly recommend Ben and whoever is interested in this topic to read his paper, which is freely available.

    Some of the points Ben makes are very problematic and should be unpicked a bit further.

    The idea that randomised controlled trials can free teachers from the shackles of political interference is debatable. As Ben also suggests, RCTs are only suited to a limited range of research questions. The problem is that, historically, these questions have favoured the control of internal validity over external validity. Ben’s paper warns about the dangers of exaggerated claims (in the Family Nurse Partnership example), but the fact remains that experimental research puts great emphasis on internal validity at the expense of external validity, with some serious consequences. As Howe reasons: “Among the major drawbacks of randomized experiments are problems with external validity, including inconsistency in implementing interventions across contexts (“dispensing a curriculum” is not quite the same as “dispensing a pill”). There is a trade-off between internal and external validity: The more investigators restrict the population and the treatment to achieve internal validity, the less external validity the study will have” (Howe, 2004: 45). The premium that RCTs place on internal validity “encourages educational researchers to focus on easy-to-manipulate, simplistic interventions and to avoid questions about existing policy and practice that for one reason or another, are not suited to being investigated via randomized experiments (…) this is the research-methodology tail wagging the educational-practice” (ibid: 45). The result is that RCTs can only be conducted at the margins of policies; they won’t be able to sustain a critical analysis of the assumptions (often ideological) that have informed those policies to begin with. As such, RCTs may paradoxically contribute to the disempowerment of teachers and researchers, as questions of meaning, purpose and value are inevitably shoved under the carpet for being “intractable”.
    The emphasis on “what works” assumes that causal relationships can only be ascertained through an experimental approach; while “qualitative” (tellingly in scare quotes in Ben’s paper) research plays second fiddle to complement and enrich the already ascertained link (“why it worked”). I think that, in this instance, Ben has chosen to ignore reasoned arguments, such as Howe’s, that can be brought to bear in the debate, and went instead for the dismissal of “researchers who lack the skills needed to conduct or even understand trials”. This feels unfair. Qualitative research in education can indeed be used to make robust inferences about causation. These are more useful for practice than the bare certainty that “A has caused B”. Take for instance the following example from Howe (who in turn draws on the thought of John Searle): “According to John Searle (1984, 1995), human behavior must be understood against a complex background of “intentionality” that defines norm-regulated practices. Documented regularities among descriptive variables do not constitute causal explanations of human behavior; they call for them (Searle, 1984). Searle turned the typical, experimentalist construal on its head: Quantitative findings documenting regularities constitute the auxiliary, discovery work; filling in the black box requires investigating matters best handled with qualitative methods. Take the following example. We begin with the observed regularity that African American students living in “Trackton” exhibit low academic achievement. This regularity in and of itself is not a causal explanation of anything. To provide such an explanation, we conduct an ethnographic study that gets at the perceptions and practices of the actors involved. We conclude that the differences among the linguistic practices of African American students and their White teachers cause distorted communication between them that in turn, causes lower academic performance on the part of the students” (ibid: 53).
    I would like to suggest to Ben that championing a research method like RCTs in education is not the same as championing rigorous evidence-based research in medicine. Education is a deeply political and value-laden domain, and discussions about research methods should always beg the question: “whose interests are being served?” Howe compellingly uncovered the ideological, largely conservative roots of the emphasis on RCTs in education. By their very nature, RCTs only allow researchers and teachers to tinker with the system as it is, while discouraging critical and non-normative research questions. The choice to favour a technocratic (“what works”) ideal of education is not a neutral one. I hope the reader will forgive the extensive quoting of Howe’s paper, but he articulates the problem much better than I could ever dream of: ““What works” is not an innocent notion. Concrete instances of the claim ‘Intervention I works’ are elliptical for instances of the claim ‘Intervention I works to accomplish outcome O’. The desired outcomes are embraced (if only tacitly) as more valuable than other possible outcomes, and the question of their value is off the table for anyone except policy makers and researchers. In this way, the aim of determining “what works” is technocratic: it focuses on the question of whether interventions are effective in achieving given outcomes (…) broader questions about values are lurking, even in a relatively uncontroversial area such as math education. For example, what approaches to math curricula and instruction best prepare students to become competent democratic citizens? What approaches to math curricula and instruction are least likely to be exclusionary of certain kinds of students? What approaches to math curricula and instruction are most likely to make students critical mathematical thinkers and to foster a healthy skepticism of mathematics as an all-purpose intellectual tool? What trade-offs are to be made among mathematics and other subjects? And so on. These are not technical ‘what works’ questions” (ibid: 54-56).

    Finally, I would like to highlight the irony of having Mr Gove commissioning such papers, while much of the educational policies of the coalition government has been advocated on grounds that can only be defined ideological. These policies have in some cases been informed by eminently “charismatic” thinkers rather than experimental studies. Lest we forget that that E.D Hirsch’s idea of cultural literacy exercised a great influence on Mr Gove’s approach to curriculum reform. In this respect, the emphasis on RCTs may actually serve a deeply political purpose, in that the forms of inquiry allowed on the “shop floor” of education are those that focus only on neutered matters of execution, avoiding the exploration of ideological influences and political agendas.

    All quotes are from the following paper:

    Howe, K. (2004). A Critique of Experimentalism.
    Qualitative Inquiry, 10 (4), 42-61. Copyright © 1998
    by Sage Publications.

  13. GrahamSimons said,

    March 19, 2013 at 1:18 pm

    I am nowhere near as pessimistic as JulianWells. In many schools individual teachers adopt their own strategies with their classes. They don’t ask parental permission for this. No school, I imagine, asked parents whether they wanted their children to do Brain Gym in lessons – or to sit to one side while their classmates did it.

    So the opportunities exist.

    The issue of deciding favourable outcomes is tougher. If it’s exam results or similar, no problem. If it’s life chances and so on, the results of the trials could take a lifetime to com through: and there are not that many research projects of that duration!

    That said, some good research might well cut through a great deal of the uninformed discussion and policy-making.

    Thank you

  14. clt47 said,

    March 19, 2013 at 3:28 pm

    I can see plenty of teachers begging on bended knees to put most of the currently imposed classroom practises through randomised trials. A quick perusal of the TES teachers’ forums will show how many teachers resent methods, especially those that seem to value paperwork over actual teaching, being imposed with no evidence to back them up.

    A body of good evidence on educational development, not just the median or typical, but the range of development at a given age, would be a useful tool. If testing were for this purpose and fit for purpose, rather than for compiling league tables or for political purposes, there would be some point to it.

    My personal first choices would be ‘smart’ targets and ‘ILP’ driven teaching for adult language learning, because I never could find actual evidence for the benefits of either of these, despite asking the managers who imposed them and spending considerable unpaid time online trying to find any evidence for either. The dogmatic imposition of teaching methods was one factor in my decision to retire from adult education.

    I would have liked an opportunity to research effective methods of teaching writing skills to adult second language learners, including RCT, but could not find a valid way to do this. My employers were not interested, and as a sessionally paid, part time adult education teacher, post graduate studies were beyond my means. Even observational studies were hard to find.

  15. ewan said,

    March 19, 2013 at 5:51 pm

    What’s fairly astonishing about a lot of the opposition, neatly exemplified by JulianWells, is that it’s seen as perfectly acceptable for teachers to do things with no evidence base whatsoever, but running a trial to get an evidence base is somehow an ethical concern.

  16. AnthonyEcc said,

    March 19, 2013 at 9:26 pm

    Is it better, i.e. ethical, to blanket-adopt a particular initiative pushed by Ofsted/Estyn(Wales) with very little in the way of RCT just because they say we should? I, for one, don’t think so.

  17. JulianWells said,

    March 20, 2013 at 9:43 pm

    Someone like Ewan, who claims to be in favour of paying attention to the evidence, might care to apply this approach to what I said, rather than imputing to me views on matters that I did not discuss.

    I suggested that it would be difficult to get consent for educational RCT on the same basis as for medical RCT and thus put it on the same ethical footing.

    One may, of course, argue that some different ethical criterion applies to educational experimentation as opposed to medical experimentation. (If so, what is it?)

    Nonetheless, it seems crystal-clear that large scale educational RCT is occurring without the informed consent of the subjects or their representatives.

    It also seems crystal-clear that some RCT enthusiasts are entirely uninterested in giving their experiments a clear ethical basis.

    This is a pity, because without it there will sooner or later be an obscurantist backlash against educational RCT which science will be ill-prepared to counter.

  18. Phil Parker said,

    March 20, 2013 at 10:30 pm

    A few thoughts I left on the facebook page.

    Ben I wonder why you discuss only randomized control trials when educational research has been at the forefront of advances in causal inference for some years now. For example, I wonder why there is no disccusion of research which has supported clear treatment effects utalising propensity score matching, regression of discontinuity, or instrumental variables. I agree with what you are saying for the most part but there are many occasions in education where randomized treatment assignment is not feasible. For example the treatment effect of university entry, repeating a school year, attending a private versus public school do not leand themselves well to control trials but research using methods above, developed largely from potential outcomes theory, have been effectively carried out in education. Educational research often consists of well meaning but largely quantitatively ignorant researchers. However, there is a sizable core who are at the forfront of causal inference research and I think it is a shame that these developments were missed in your report.

    I suppose my concerns are that by only discussing RCT people may get the impression that such research is the only avaliable avenue to derive causal effects. It is my experience that this can lead to an unhealpful climate that leads those unfamiliar with advances in causality research and theory to believe that only RCT research provides evidence of caluality and thus dismiss research that is not RCT but still provides similar evidence about the efficacy of a given treatment effect.

    P.S. While I am at it there are complexities of RCT that are not mentioned in your report but are critical for educational science to move forward. For example, what unit of analysis should randmoization occur at? Much of the RCT research in education conducts randomization at the student or teacher level. This can be clearly inappropriate when it leads to violations of stable unit treatment value assumptions that can make many within school RCT questionnable. It is thus critical that RCT not be rolled out unthinkingly within schools where randomization at the school level is required. As such investment in RCT will require huge investment on the part of governemnt funding bodies like UK Economic and Social Research Council so that randomization can be carried out at the level of the school (often needing N=50 or more).

  19. Phil Parker said,

    March 21, 2013 at 12:21 am

    Jullian said “Nonetheless, it seems crystal-clear that large scale educational RCT is occurring without the informed consent of the subjects or their representatives.”

    I am not really sure where you are getting this from. Most educational research is conducted via universities where all research projects involving humans are required to lodge an ethics application and have it approved by an ethics board that consists of a broad cross-section of relevant stakeholders(not an easy process I assure you). Informed consent is a requirement of such ethics application as is informed parental consent for all children under the age of 16 (except in very special circumstances). In addition, grant funding also requires research be approved by an ethics board as do many/most journals. It is highly unlikely that any RCT conducted in western countries and in published literaure will not have gone through an ethics application process.

  20. JulianWells said,

    March 21, 2013 at 9:22 am

    As a university lecturer I’m well-aware of the existence of ethics boards.

    However, I’m still waiting for someone to confirm that the EEF experiments that Goldacre promotes so relentlessly have gone through such a process.

    As shown by the calculations in my original post, the probability of this being so appears to be vanishingly small.

  21. cellocgw said,

    March 21, 2013 at 1:25 pm

    Hey, I’d settle for convincing MDs *today* to accept evidence-based decisions. A certain pediatrician, who I’ll anonymize by calling her “my wife” :-), has to deal with colleagues on a daily basis who, in response to incredibly clear-cut data showing the right way to go, say “Oh, I’m just not comfortable with that. ” There’s still a lot of MDs who see change as a threat to their authority or capability, and resist mightily.

  22. John_Clarke said,

    March 21, 2013 at 1:52 pm

    I am worried about Ben G. coming in on Goves ticket. Ben is an expert on medical randomised trials and a great supporter of science. So am I. However, as someone in Education Research with 30+ years teaching experience, 25 in schools at the chalk board, I can say that in educational situations one cannot mostly get a situation constructed around randomised trials. Overall the variables themselves change each time the experiment is run; the kids aren’t the same, the teachers aren’t the same the situation dynamic isn’t consistent. Education isn’t medicine.
    Ben has a bee-in-his-bonnet about randomised trials and is being manipulated by Gove to pretend that teachers are against evidence. We aren’t, evidence is produced everyday in classrooms concerning how things are taught but the evidence demonstrates there aren’t universal outcomes. One cannot use randomised trials when the trials themselves are chaotic in behaviour. If each time a medicine was taken the outcome depended on the time of day, the day of the week, what the subject had just done, what the subject was about to do, what the person who handed the pill had just done and was about to do, what colour the pill was, how many pills the subject had taken from this particular handing-outer etc etc……Nobody would have a clue which medicine to take. Sorry but Gove is using Ben to pretend he, Gove, uses evidence.
    Evidence suggests otherwise……..(1) Current evidence suggests academies do not produce exam results any different from Local Authority schools, despite Gove saying the opposite. Did he listen to evidence? No………(2) Of the first 9 Free Schools inspected by Ofsted 33% are in need of improvement. This is higher than the national figure for Local Authority schools. Evidence suggests Free Schools have been a waste of valuable resources. Is Gove changing course? No. (3) Evidence in Teacher Training shows less teachers are being trained at a time more teachers are needed. Has Gove altered his direction. No.
    How many examples would Ben like before he concludes Gove doesn’t “do” evidence?

    I generally like Ben G.’s work but aligning himself with Gove and his FALSE use of evidence does Ben no favours.

  23. RAllison said,

    March 21, 2013 at 4:40 pm

    Enjoyable article – I think more would be inclined to teach if more freedom to improve (and *show* it) like this was a part of the job.

    Building an evidence base for effective teaching strategy seems like a very worthwile use of time, effort and money.

    JulianWells – I’m not sure I agree with your statistics… Yours is a simple model, a prior or two may improve the realism of your numbers. It is probably not one to hang your ‘lack of ethics’ arguement on.

    I’m also unsure if one could sensibly argue that parents even know what is best for their children in an educational setting (and how could they, seeing that there is no *evidence* on which they can base thier opinions) – why should uninformed parents even have a say?

    John_Clarke – I would imagine that some (most) of the variables that you mention would come out in the wash, and i think i would be even more surprised if they had a significant impact – although I am happy to stand corrected, you experience in schools far outstrips my own!

    I’m not sure I agree with two of your evidences that Gove ignores evidence: 2) The free schools are new, and are only finding thier feet. Considering that there are still so few of them i would cut a little bit of slack for them, before we slam them; 3) I’m not sure which of Gove’s policies you think are leading to the down turn in teacher training numbers? I think there are probably other factors that are the effect makers there?

  24. John_Clarke said,

    March 21, 2013 at 5:34 pm

    ewan,

    Quote: “What’s fairly astonishing about a lot of the opposition, neatly exemplified by JulianWells, is that it’s seen as perfectly acceptable for teachers to do things with no evidence base whatsoever, but running a trial to get an evidence base is somehow an ethical concern.”

    No evidence base eh? Teachers use EXPERIENCE, that experience is not evidence free but it hasn’t been subjected to rigorous peer review by published articles. It has, however, been peer reviewed by other teachers in the form of observation and reflection.

    To be effective a Randomised trial must be randomised; given that evidence shows that teaching is effective in well developed relationships between teacher and pupil, how exactly do you randomise my year 10 maths set? If I teach two sets different ways, how do I control for all the other variables other than change-of-technique?

  25. John_Clarke said,

    March 21, 2013 at 7:02 pm

    RAllison,
    Quote: “I think more would be inclined to teach if more freedom to improve (and *show* it) like this was a part of the job.”

    It is and always has been part of the job. I think you reasons for teaching are a bit flawed though.

    Quote: “Building an evidence base for effective teaching strategy seems like a very worthwile use of time, effort and money.”

    But we already have THAT, it is called experience.

    Quote: “John_Clarke – I would imagine that some (most) of the variables that you mention would come out in the wash, and i think i would be even more surprised if they had a significant impact – although I am happy to stand corrected, you experience in schools far outstrips my own!”

    And it outstrips Ben, Gove and many other self appointed ‘experts’. You imagine, would you? Very randomised, control then!

    Quote: “I’m not sure I agree with two of your evidences that Gove ignores evidence: 2) The free schools are new, and are only finding thier feet. Considering that there are still so few of them i would cut a little bit of slack for them, before we slam them; 3) I’m not sure which of Gove’s policies you think are leading to the down turn in teacher training numbers? I think there are probably other factors that are the effect makers there?”

    You are ignoring the evidence on Free Schools because it doesn’t suit your agenda. There is additional evidence from Scandanavian countries which have abandoned the idea of Free Schools, the same countries Gove used to justify his policy in the first place. Then again you aren’t interested in evidence are you?

    Gove’s current policies on ITT (Initial Teacher Training) are all about moving from a Nation based scheme looking at overall numbers and run through university PGCE courses to School based, local schemes run through schools, which are firstly more expensive and secondly not capable of training large numbers of shortage subject teachers. Over the last two year to three years evidence has shown trained maths teacher numbers have dropped from c3000 to c2500 and are due a ‘worrying’ drop again this year. This followed several successful years of increasing maths trainees recruitment on PGCE courses; universities HAVE the capacity, but schools don’t, but THAT doesn’t fit Goves ideological agenda. This year he may close SKE courses in maths which allow access to maths teaching with degrees in maths; more dramatic falls in numbers will follow. In 2015 the population bulge, currently in the Primary sector, starts to feed through to the Secondary sector. A perfect storm of reduced teacher numbers in maths and science will arrive just as Gove is kicked out of office! Are you going to ignore all THIS evidence?

    So much for an evidenced based ways of working eh?

  26. namay said,

    March 21, 2013 at 8:19 pm

    I have always enjoyed Ben’s ability to cut through the ‘bull’ that surrounds so many fashions and trends in many different areas. Much so-called educational research seems totally devoid of the concept of ‘scientific method’ (start form intuitive feeling about what works, then work back from there), so applying some may not be a bad idea. However, as alluded to by other contributors, trying to ensure that any ‘trial’ is truly randomised in education is going to be like nailing down a blancmange. Gove is the worse at foisting ideas from some sort of educational Jurassic Park onto the pupils and teachers of today. No improvement s are going to be effected against background of vindictive and ideologically & politically motivated attacks against teachers (not too many Tory votes to be lost there!), their unions, and the ‘educational establishment’.

  27. Phil Parker said,

    March 21, 2013 at 9:48 pm

    I wonder how much research from the Journal of Educational Psychology, Learning and Instruction, etc. you guys have to read to be so confident that educational research is devoid of scientific merit?

    I like Ben Goldacre a lot and have brought Bad Science on seven different occasions to give to friends BUT surely a report on educational research should come from an educational researcher who knows the state of the field. It is not as if there is a shortage of excellent educational researchers in the UK who are experienced with RCT and causal inference research in education.

    On a side point it is perfectly reasonable to conduct RCT in education. The trick is that most of these need to be randomized at the school level, often requiring 50 or more schools. When we can get sufficent funding, this is exactly what we do at my institute.

    The reason for this is that an assumption of RCT is that one person’s treatment assignemt should have no effect on another person’s treatment outcome. The problem with within school experiements is that you are conducting a RCT within an established community with well developed lines of communication. Thus when assigning classes or children to different treatment conditions in the same school it is almost impossible to insure diffusion will not take place (kids talk in the playground, teachers in the staff room, in high school kids move from class group to class group and from teacher to teacher). Hence in many cases RCT where randomization occurs within school will violate assumptions and results will potentially be biased.

  28. John_Clarke said,

    March 21, 2013 at 9:53 pm

    Namey, Gove is dragging people into his artificial ‘debate’ on evidence to disguise the fact he NEVER uses evidence. Ben has fallen for this hook line and sinker.

    How many Hawthorne experimental effects will be recorded and subjected to great debate before Ben, and Gove, realise that the social interaction which is called teaching is a complex multivariative interaction more akin to the mathematical chaos of weather prediction than a simple science experiment.

    I am a mathematician and I used to be very quantitative in my outlook, but once you realise that teaching and learning are both qualitative experiences not always linked in obvious ways you realise that Goves idea that teaching causes learning, full stop, and that both the teaching and learning can be measure, quantified and linked is soooooo naive as to be laughable.

    Quote: “Much so-called educational research seems totally devoid of the concept of ‘scientific method’ ” As someone trained in ‘scientific method’ I realise that rocket science is ‘easy’ because it is susceptible to the scientific method; however something as ‘simple’ as investigating why Fred in Mr. Smiths class got a higher mark in a test than Stan in Mr. Siddons class really is harder to explain than rocket science!

  29. PaulJH said,

    March 22, 2013 at 3:14 pm

    Nobody seems to have mentioned the evidence based teaching work by John Hattie (Visible Learning)or Marzano (Classroom instruction that works, though the 2nd ed. was ed. by Dean). Both are quite good, based on meta analyses, though they give slightly different recommendations.
    If you can do RCT and then evidence based psychotherapy, you can do it for teaching.

    Paul

  30. namay said,

    March 22, 2013 at 11:12 pm

    I feel that you are ‘right on the money’ with your observations, John Clarke. As Ben will be only too aware, we have frequently have ‘new’ ideas foisted upon us in education, many of which the more experienced practitioners recognise as the same old ideas in new clothes. No-one (at least, hardly ever) has the balls to challenge obvious crap, since it is not a great career move to do this (and the new STPCD will further suppress critical debate). I remember sitting thorough a whole training session on “brain gym’ thinking, “this is a load of bull..”, but not challenging it openly. Some build their careers on jumping a trend in education WHETHER THEY ACTUALLY BELIEVE IT OR NOT. In fact there is no shortage of what I regard as a type of educational parasite who go around earning a small fortune from offering training courses/days to schools, based upon old ideas dressed up as something different. The ‘research’ on T&L ‘Working inside the black box’ seemed largely devoid of any clear scientific method, but rather seemed to derive from intuitive feelings which the ‘researchers’ then looked to find the evidence to back up. The educationalist I have the most regard for who seems to have a very clear understanding of what learning involves is John Yandell at the Institute of Education (Univ of London) – he regards the OFSTED (and Gove’s) view of learning as linear and reductive – I believe he is right. In education we are in my experience becoming so obsessed with the process and box-ticking, and very unreliable data, that we lose sight of the outcomes. In the NHS, as Ben will appreciate, this same approach has more severe consequences in that it kills people. I do not entirely discard the idea that there MAY be a way of analysing impact of teaching, but I remain to be convinced that ALL possible variables may be excluded by the methods used.

  31. JulianWells said,

    March 23, 2013 at 4:22 pm

    Phil Parker and I are, I think, on the same side, in that we recognise the potential in RCT while aware of possible difficulties.

    What I’m still waiting to hear, however, is the ethical basis on which whole classes in 50 schools have been enrolled in the trials.

  32. ewan said,

    March 24, 2013 at 1:55 pm

    Maybe you could explain the ethical basis on which people are subjected to current, unevidenced, interventions?

    You seem to think that doing an intervention just because a teacher feels like it is OK, but as soon as there’s some structure of a trial put around it it suddenly requires a different, much higher, degree of consent – can you explain why that is?

    Participants in a medical trial have to consent, but then so does any patient outside of a trial too; with the exception of emergencies, a medic cannot subject anyone to any intervention without informed consent. The same is not true in education.

  33. John_Clarke said,

    March 25, 2013 at 8:22 am

    Ewan,

    Quote: “Maybe you could explain the ethical basis on which people are subjected to current, unevidenced, interventions?”

    I do wish you’d done your homework.
    ANY research undertaken through Initial Teacher Education or instigated through a university based CPD scheme is subject to the full scrutiny of a university ethics committee just like any medical drug trial.

    Quote: “You seem to think that doing an intervention just because a teacher feels like it is OK, but as soon as there’s some structure of a trial put around it it suddenly requires a different, much higher, degree of consent – can you explain why that is?”

    A straw man argument if ever I saw one. Who said any of that?

    Quote: “Participants in a medical trial have to consent, but then so does any patient outside of a trial too; with the exception of emergencies, a medic cannot subject anyone to any intervention without informed consent. The same is not true in education.”

    But I can evidence your opinion to be based on little or no research, by you. Visit the BERA website read their guidelines on ethics, read some educational research. Then realise how silly you’ve been.

    Thanks.

  34. ewan said,

    March 25, 2013 at 10:40 am

    You seem to be missing, or avoiding, the point. You say “ANY research […] is subject to the full scrutiny of a university ethics committee” which is about doing research. Then you refer to “Visit the BERA website read their guidelines on ethics” which again, refer to doing research.

    I’m not disputing that the educational establishment currently requires strong ethical approval for anyone doing research; quite the reverse – it clearly does.

    The point is that a teacher can deploy an intervention outside the framework of ‘research’ without any such approval. This is how unevidenced twaddle like ‘Brain Gym’ comes to be used. It actually easier to just do something without any idea whether it’s useful or not than it is to do the same thing within a research framework designed to find out.

    That is a problem, and it’s a result of people (non of them me) being very silly.

  35. Marcus Hill said,

    March 25, 2013 at 11:00 am

    I don’t know if it’s comedy or tragedy that you’ve been roped into this. Michael Gove, whose total teaching experience is zero, frequently ignores the evidence from education researchers (the vast majority of whom have been serving teachers) that doesn’t back up his politically motivated preconceptions. He’s not unique in this respect. Successive governments of all political leanings have commissioned expert reports to look at the evidence and make recommendations, and then ignored those reports because their findings were ideologically inconvenient.

  36. John_Clarke said,

    March 26, 2013 at 4:45 pm

    Ewan,

    I wasn’t avoiding the point which you have now clarified.

    Teachers are professionals and make professional judgements and decisions. A professional judgement, or decision, to undertake an ‘intervention’ is not research and doesn’t pretend to be research so is not covered by research ethics. Just the same way changing textbooks or using resources or many other things isn’t subject to ethics. So in a way you answered your own query; if it ain’t research why would it need research ethics?

    Brain Gym was ‘sold’ to schools by unscrupulous charlatans who pretended it was backed by research. Some of us saw through it on day one. Like homeopathy, one always needs to be aware of snake-oil salesmen.

    I always point out to doctors that even if we have great research in medicine and RCT all over the place; you can still get homeopathy on the NHS.

  37. Dr R. K. Smith said,

    March 27, 2013 at 5:56 pm

    I’m an admirer of Ben’s work and his latest Education paper. I’m late to the debate but have some builds that I hope Ben and the community find useful.

    I’m a strategic change consultant with a research background that includes agriculture and ecology. Both of these are fields that are inherently challenging for the development of evidence. I believe they are arguably even more complex than medicine. The fact that the use of proper science in these fields is relatively non-controversial may be an encouragement to the Education community, some of whom see obstacles which I touch on below.I believe there are valuable insights and learning points to be gained by looking beyond just Medicine as a reference field

    Agriculture – another field that offers insights for Education

    Agriculture was the field that really pioneered proper experimental design linked to statistics. In Agriculture, you have to cope with huge variation. Obvious examples include the weather that is outside your control and soil composition that may change as you move across a field. If you set out to test a given cultivation practice or agricultural chemical, you have to do it in such a way that you can measure the effects associated with the treatment under test after allowing for these factors and other noise that will affect your results.

    Responding to Variation

    Using the scientific method, faced with inconvenient variation, the investigator responds in well-established ways like increasing the number of replicates in the trial. High variation, increases the importance of proper scientific experimentation rather than being a reason to dismiss it as inappropriate and unworkable. There are designs (e.g. the randomised block design) that are helpful if there are identified sources of variation across the test sample – in the case of farming this might be wet, normal and dry soil. In Education it might be children with lots, some and no prior experience of a novel teaching approach.

    Ensuring research is truly representative

    One concern voiced by commentators is that a given trial may not be representative of the population that the prospective practice will be applied to in the future. Getting this right is fundamental and should shape the conception of the trials. It is conceivable that the population included in a single test could be representative of the intended user group (= patients, children etc) but that would be uncommon. A key reason to stress programmes over individual trials – see below. I think the importance of getting the sample to be representative isn’t sufficiently conveyed by Ben’s sentence: “We simply take a group of children, or schools (or patients, or people): we split them into two groups at random: we give one intervention to one group etc.”

    Formulating solutions – another reason to emphasise programmes not just single trials

    When you formulate ‘solutions’, you will almost certainly need to have different versions for different situations e.g. grapes grown for dessert use or for wine. It is very rare that anyone would come up with one universal solution relying on the findings of a just one randomised trial. I realise that Ben understands this perfectly and has covered it in his paper. But, judging by some postings, this still needs hammering home.

    In agriculture, to draw reliable conclusions and formulate solutions, it is hard to overstate the importance of investigations that integrate intelligently designed trials with other approaches like survey work, monitoring and modeling. It is really the synthesis of this lot that qualifies as reliable evidence. I recommend everyone should talk more about programmes of scientific investigation than talking about what everyone has latched onto – the ‘RCT’.

    Social & Cultural understanding must be central

    Another relevant parallel with agriculture is the huge importance of social attitudes and culture. An owner/farmer with 5000 Ha of valuable fruit in California is very different to a share-cropping subsistence farmer in Africa. This social understanding is key to understanding why something may work in one situation and not in another. A properly conceived programme of investigation will be structured to cover the various type of potential users (=patients or children). More than that, the social understanding should be the stimulation for innovative ideas. Through a long history, agricultural researchers have had to embrace all this. I’m not saying that all is perfect or that the science based evidence can ever be all that is needed. But I believe the usefulness of science is understood and accepted in a way that it seems not to be in Education as evidenced in the posts.

    I’ve some further thoughts on intelligent ‘nudges’ that the DfE should apply. Another post.

  38. John_Clarke said,

    March 27, 2013 at 9:44 pm

    Quote: “I’m a strategic change consultant with a research background that includes agriculture and ecology.”

    So, like Ben, your field is not anything to do with Education, Initial Teacher Education, Secondary Eucation or Primary Education. That said, you make some very good points, which help illuminate why Ben’s niave model of Education cannot explain the Educational setting. In addition, however, Agriculture isn’t a socially constructed activity in the sense that plants aren’t social creatures. Therefore however complex and chaotic the variables are in Agriculture; Education will be more complex by the nature that even the terms ‘teaching’, ‘learning’, ‘knowledge’ etc are contested, not ‘measurable’ to any agreed scale and change merely by the fact that they are observed or measured. Think Hawthorne experimental effects from Psychology, but then think of those effects on the Teacher, the pupil and the observer.

    In addition don’t forget this is all deliberate ‘fog’ from Gove to hide his ideological attack on state education.

  39. heavens said,

    April 4, 2013 at 4:41 am

    Education trials aren’t hard to perform ethically. The simplest option is a head-to-head comparison of two competing theories that are believed to be equally good, to find out which one is better. Another is a basic waiting-list trial: find something believed to be good, that nobody in ten schools can get, and everybody wants, and pay for it to be implemented in half of them. What’s unethical about giving half the kids something that everyone wants, when otherwise none of them could get it?

    The “parental consent” issue is also a red herring. Parents *already* don’t get to consent to teaching methods, so why should they be able to veto the choice of teaching methods in a trial?

  40. John_Clarke said,

    April 4, 2013 at 11:49 am

    Competing ‘theories’ surely you mean competing hypotheses? If you aren’t aware of the difference then…..hmmm…..neither is Gove anyway, so it doesn’t matter it’s only (sarcasm alert) teaching isn’t it? Anyone can just set up and have a go. How hard can it be?

    How does one judge ‘better’? If this was THAT easy don’t you think people would be doing it? Put your money where your mouth is and spell out how YOU judge ‘better’.

    BTW If you don’t know what is unethical about what you propose then I wonder if you know what ethics ‘are’?

  41. BernieG said,

    April 9, 2013 at 3:06 am

    Sorry Ben, we think you might have rather misjudged this one. Here is why: blogs.ubc.ca/realscience/2013/04/08/the-great-divide-it-hasnt-gone-away-it-seems/

  42. Dr R. K. Smith said,

    April 10, 2013 at 12:18 pm

    Yes, let the profession change itself but reinforcing nudges from DfE are needed too.

    In an earlier post, I explained a bit about my background and interests in this topic.

    I think it is wise that Ben’s paper has put the focus on the profession changing from within. This is certainly preferable to prescriptive changes being conceived on high and passed down to the profession. However, I think there are some highly complementary measures (nudges) that should be adopted by DfE policy makers.

    I have communicated suggestions to the Permanent Secretary, Chris Wormald in an e mail dated 9/2/13. Below are extracts. Suggestions 1-3 have also been communicated to Carole Willis who is Chief Scientific Advisor (CSA) ,DfE.
    Strengthening the CSA role is surely part of the answer to fuller use of evidence in policy-making. I don’t think Ben’s paper made reference to this. One of the expectations set for the CSA is that he or she should be able to take a reasonably objective and independent voice meaning sometimes expressing a view that may be at variance with ‘official lines’. Evidently, there is a long way for this to be expectation to be fulfilled in practice. E.g. the presentation by Lord Krebs to the Parliamentary Grand Committee on 17/10/12 www.publications.parliament.uk/pa/ld201213/ldhansrd/text/121017-gc0001.htm)
    I thought of these when reviewing an initiative in education that is seemingly forging ahead without due regard to evidence and experimentation. I’ve informed Ben of this separately.

    Extracts:

    1. A statement of alignment to policy-making based on evidence and experimentation.

    Why not require that every significant new policy, initiative or programme should carry a statement on the degree to which it is based on good practice. Importantly, the elusive and unrealistic scenario of a perfect evidence base would not be the only scenario to qualify as good practice. Proceeding with a policy based on very limited evidence could count as ‘good practice’ under certain conditions – e.g. if there is urgency to act, if a genuine search for evidence had been undertaken, if the scale of implementation is limited, if provision has been made for ongoing evaluation and if research is being commissioned to generate evidence.

    It would be wise if this requirement were built into processes for the approval or release of funding for interventions. Perhaps this could be a Treasury stipulation to confer greatest weight. After all, the degree of compliance surely has a direct bearing on the risk associated with the investment.

    Is there any reason why this could not be designed and announced to take effect from, say, 1 March 2013?

    2. Forum for highlighting and reviewing shortfalls from good practice.

    It would be very helpful if there were some kind of forum (on-line and/or real) for highlighting potential cases of ‘bad practice’ policy making.

    3. Key Role for Chief Scientific Advisors.

    It would surely make sense to charge Chief Scientific Advisors with overviewing how far each of their departments’ initiatives demonstrate good practice in better policy-making. I say ‘overview’ because I realise that it may be unrealistic for them to be asked to thoroughly review evidence for all initiatives overnight. However, the overview step would highlight the priorities for more exhaustive reviews that could then be taken-on in stages. There may be scope to contract trusted external reviewers to undertake some of this.

    4. Building relevant indicators into Civil Servants’ performance evaluations.

    How about announcing almost immediately that in the next round of individual performance appraisals, there will be a specific focus on how far policy-making has demonstrated good practice. It could even be stated that there will particular focus on individuals demonstrating innovative ways in which they have challenged and overcome blockers. This would be a potent nudge!

  43. Dr R. K. Smith said,

    April 12, 2013 at 4:46 pm

    I am re-posting what I said on 10 April since, oddly it hasn’t shown up.

    Yes, let the profession change itself but reinforcing nudges from DfE are needed too.

    In an earlier post, I explained a bit about my background and interests in this topic.

    I think it is wise that Ben’s paper has put the focus on the profession changing from within. This is certainly preferable to prescriptive changes being conceived on high and passed down to the profession. However, I think there are some highly complementary measures (nudges) that should be adopted by DfE policy makers.

    I have communicated suggestions to the Permanent Secretary, Chris Wormald in an e mail dated 9/2/13. Below are extracts. Suggestions 1-3 have also been communicated to Carole Willis who is Chief Scientific Advisor (CSA) ,DfE.
    Strengthening the CSA role is surely part of the answer to fuller use of evidence in policy-making. I don’t think Ben’s paper made reference to this. One of the expectations set for the CSA is that he or she should be able to take a reasonably objective and independent voice meaning sometimes expressing a view that may be at variance with ‘official lines’. Evidently, there is a long way for this to be expectation to be fulfilled in practice. E.g. the presentation by Lord Krebs to the Parliamentary Grand Committee on 17/10/12 www.publications.parliament.uk/pa/ld201213/ldhansrd/text/121017-gc0001.htm)
    I thought of these when reviewing an initiative in education that is seemingly forging ahead without due regard to evidence and experimentation. I’ve informed Ben of this separately.

    Extracts:

    1. A statement of alignment to policy-making based on evidence and experimentation.

    Why not require that every significant new policy, initiative or programme should carry a statement on the degree to which it is based on good practice. Importantly, the elusive and unrealistic scenario of a perfect evidence base would not be the only scenario to qualify as good practice. Proceeding with a policy based on very limited evidence could count as ‘good practice’ under certain conditions – e.g. if there is urgency to act, if a genuine search for evidence had been undertaken, if the scale of implementation is limited, if provision has been made for ongoing evaluation and if research is being commissioned to generate evidence.

    It would be wise if this requirement were built into processes for the approval or release of funding for interventions. Perhaps this could be a Treasury stipulation to confer greatest weight. After all, the degree of compliance surely has a direct bearing on the risk associated with the investment.

    Is there any reason why this could not be designed and announced to take effect from, say, 1 March 2013?

    2. Forum for highlighting and reviewing shortfalls from good practice.

    It would be very helpful if there were some kind of forum (on-line and/or real) for highlighting potential cases of ‘bad practice’ policy making.

    3. Key Role for Chief Scientific Advisors.

    It would surely make sense to charge Chief Scientific Advisors with overviewing how far each of their departments’ initiatives demonstrate good practice in better policy-making. I say ‘overview’ because I realise that it may be unrealistic for them to be asked to thoroughly review evidence for all initiatives overnight. However, the overview step would highlight the priorities for more exhaustive reviews that could then be taken-on in stages. There may be scope to contract trusted external reviewers to undertake some of this.

    4. Building relevant indicators into Civil Servants’ performance evaluations.

    How about announcing almost immediately that in the next round of individual performance appraisals, there will be a specific focus on how far policy-making has demonstrated good practice. It could even be stated that there will particular focus on individuals demonstrating innovative ways in which they have challenged and overcome blockers. This would be a potent nudge!

  44. SallyMorgan said,

    April 19, 2013 at 12:34 pm

    As a clinical psychologist (and therefore an evidence-based practitioner) who works a lot in schools I am constantly impressed by the high level of evidence collection and analysis that my education colleagues do. It does come across as a little patronising to suggest that health has something to teach education about evidence-based practise when they are so much better at collecting evidence than almost any health professional I know. If however this indicates that there will be additional resources put in to supporting large scale research then I’m sure this will be very much welcomed by the institutions that are already carrying out high-quality education research, such as the Institute of Education. I suspect however the sticking point will be the government actually paying any attention to the research as they have shown little inclination to listen to educational professionals so far.

  45. StanBlakey said,

    April 21, 2013 at 12:00 am

    BernieG,
    How does the Skidmore paper support the claim on your blog?
    You cite it after claiming “All in all, other forms of educational research than RCTs are much more likely to be effective to inform practice (and have been), and frankly the assumption that educators might not have thought of this before is a tad patronizing”

    Nothing I read in their paper suggest they support your statement.

    The authors seem to be proponents of RCT’s whose concern is that with the confidence that evidence brings it is more important than ever to watch for errors that will undermine this confidence.

  46. StanBlakey said,

    April 21, 2013 at 12:19 am

    Namay,
    I think you are seeing the glass half empty here. If you sit through the next Brain Gym intro and then enthusiastically ask if this is part of a new evidence based program and ask about the evidence so that you can help get buy in no one can fault you for asking for that.

    Similarly if Ben is asked to lend a hand here and then the government ignores his efforts what do you think the author of Bad Pharma will do and how that will play out?

  47. ewan said,

    April 22, 2013 at 2:51 pm

    “A professional judgement, or decision, to undertake an intervention’ is not research and doesn’t pretend to be research so is not covered by research ethics.”

    This is the fundamental error in your approach; you seem to think that there’s some sort of ‘research ethics’, separate to other ethics, bound up simply with the conduct of research per se. This is wrong.

    The point of ‘research ethics’ is not whether it’s ethically OK to do research, it’s whether it’s ethically OK to intervene in a life for a particular research purpose. That’s why a researcher in (say) a physical science doesn’t need to seek ethical approval for a lab experiment that involves no living thing, but a psychology researcher does need it for a lab experiment involving people. The thing that requires the ethical approval is the ‘having an effect on a life’, not the ‘doing research’.

    That’s why, in education, if you’re going to have (say) one school using a novel approach and another not using it, and you’re already ethically OK with that happening (you can argue whether or not one should be, but it’s clearly widely accepted at the moment), then you don’t suddenly start needing ethical approval when you start paying attention to the results and call it a research project.

  48. yish said,

    April 23, 2013 at 11:45 pm

    Ok, let’s do an experiment. Not a RCT, just a thought experiment.
    * Think about the one teacher that changed your life.
    * Explain how she / he did it.
    * Phrase that as an hypothesis that can be refuted or verified by a RCT.

    I agree that education needs to be more evidence-based, but I’m worried about the narrow definition of evidence you’re promoting, and the narrow view of scientific method.

    Education is a complex system, highly contextual, and value driven. The effects of educational actions are seen in a distance of years and decades. The study of education is, or should be, a design science – as well as social and natural science.

  49. StanBlakey said,

    April 27, 2013 at 1:21 pm

    Yish,
    Let’s think of one doctor that has changed the way the public perceives medicine.
    Dr Goldacre is proof of the existence of people that like evidence but are also able to change perceptions and be engaging while doing it.

    Health fits all the terms you list for education.

    Medical evidence includes research done over tens of years. There is no reason to think that the same won’t apply in education.

  50. StanBlakey said,

    April 27, 2013 at 1:55 pm

    I think the concern among many reading Goldacre’s proposal is that there will be good teachers who are told to stop doing things their way and do it the way some study has shown to be the right way.

    The good teacher can’t afford to do a competing study and so won’t be able to support their case with evidence.

    To get this to work we have to assume the teacher does do something better that is incompatible with the best way indicated by the study. Presumably if the study included an examination of the good teachers approach and found it incompatible with the best way and the evidence was conclusive the good teacher wouldn’t have a problem being the one to change.

    So we have to have that the good teacher’s method was not studied or there are situational differences that make the study irrelevant to the good teacher’s situation.
    Obviously the more the good teacher was engaged with the research in education the less chance there is of this happening.

    So now the good teacher has to figure out if they should continue with their method. Perhaps this is okay. It’s not medicine, their choice won’t kill their students. Perhaps with some objective peers they can make the best informed decision. Either way in a year another teacher will make the decisions.

    At the level of the individual teacher we can all recall one teacher that we wouldn’t want stifled by outside interference. Maybe that’s not where RCT’s and evidence based policy should be dictating the one true method.

    There is still plenty of room for RCT’s and evidence based policy to be applied at the organization level for the recommendations, training and materials.

    At that level RCT’s are only a win for the good teacher as under an evidenced based policy regime no one dictates a method without meaningful evidence and the good teacher has the opportunity to provide input and critique the research.

  51. Marcus Hill said,

    May 10, 2013 at 12:12 pm

    There are plenty of longitudinal studies in education that last a number of years. The problem is that any major changes to the education system are, by thei nature, driven by government. It’s easy in medicine to make incremental changes to practice and research how best to treat particular ailments. There’s nothing in medicine as all-controlling as the National Curriculum or the public examination system. Education secretaries tinker with these, but major changes do need to be backed by evidence. Ideally, such a change would start with a consultation exercise leading to draft new documents – that’s around two years to do properly. You then need to pilot these with at least two cohorts to fine tune things, and then consider whether the evidence warrants a national roll-out. That’s another three years of work. So, from initial idea to national roll-out you’re looking at a minimum of five years if it’s done properly. The problem, of course, is that there is bound to be a general election in the next five years, and the Secretary of State needs to have his or her grand new scheme in place before that. This is why education is beset by wave after wave of sweeping change, politically motivated and insufficiently researched.

  52. JFB said,

    May 21, 2013 at 12:29 pm

    Both Rebecca’s Allens and Geoff Whitty’s comments to Ben Goldacres’ DfE Analytical Review seem to be saying ‘well of course we need randomised controlled studies in education, I have been saying this all along, however, I have always advocating going much further and using much more sophisticated and nuanced interpretation that applies only to education to say what works for whom and under what conditions’. This humbug is beyond tolerance on several levels, as firstly during Geoff’s 10 years premiership of the leading education research institution his organisation did not carry out any randomised control trials. Similarly, throughout Rebeccas’ career, apparently dedicated to this paradigm shift she has not previously carried one out. So this adherence to RCT seems a rather rapid conversion suspiciously coincidental with recent rise in criticism of the methods that come from outside education research establishment they are central members of. Secondly, their understanding of RCT methodology suggests the topic is ‘Google new’ to them as they say its’ different in education than it is in medicine because unlike medicine in education we must know what works for whom and under what conditions. When in medicine it is also necessary to know what works whom and under what conditions for example a drug supposed to treat Parkinson’s disease (what) should help those with Parkinson’s disease (whom) at this dose at this stage of the disease (what conditions). It is hard to imagine what experiments they are thinking of that doesn’t specify what, whom in what conditions. Moreover these comments seem to attempt to claim credit for inventing afresh the idea of what’s called the ‘methods’ section in the write up of any experiment in any research in any field and taught in the first week of any research methods course (apparently apart from education). Their hubris is further compounded by their attempt to imply that they have added substantially to the debate by suggesting that readers of education research need to be careful to bear in mind that studies on for example, 15 year olds Maths in average UK schools, would only apply only to 15 year olds Maths in average UK schools and not 4 year olds music lessons in a Viennese conservatoires. These comments seem to suggest that the Rebecca and Geoff think that teachers will not be able to make this inference.
    Further clarifying the recency of their thoughts on the subject they seem to think it is informative to point out that experiments without theory aren’t very helpful, when any view of the history of experimental science suggests fairly clearly that doing experiments on things you think might work is more helpful that doing experiments on things you don’t think might work. They go on to suggest a solution to this problem is to ask for expensive qual studies with RCT, a call likely to put off funders rather than encourage them at a time when as Rebecca rightly points out schools and funders are million miles away from buying into the need for RCT. A step likely to preserve the perennial problem stymieing education research where whenever asked to research anything education researchers respond ‘oh its very complicated’, interpreted by funders to mean expensive and producing inconclusive results. All this seems to suggest Rebecca and Geoff may have gathered their thoughts on the issue of promoting RCT rather more hurriedly than they may have us believe.

  53. Ben Goldacre said,

    February 9, 2014 at 7:48 pm

    Hi there

    sorry, I missed most of these comments.

    The issues raised are covered at length in the comments below my Guardian piece on the topic, where I spent about a day engaging with some very confused arguments. I strongly recommend it, whether you’re rabidly against RCTs, or just interested in the reasoning and culture of those who are. I would repeat the exercise here but it’s quite time consuming addressing the same canards.

    www.theguardian.com/education/2013/mar/18/teaching-research-michael-gove

    It’s genuinely interesting – from a scientific perspective, and from a cultural one – how much special pleading, territoriality, and misunderstanding this issue elicits from people. Some of the misunderstanding is legitimate; some, I would say (and not lightly) is deliberate. Because of that, and the scale of the problem, I am going to do a much larger piece of work on RCTs.

    If any of you have any good references to people making the anti-RCT arguments made here and elsewhere, but in more concrete settings – ideally academic papers, and books – please do send them over, either here or by email ben@badscience.net. They are made commonly in conversation and blog posts, but it’s harder to find people willing to put their names to them formally in print. I’m particularly interested in people arguing:

    – “you can’t know what to measure” whilst defending observational research

    – “randomistas want to do only trials”

    – “trials are unethical”

    – “trials can’t answer questions about the purpose of education” (a bit like complaining an aeroplane can’t make toast)

    and so on.

    For those very interested there was this recent interaction with the head of policy at the ACSL, where the comments are worth reading.

    www.ascl.org.uk/news-and-views/blogs_detail.html?shorturl=the-problem-with-evidence-based-teaching

    I don’t know what it is that would make a union take this position on RCTs and evidence.