Boris Johnson and his innovative trial methodology

July 31st, 2010 by Ben Goldacre in evidence, evidence based policy, politics, schools | 95 Comments »

Ben Goldacre, The Guardian, Saturday 31 July 2010

It’s the near misses that really make you want to shoot your own face off. This week the Centre for Policy Studies has published a pamphlet on education which has been covered by the Mirror, the Mail, the BBC, the Telegraph, the Express, the Guardian, and more. Boris Johnson endorses it.

The report examines why one third of children have reading difficulties at the age of 11, and concludes it is because of a lack of discipline, and the absence of a teaching system called “synthetic phonics”. The report contains lots of anecdotes, but barely mentions the evidence.

In 2006 the government published a systematic review and meta-analysis of all the trials ever to look at phonics, which you can read in full online. Skip the rest of this boring paragraph. There were 14 trials in total looking at reading accuracy as their outcome, and collectively they found some evidence that phonics are a little better. Then there were 4 trials looking at comprehension, which found only weak evidence of benefit. Finally there were 3 trials on spelling, which collectively found no benefit for phonics. All of these trials were tiny, and when I say tiny, I mean they had between 12 and 121 children, mostly at the lower end of that range. Only one trial was from the UK.

Many teachers feel the evidence is not compelling, and don’t like phonics. To be fair, there really isn’t enough evidence to say phonics definitely works. The pamphlet recognises this. So how do we move forward? Should we run a large, well-conducted randomised trial?

No. The think tank have it all worked out, and so does Boris. Their innovative solution is taken seriously by every newspaper in the country. “It is time to end this culture war,” says Boris in the Telegraph: “to try to settle once and for all, in the minds of the teachers, whether synthetic phonics is the complete answer or not… It is surely time for the Government to organise a competition, a shoot-out between the two methods, to see which is the most effective for children of all abilities.” Both expand on this idea. Read for yourself. They don’t mean a trial. They really do want a competition.

By now you don’t need me to tell you how dumb this suggestion is, but in case anyone in power is reading: there is no room for debate here, a “competition” between schools who’ve chosen one or other method is definitely and unambiguously flawed by design. We run randomised trials, where the schools are randomly assigned to one method of teaching or another, for one very simple reason: we want to make sure that the two groups of schools – the ones doing the phonics, and the ones using the other methods – are as similar as possible for all other factors.

If we don’t randomise, “using phonics” might not be the only difference between the two groups of schools. Maybe the schools using the strict phonics systems tend also to be run – and attended – by hardworking disciplined nerds like me: if this is the case, then those schools might do better on literacy tests because of the nerdiness, rather than because of the phonics.

Why have large, robust, randomised trials not already been done? Because people like Boris don’t demand them; because teachers often believe – as doctors once did – that their expertise and intuition make such tests irrelevant and undesirable; and lastly, because many academics in the field of education inexplicably resist them.

This is a relatively new tragedy. In education, as in medicine, there is potential to do enormous good, but also incalculable enduring harm through failure: and recognising that, some of the earliest examples of randomised trials are from education. Many of these predate the 1948 MRC trial of streptomycin which is widely (and incorrectly) regarded as the first proper randomised trial. In 1928, Remmers took the worst 200 students of one freshman year and randomised them to receive either remedial teaching, or teaching as usual, and measured the difference in outcomes at the end of their course. In 1931 Walters did a randomised trial to see if counselling improves student performance. In 1933 Remmers was at it again, running a randomised trial to see if having exams at the end of the first term on a course improved a pupil’s outcome in final exams. Education researchers helped to pioneer randomised trials, a lifetime ago, but then abandoned them.

We expend a vast amount of money and effort on assessing children, without much evidence that this does them any good at all; but we make no attempt to cheaply and systematically assess the teaching profession’s various education methods, despite knowing for an absolute fact that this would bring incalculable benefits for every generation to follow. Instead we have Boris and some think tank wittering on about a “competition”: and everyone takes them seriously.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

95 Responses



  1. clobbered said,

    July 31, 2010 at 1:07 am

    Having recently started to pay attention to the educational establishment (offspring reaching school age), I really want to cry. It’s all anecdote – this school did X! And it had great results! But of course teaching is one of those professions where so much depends on the quality of the teacher, that it is very hard to disentangle the quality of the staff from the quality of the method, even for kids with equivalent backgrounds.

    This is why you need seriously large studies, to factor out the chance that a couple of superstar teachers are skewing your results (i.e. you need large number of classrooms enrolled, not just children). The irony is that state school systems provide all that infrastructure that can be leveraged for studies… but are not.

    Even if one thinks that this is say, politically impossible (little Johnny’s parents are gung-ho phonics and don’t want him to be taught in a non-phonics way), there is precious little research even when consent is not an issue, eg. by comparing different national school systems that use different methods.

    Bring on the evidence-based education I say. If there is such a thing and I have missed it, please, please point me to it.

  2. mcjhn said,

    July 31, 2010 at 1:11 am

    Totally agree, I’ve just started out in education and admittedly its just from personal experience but evidence based practice wasn’t nearly as central as it should be. To top it off I had to do a comprehension exercise on a Brain gym article for my English skills test.

    We have a lot to learn from medicine, I wish I knew where it went wrong and how to sort it out…quickly. It puts me off going back. On the positive side, I remember a Y8 class being in total disbelief when I told them about homeopathy, then, once I convinced them it was real, laughing aloud.

  3. Kimpatsu said,

    July 31, 2010 at 2:59 am

    The problem with politicians is that they won’t endorse anything that will come to fruition after they have left office, because their worst nightmare is the incoming “other lot” get to reap the benefits. Quick-fix, “I have all the answers” politicking at its finest!

  4. ellieban said,

    July 31, 2010 at 3:10 am

    At least some of the problem is caused by the amount of paperwork teachers have to do – I’m sure there are many who would love to spend some of their time figuring out how teaching can be done better, but they have too much paperwork :/

  5. Daniel Rutter said,

    July 31, 2010 at 3:39 am

    Language Log had a post the other day about another pitfall of these sorts of studies:
    languagelog.ldc.upenn.edu/nll/?p=2434

    In brief, people who get poor scores in a test may just not be taking the test seriously. This tends to make “low achievers” look much worse than they really are, because if you’re used to being the disregarded underdog, you’re probably not going to care much about a test that you perceive as yet another meaningless ritual that will make no difference to your life.

  6. Dr Grumble said,

    July 31, 2010 at 6:36 am

    We have done some trials exploring the best ways of teaching medical students. It is not easy. As you well know, if you compare two drugs which work you need very large trials indeed to show a small difference. The same is true of two teaching methods which work. But it is worse. If medical students become aware of deficiencies in their training they take their own remedial action. This does not happen with patients in drug trials.

    To give you an example of this, when I was at school I got allocated to the very poor teacher of English literature. Another group got the excellent teacher. All those allocated to the weaker teacher went into an early panic and bought all the cramming books they could. We worked twice as hard to make up for the weak teaching. To this day I know an awful lot about the prologue to the Canterbury Tales. Those with the good teacher coasted. The exam results in the two groups were much the same. I recall ours were better which we thought was a travesty.

    There is another problem with trials of teaching which is that the less knowledge you have the more easy it is to teach you. If you speak English as your mother tongue you will not learn anything from a beginners’ English class. The less you know the more likely you are to learn something. So in a trial of methods there is always a bias which tends to mean that those that slip behind from a poor teaching method have a tendency to learn more.

    My understanding is that some believe that these problems make sound trials of teaching methods virtually impossible.

  7. MFunnell said,

    July 31, 2010 at 7:40 am

    To be fair to Boris, this is what he says;

    “If, as the Centre for Policy Studies suggests, an
    annual competition can be devised to discover which schools are best at teaching children how to read – with adequate controls – then I would certainly give the venture my full support.”

    OK, he does use the term ‘competition’, but he also refers to ‘adequate controls’. I will have to read the whole thing a bit more carefully to see what the CPS actually say about this.

  8. oldandrew said,

    July 31, 2010 at 8:41 am

    I wish you hadn’t written this. It will be quoted in internet discussions for years to come as one of the few scientific-sounding sources to suggest that the evidence for phonics isn’t overwhelming (which it is).

    The problem is that study quoted makes the assumption, understandable from a doctor’s point of view but pretty absurd to a teacher, that a Randomised Control Trial is the best method of evaluating teaching methods. By assuming that it is then the research on phonics is cut down to almost nothing and so, of course, there appears to little research to justify teaching synthetic phonics (or for that matter to avoid teaching it, or indeed for any conclusion about any topic in education).

    The problem is that teaching is not like medicine. You can’t teach a placebo lesson. You can’t treat classes as isolated individuals, you are obliged to consider the mix of personalities within a class. You can’t keep the students ignorant of what “treatment” they are being given. Most importantly of all, you can’t keep the teacher ignorant of what treatment they are delivering, or require them to deliver their lessons in a uniform, prescribed way.

    For these reasons, RCTs are not standard in education research. Researchers will try to avoid obvious biases in the selection of classes used in trials but will not usually randomise.

    Now if we include the vast majority of peer reviewed studies, including those which aren’t RCTs, then the evidence for synthetic phonics is overwhelming. The 2006 study is not the only research worth considering (its use of tiny studies, and studies that haven’t ben published makes it less than ideal in itself) and relying on it will, of course, result in the conclusion that there is insufficient evidence.

    My challenge, whenever anybody suggests that there is not enough evidence for phonics, is to ask the following question: name one other result in the social sciences that has better evidence for it than the effectiveness of phonics.

  9. dvavasour said,

    July 31, 2010 at 9:37 am

    Different children need different approaches to learning to read. The ones who are going to learn easily will thrive however they’re taught. The ones who struggle need a mixture of synthetic phonics *and* look and say *and* contextual clues.

    They don’t need a randomised controlled trial on teaching methods, they need skilled teachers who can teach them to read using the right blend of methods for the particular child – particularly for slow readers.

    Remember that people writing on this blog are already keen readers. It is also important that teachers foster a *desire* to read. Proposing a study of how long teachers spend sounding out letters is rather missing the point of education.

  10. aka_pigen said,

    July 31, 2010 at 9:56 am

    Unfortunately it is not just schools that get this wrong. All lecturers at universities now have to take a course designed to improve their teaching. A friend recently attending this course asked for the evidence that it was improving performance – of cause that was regarded a silly question. Most of the sessions seemed to consist of group discussions amongst the participants and have little input from the teachers – i.e. your ideas and intuition is what it is all about. Discussion topics include:” should a male lecturer giving a tutorial to a female student keep his office door open or shut?” the course finishes with all participants having to write a self-reflectory essay on how the course has improved their teaching – as you have to pass this is a great way for the course to get loads of positive feedback! Am I the only one who thinks this is a real opportunity missed and that universities should be about research based teaching whether education their students or own staff?

  11. Jut said,

    July 31, 2010 at 9:57 am

    Andrew, usually I think you’re spot on with your views. But your post here just makes me sad :(

  12. garethandjane said,

    July 31, 2010 at 10:27 am

    As a forensic scientist and school teacher (respectively) we would like to make the following observations.
    1. The evidence regarding the value of the phonics method of teaching reading should not be considered in isolation. The methods against which phonics is being compared need to be clearly defined the evidence relating to their value factored into the debate. The Sally Clark case teaches us that evidence cannot be evaluated unless at least two hypotheses are considered.
    2. Whilst acknowledging that the purpose of this thread was to point out the absurdity of the exercise promoted by Mr Johnson the debate needs to be a bit more sophisticated than whether or not phonics provides a magic reading bullet. My experience as a primary school teacher (including several years teaching children with special educational needs) tells me that there are 3 keys to success in this field (and yes, I accept this is a subjective analysis – over the years I haven’t split my puils into a group that will receive ‘good teaching’ and another that won’t!). They are – i. teachers need to have an understanding of the range of different methods (including phonics) that can be used. ii. Teachers need to be able to able to apply these methods in combination. iii. Teachers need to be able to accurately assess the combination of methods that best meets the needs of the individual pupil.
    The need to improve the reading skills of children in the UK would be better served by discussion of these three latter points than by running a competition that would provide outcomes that were at best unhelpful and at worst misleading.
    Regards.

  13. Mike Kelly said,

    July 31, 2010 at 10:28 am

    old andrew:

    You can do comparator trials without a placebo. you can have decent group sizes and appropriate stratification to correct for individual variation and it isn’t a good thing to end your argument with honking “tu quoque” logical fallacy

  14. pob said,

    July 31, 2010 at 10:43 am

    MFunnell, you’re misleading yourself if you think Boris means anything other by adequate controls than, “here’s a new phrase I learnt yesterday.”

    What further proof do you need than him touting the idea of competition? To trivialise education in such a way is voluminous of Boris’ thinking.

    As for OldAndrew, I find my head in my hands reading your statement. Clearly your phonics-biased from the start and like most proponents of an idea who find themselves forced into a corner, you’re claiming that any attempt to prove the value of the idea is unfair and distorting. Your mind is already preconceived in its evaluation of phonics. Feel free to advise me otherwise.

    In my view the focus is narrowly about schools. Acquiring reading ability extends beyond school, it involves parents and family also. It requires the interest and will to read and recognition of the beauty of language – something I saw as a child yet no teacher I know expressed the same enthusiasm for it.

    I’m interested in what the stats are for say, European countries and illteracy, since it is fair to say English is an unannotated language that provides no structure from inferring the difference between say, cat and kick. It’s entirely acquired.

  15. oldandrew said,

    July 31, 2010 at 10:51 am

    Mike Kelly,

    I am explaining why most of the research in education, and in phonics in partcilaulr, doesn’t consist of RCTs. I am not saying that none of the reasons for this could be addressed if somebody did want to spend a lot of money on RCTs to investigate something for which there is already a huge amount of evidence.

    By the way, I suggest you have another look at what a “tu quoque” fallacy is. What I did, which was to suggest anyone making declarations about insufficient evidence about an educational policy, look at social science, not medicine, for their standards of evidence is not a “tu quoque” fallacy by the usual definition.

  16. oldandrew said,

    July 31, 2010 at 10:55 am

    “Different children need different approaches to learning to read.”

    And the evidence for this is?

  17. oldandrew said,

    July 31, 2010 at 11:11 am

    Pob,

    I can advise you that I didn’t condemn “any attempt”. I am quite happy with research into this area. My issue is with setting a particular, highly debateable, standard for research and then concluding that there isn’t enough research to prove the point conclusively, and then using that as grounds to condemn other people’s conclusions.

    That said, I do find it interesting that my views on research methods in education are being condemned on the basis of ad hominems about my views on phonics. Do you really think that I would have formed all my views about the validity of education research from the one issue of phonics? Isn’t this pretty much the same way climate change deniers condemn all the standard methods of studying the climate as being contrived to prove climate change?

  18. deevybee said,

    July 31, 2010 at 11:23 am

    Re different children needing different approaches: there is growing evidence that some children just don’t get it, even with intensive state-of-the-art intervention. Problem is nobody really knows what will work with them. See
    Torgesen JK (2000) Individual differences in response to early interventions in reading: The lingering problem of treatment resisters. Learning Disabilities Research and Practice 15: 55-64.
    (And no, I don’t like the term ‘treatment resisters’!)

  19. Mike Kelly said,

    July 31, 2010 at 11:34 am

    Phonics is Ok because social sciences is bad? Tu quoque by any other name

  20. oldandrew said,

    July 31, 2010 at 11:40 am

    Mike Kelly,

    That’s not what I said.

    And even if it had been, that wouldn’t be a tu quoque.

  21. Mike Kelly said,

    July 31, 2010 at 12:20 pm

    “My challenge, whenever anybody suggests that there is not enough evidence for phonics,(paraphrase: if you think phonics is bad) is to ask the following question: name one other result in the social sciences that has better evidence for it than the effectiveness of phonics (paraphrase: social science is as bad or worse).”

    So what did you mean?

  22. oldandrew said,

    July 31, 2010 at 12:36 pm

    What I said. I asked a question that would put certain arguments in perspective. I never stated or implied the social sciences were bad. The key point I’m making is that social science research is different to (not worse than) medical research.

  23. mch said,

    July 31, 2010 at 12:36 pm

    Well, “a compemtition, a shoot-out between the two methods, to see which is the most effective for children of all abilities” is a decent aim. Any medical trial that compares treatments for efficacy is exactly that.

    As for the method used, properly run RC trials are good, but then so is introducing a set of incentives across the board to encourage better results everywhere.

    These are not necessarily incompatible.

  24. thermalCat said,

    July 31, 2010 at 12:59 pm

    I remember scanning thru a book called ‘Evidence based teaching’ (Petty 2006) while doing a PGCE a while back. I don’t know whether it addresses phonics specifically, but I found it to be by far the most useful of the recommended reading on my course.

    Stepping back from the specifics of phonics & which experimental techniques are as appropriate in education, I agree that an evidence-based approach to education is the only sensible way to resolve these questions (resolve meaning pretty good agreement, rather than final, no-argument resolved).

    I would argue tho, that the assertion ‘teaching is too complicated to reduce to simple measurements’ is just as corrosive to progress in education as it is in medicine.

  25. iainhouston said,

    July 31, 2010 at 1:50 pm

    My wife and I are both teachers and often have to attend training on the latest magic bullet that will solve all the problems in education. Usually this is backed up by an anecdote. Something like, “One school in *insert faraway place* was very sceptical but afterwards a teacher told me about the difference it has made. None of the pupils in her class could read at the start and she had tried everything. Now, after only three months, those pupils have formed a book group in their own time where they conduct textual analysis of shakespeare plays and one boy told me he thought reading had saved his life.” That appears to be all the evidence they need.

    There was a period when I remember us being encouraged to conduct our own research. This was to consist of taking an approach, using it with a class for a while and then writing about the difference it had made. When my wife asked if there shouldn’t be a control group of some kind the attitude was, “that isn’t ethical, because you are depriving that group of the benefit of the new approach”. Even though no benefit had been shown. All teachers present were explicitly told that all pupils had to receive the same teaching for this reason.

    At a recent training course we were shown brain scans of children who had been victims of the most horrendous neglect in Romanian orphanages. These were compared to those from “normal” children. This showed the physical effect of neglect on the brain and she provided the scientific paper which showed these results. Then we were told that this showed that children who had suffered from parental neglect had enduring, permanent neurological damage which was untreatable and caused poor behaviour. In exactly the same way, I am sure we all recognise that if someone is held underwater for 10 minutes they will drown. This must mean that going out in the rain would drown you a little bit. When I challenged the trainer she agreed she didn’t have any evidence for what she was saying and agreed that she was guessing. She was hoping to roll out this course to the while authority. I don’t think that ever happened.

    Thank God at teacher training we were only encouraged to use rigorously assessed and scientifically based approaches like “Brain Gym”.

  26. daven said,

    July 31, 2010 at 3:00 pm

    It is possible to do controlled trials in education. But you have to be as clever (and sometimes devious) as a psychologist.

    See, for example, D. R. Newman, Chris Johnson, Brian Webb and Clive Cochrane (June 1997) Evaluating the quality of learning in Computer Supported Co-operative Learning, JASIS, with more detail in Newman, D. R., Webb, B. & Cochrane, C. (Apr. 1995) A content analysis method to measure critical thinking in face-to-face and computer supported group learning, IPCT-J, 3(2), 56-77 and D. R. Newman, Chris Johnson, Clive Cochrane and Brian Webb (Feb. 1996) An experiment in group learning technology: evaluating critical thinking in face-to-face and computer-supported seminars, IPCT-J, 4(1), 57-74. (all linked from www.qub.ac.uk/mgt/papers/abstract.html#cscl).

    When we started using computer conferencing as an alternative to seminars, we wanted to see if they had different effects on learning. To control things, each student group used CC for one topic, and face-to-face seminars on another. Some groups did topic A f2f, others did topic A using CC. Then they swapped over. So all students benefited (or suffered) from both techniques – there was no ethical difference.

    The biggest problem was working out what to measure – defining the important questions. That is something rather easy to do in chemistry or some kinds of medicine, but is harder in psychology and sociology. There was no point comparing exam results weeks later – there are so many other factors that can influence them. That is the real danger of Boris Johnson’s approach – not the lack of controls, but asking the wrong questions or measuring the wrong things. In Lee’s paper on “Generalising generalisability” he explains how qualitative research helps you define the right questions and hypotheses, leading to generalisability, while quantitative research is better at measuring the answers to the right questions (or the wrong ones if you ignore the qualitative).

    Instead we needed to evaluate the actual learning taking place in the two types of discussion. Since the goal of the discussions was to get students to think critically of different things they had read, we used Garrison’s Theory of Critical Thinking. We transcribed everything every student said during the sessions, then did a content analysis according the the positive and negative approaches to problem-solving (e.g. giving evidence for a position as opposed to just stating it). We also used a typical post-experience questionnaire where the students compared how much each technique helped them in particular dimensions of learning.

    We only managed to do all that because the university funded a research assistant to do all the hard work. It takes far more time than any teacher has available normally.

  27. rlongstaff said,

    July 31, 2010 at 3:03 pm

    Sounds very much like the attitude may be critical of ‘socialist, top-down, big Government’ full randomised trials – to which the solution is obviously to let the free market decide by means of a competition, because that’s the lifeblood of the market and competition always leads to the best solution, doesn’t it?

  28. lasker said,

    July 31, 2010 at 3:26 pm

    What is the point of all this? Surely the teaching of reading is primarily an interaction between a single child and an adult hearing her read. Why should the best method of teaching not depend on the characters and intellectual strengths of the individuals involved? Then learning can proceed by a process of trial and error appropriate to those two individuals.

    If a randomised, controlled study was undertaken it would take time and resources and lead only to a generalisation which would be meaningless in the context of any specific learning interaction.

    We all know who were the best teachers at out schools but we don’t have a clue what their educational philosophies were. What they knew was how to inspire.

    And disruptive children. They should just be allowed to play sport all day. Then teach them sport science.

  29. oldandrew said,

    July 31, 2010 at 3:31 pm

    “Why should the best method of teaching not depend on the characters and intellectual strengths of the individuals involved?”

    As I understand it the evidence from cognitive psychology suggests people learn in basically the same way, or at least, that we can’t identify individual learning styles and teach to them.

  30. phayes said,

    July 31, 2010 at 3:57 pm

    Ceci n’est pas un comment.

  31. timbod said,

    July 31, 2010 at 4:04 pm

    “The report examines why one third of children have reading difficulties at the age of 11″

    This reminds me of headlines 5ish years ago saying “20% of 11 year olds are illiterate”. Reading further I discovered that the category “illiterate” was defined using the benchmark of the 20th percentile of 11 year olds’ reading abilities. Doh!

  32. StanBlakey said,

    July 31, 2010 at 4:48 pm

    Sorry OldAndrew, I don’t get your objection to randomized controlled studies for education. It is not a comparison to a placebo lesson, it is a comparison between two plausible methods to see which is better. If there were enough evidence to conclude one was better then there wouldn’t be the need for the study. So perhaps this needs to be done with teachers that are not so biased or unaware that they can see that they don’t have information to know the answer.

    (I see you claim there is enough data in this instance so my point is a general one – what is wrong with RCT’s?)

    To some other points – a problem with a competitive market approach is that if you were going to compete you would want good data on what method works. You want exactly the study competition is supposed to replace. Otherwise you are just gambling.

    To the complaint that there are too many other factors to separate the effect of the item being investigated – if so then perhaps the answer is the effect is too small to matter and some other item is more significant to the outcome.

  33. oldandrew said,

    July 31, 2010 at 5:16 pm

    “Sorry OldAndrew, I don’t get your objection to randomized controlled studies for education.”

    How many more times? I don’t object to them. I am just pointing out that it is ridiculous to only pay attention to RCTs, to the point of ignoring decades of research.

    “If there were enough evidence to conclude one was better then there wouldn’t be the need for the study.”

    That’s kind of the point, isn’t it? No amount of evidence for the effectiveness of phonics is ever enough for some people.

  34. ofoi said,

    July 31, 2010 at 6:07 pm

    I am an education researcher. I essentially agree with Ben’s complaint about the field, although it is also true to say that experimental studies in education are going to be harder than in medicine because of the multi-level modelling that will be required. Still, it ought to be possible.

    I think that the real reason that hardly anyone does large scale experimental tests of new teaching techniques is not as inexplicable as Ben thinks: There simply isn’t the funding to do it.

    In their Educational Researcher paper from 2003 Burkhardt and Schoenfeld calculated that around 5-15% of the entire US spend on medicine is devoted to research and development. In contrast, they found the figure in education is 0.01% (about $30m out of a total education budget of $300bn).

    Unless the Research Councils would be willing to invest substantially more cash in education research it simply won’t be possible for education to resemble medicine.

  35. raven said,

    July 31, 2010 at 6:47 pm

    I had a look at the CPS document. Couldn’t see any supporting statistics in it.
    Some of it didn’t make sense and seemed to have a political agenda (not surprising). There was a bit on using group learning in primary classes, followed by a rant about mixed ability learning. Which doesn’t make sense as in the admittedly small number of primary classes I’ve seen in action (6 ish) the groups have always been ability groups.

    So I’d agree with Ben’s assesment of that, and that we really need better evidence on the best way to teaching reading. (I agree with garethandjanes points about teachers assessing individual pupils too.)

    Small query with this bit in Ben’s article though – “Many teachers feel the evidence is not compelling, and don’t like phonics.” Is that right?

    All the primary schools I’ve seen have used phonics, all the teachers have been quite happy to use phonics.
    Many of them may not like ‘synthetic phonics’ programmes, which is not quite the same thing as not liking or using any phonics.

    PS The US have recently been trying to encourage more research into reading methods. There’s a site called “What Works Clearinghouse”, which looks to evaluate educational research. I was looking at it a few weeks ago, and even there the evidence seemed pretty thin so far on many of the commercial reading programmes.

  36. CoralBloom said,

    July 31, 2010 at 6:53 pm

    Right, what scientists have we got that are electable? A few hundred up and down the country should do it?

    Aren’t there any in the civil service who could be advising politicians, little post-it note reminders here and there?

  37. Paul Connolly said,

    July 31, 2010 at 7:54 pm

    Well said Ben. It is quite possible to undertake randomised controlled trials in education. The Centre for Effective Education at Queen’s University Belfast is currently running eight randomised trials; most of these are based on fairly reasonable sample sizes (typically from 400 to 1,000 pupils) – see: www.qub.ac.uk/cee . From our experience, schools and teachers have no problem participating in trials and, for the most part, very much welcome the emphasis on outcomes. Yes, there are limitations to the use of trials in education and dangers associated with generalising from small trials involving particular populations. However, this is where systematic reviews can play such a critical role in not only weighing up the evidence of the effectiveness of particular educational approaches but also whether they tend to be more effective for specific subgroups and/or if delivered in particular ways.

  38. Qilin Tracker said,

    July 31, 2010 at 8:22 pm

    Another problem with the pamphlet, judging from your post, is that it doesn’t really question the relevance of initial reading instruction method for 11 year olds. It may be that the phonics/other method debate is applicable for younger children but that by 11, what really matters most is ‘number of words read/month’. Or perhaps even something else.

    P.S. you’re right to point out that only one trial was from the UK, though it would have been more interesting to know how many were from English-speaking countries. Languages differ quite a lot in the reliability with which they map sound to letter combinations.

  39. Ian said,

    July 31, 2010 at 10:00 pm

    Before I was teacher I worked for a large retailer as a manager. I got fed up being told “this marvellous new system has been trialled in some stores/departments and productivity went up by 25%”. Unfortunately a few weeks later we were all back where we started. On more than one occasion I introduced a marvellous new method (the one we used to use) and productivity went up by 25%.

    Every time you make some meaningless change everybody’s enthusiasm is renewed for a while because it’s different. I believe Ben wrote a chapter in his marvellous book (but I can’t find my (autographed) copy – so I forget what the effect is called).

    Now I teach science and we keep being given marvellous new ways of teaching which are much better than the old ways. (Here in Scotland we have The Curriculum for Excellence which involves kids doing stuff instead of being told about it or shown it – who’d’ve thought that’d be more fun of them?) Other, younger teachers are constantly amazed by my enthusiasm for new methodologies. As are other, experienced teachers who cynically ask if I really think these new ways are better. No. I reply, but they’re different and that’s enough.

  40. gnadori said,

    August 1, 2010 at 7:12 pm

    I tried to find the Remmers 1933 study, but could not. Does anybody know what was the outcome of the trial?

  41. hastalik said,

    August 1, 2010 at 11:40 pm

    thanks for this trial methodology

  42. David Perkins said,

    August 2, 2010 at 10:00 am

    I’d like for large scale randomized trials to be be an effective approach to sociological problems, but there are many hurdles that I’m not sure how to get over.

    To pull off large scale studies like this there needs at the very least to be general public consent. Let’s skip straight to the (fairly inevitable) doomsday scenario. Eventually one of these studies will show significantly different outcomes under different conditions. The students and their families in less successful groups feel harmed and blame the study. What does one do in response to maintain public will to permit these studies? Since this scenario can repeat itself over and over, without a very convincing counter-strategy public backlash will eventually stop the studies.

    These are many arguments that a disinterested person might accept to justify these studies, but parents are not normally disinterested in their children and are unlikely to find those arguments compelling.

  43. Dr Spouse said,

    August 2, 2010 at 10:34 am

    It does sound more like he wants to do a randomised trial, but doesn’t know what it’s called.

    One major problem is that even if the trial shows the same as the previous studies (effects on accuracy, not much on other aspects of reading) then teachers, education researchers, and psycholinguists are all fixated on reading accuracy anyway – they think it is the be-all and end-all of reading instruction. So they’ll be happy with the results and implement synthetic phonics.

  44. anewcombe said,

    August 2, 2010 at 11:24 am

    I think Ben’s point is bigger than the phonics or not stuff (interesting as you may or may not find it).

    Medics found that RCTs were instrumental in saving more lives and winning arguments.

    Teachers must also realise that our methods need to be proved to be effective (like, really incontrovertibly proved to be effective) or they will continue to be hijacked by politicians.

    I also hope that we will soon be able to look past the current line that teachers are gifted, intuitive, really actually very hardworking types with paperwork who also have a vocation. It is heroic bullshit.

    @oldandrew
    In defending the current state of educational ‘research’ you are, quite deliciously, flogging a dead horse (but you do it quite stylishly!).

  45. Marcus Hill said,

    August 2, 2010 at 12:00 pm

    As an education researcher, I have many concerns about the state of education research. I’m also a mathematician, but many of my colleagues have little or no mathematical training or aptitude, and so education is replete with qualitative research. This can be genuinely useful in many ways, though it goes against my instincts. In a few cases, however, published studies draw inferences from basically analysing interviews with half a dozen teachers. I kid you not.

    However, the RCT is a very difficult beast to implement, especially when it comes to teaching methods. Teaching is far more art than it is science, and a method that works wonders for one teacher and set of learners may be useless if you change either the teacher or the learners, not to mention any number of other less important factors. For this reason, it’s difficult to accurately measure whether teaching method X works, or whether it only works when delivered by teachers who are enthusiastic about X. Of course, if the massive top down control of teaching states that Y is the only permissible method, you won’t even get enthusiastic proponents to use X in a trial as Ofsted may descend at any point and ask why they’re not toeing the line.

    The best trials are, of course, randomised, but bear in mind that a properly randomised trial needs to have a sufficiently large set of participating schools that they can be split into two groups with comparable (and representative) profiles in terms of a number of factors which inevitably affect educational outcomes: school size, the ethnic and socioeconomic mix of the pupils, school location, the level of experience of the teachers and so on. Couple this with the necessity to have all the teachers involved be genuinely undecided on the merits of the methods being trialled and you can begin to see what a major undertaking this is, and why it is more usual to have smaller trials using sets of comparable schools, but whose results cannot therefore be more broadly generalised.

    Another barrier to education research is that many of us in a position to do it spend most of our time training teachers rather than doing research. We do try to give trainees the tools to access and assess the quality of education research to try to influence these new entrants to the profession to make teaching more evidence based, but the sad fact is that in the UK education is run by suits in Whitehall who want quick fixes and who ignore evidence in this area as much as in other fields. Just one example is the way that so many politicians laud the virtues of teaching pupils in ability groups “especially in subjects like Maths”, when the research evidence tends to indicate that mixed ability teaching in Maths is significantly better for the lower ability pupils and is no worse than teaching in sets for higher ability pupils.

  46. oldandrew said,

    August 2, 2010 at 12:02 pm

    “@oldandrew
    In defending the current state of educational ‘research’ you are, quite deliciously, flogging a dead horse (but you do it quite stylishly!).”

    I’d be the last person to suggest that what passes for education research at the moment is in any way good enough in general. In fact, I think this may be the problem, that the empirical, quantitative tradition in education research might be so weak that people from other discipline do tend to just barge in and say “you must do it like this”.

    However, the existing education research does seem more than adequate for demonstrating the effectiveness of phonics. In fact it could be argued that this is the one and only thing that education research has shown.

  47. raven said,

    August 2, 2010 at 12:03 pm

    The biggest problem with RCT in education is probably the fact that you can’t control a major factor – parents. In medicine you have controlled access to treatment, i.e. patients don’t have a lot of access to prescription medicine outside of your trial. In education, you may have to control for how parents teach reading at home, or use a private tutor.

    It’s not just parents either. I read a trial report on reading which acknowledged problems with the control group because the staff had abondoned the control method and switched to the other method mid trial because it seemed more successful.

    But these difficulties with RCT don’t excuse or explain the lack of good trials in education generally. Funding probably has a part to play, but also who would organise a big RCT? Some of the bigger ones seem to have been organised at LEA level, some through universities, some seem to be funded by the DfE. Is patchy organisation part of the problem?

  48. oldandrew said,

    August 2, 2010 at 12:10 pm

    “The students and their families in less successful groups feel harmed and blame the study. What does one do in response to maintain public will to permit these studies?”

    To be fair, the same complaint could be made in response to medical trials too. After a test is complete a person recieving a placebo or a less effective treatment, even if the only evidence they recieved it is that their condition hasn’t improved, could complain that they have been let down.

    The difference is that in education research nothing can be “blinded” so the complaints can begin as soon as the students (or even the teachers) see what intervention is to take place. Can you tell a teacher that they must teach in a way they are sure is ineffective? Can you tell a parent that they must do nothing about an ineffective teaching method being used on their children?

  49. nezumi said,

    August 2, 2010 at 12:12 pm

    One of the things I do in my job is teaching Dyslexic adults how to spell. I have, anecdotally, had some great results from a system called MUSP (Outlined in Making the Curriculum Work for Learners with Dyslexia by Jenny Lee). I would love to see this tested properly but, as a teacher not a scientist, I wouldn’t have the first idea how to go about it. The program seems to work very well for all kinds of learners, even learners who don’t have Dyslexia.

    It’s very testable because it’s a relatively new system and isn’t shoved down teachers’ throats as phonics is now.

    Although I would stick with some version of phonics* for very early work other methods become useful later on, and I think it is this later stage at which learners are left to their own devices to learn to spell. At this point, phonics becomes a tool rather than a be-all-and-end-all method.

    The thing these politicians don’t seem to realise is that using phonics alone won’t teach anyone anything. It’s just a system for learning to recognise and pronounce the sounds of letters. That’s all! Since the sound of the letters and the spelling are often very different, it’s just not going to work as completely as these people expect. “Know”, “their”, “there”, “two” etc.

    MUSP is good because it treats each word separately, uses visual, audio, sometimes even physical cues, the learner’s own knowledge and experience and the learner’s ability to “take a photo” of something they have paid deliberate, sustained and conscious attention to. It’s not something to use on it’s own, of course, but it’s definitely beneficial.

    The key with spelling is to avoid using one single method to learn every single one. The key is for the learner to choose the method that will best help them remember the spelling of that word.

    *as well as other related methods such as air-drawing, texture mats, 3D letters, my own slant on traditional spelling rules, rhymes and mnemonics, deliberate mispronunciation, grouping words by pattern etc.

  50. oldandrew said,

    August 2, 2010 at 12:25 pm

    “Just one example is the way that so many politicians laud the virtues of teaching pupils in ability groups “especially in subjects like Maths”, when the research evidence tends to indicate that mixed ability teaching in Maths is significantly better for the lower ability pupils and is no worse than teaching in sets for higher ability pupils.”

    You got this from quantitative research? Can you give me the reference, please?

  51. nezumi said,

    August 2, 2010 at 12:42 pm

    Just noticed I made an apostrophe mistake. Should take my own advice and proofread before posting :-P

  52. maizie said,

    August 2, 2010 at 2:19 pm

    I’ve ‘borrowed’ this from someone else’s post on this topic on a completely different forum:

    “The Rose Report 2006, which advocated systematic synthetic phonics and getting rid of multi-cueing, WAS based on good evidence- and this evidence was then checked by the all-party Science & Technology committee who gave it a clean bill of health:www.publications.parliament.uk/pa/cm200506/cmselect/cmsctech/900/900we24.htm

    For an analysis (Prof. McGuinness is trained in statistical analysis) of the Torgersen, Brooks and Hall review:
    dyslexics.org.uk/comment.pdf”

    The Torgerson et al review is that erroneously described by Ben as “…. a systematic review and meta-analysis of all the trials ever to look at phonics” It actually only looked at a tiny number of all the trials ever done…

  53. Marcus Hill said,

    August 2, 2010 at 2:39 pm

    @oldandrew: I don’t have time to dig out a lot of references, but I’ll give you the ones I have to hand. If you hit Google Scholar up for recent papers by S Hallam and J Ireson on grouping, you’ll find a number especially on its effect on pupil motivation in primary schools.

    Rubin, Beth C.(2006) ‘Tracking and Detracking: Debates, Evidence, and Best Practices for a
    Heterogeneous World’, Theory Into Practice, 45: 1, 4 — 14 is a decent (US based) overview of “detracking” (i.e. mixed ability teaching) research (not Maths specific).

    There’s a section on alternatives to abity grouping in maths in the literature synthesis by Slavin, Lake and Groff (2008) at mac.wiki.ciu20.org/file/view/mhs_math_Sep_8_2008.pdf – actually, the review is relevant to this discussion as a whole (as I realised on skimming it after over a year since I last looked at it!) as it has a collection of studies in maths education with some fairly stringent quality criteria for inclusion. One of these is that the studies had to be contolled and either randomised or matched, and there is a bit of analysis at the end of the paper showing that matching seems to give results as good as randomising (end of p40), so “randomised” seems to be a lot less important than “controlled” in education research.

  54. oldandrew said,

    August 2, 2010 at 3:44 pm

    Marcus Hill,

    None of your references above fit the description you gave. Just to remind you, you suggested, in a comment where you appeared to favour quantitative research that “the research evidence tends to indicate that mixed ability teaching in Maths is significantly better for the lower ability pupils and is no worse than teaching in sets for higher ability pupils”.

    Do you actually know of quantitative research which shows this?

  55. raven said,

    August 2, 2010 at 3:55 pm

    @ Marcus – does the evidence you mention show mixed ability teaching for maths is more effective at all ages, or just primary?

    I don’t know if our experience a typical one for the UK – from Yr1 onwards work in groups of 5 or 6 in Maths & English is in ability groups and the work is ‘differentiated’ to match ability?

    I have wondered if perhaps using that system throughout primary is actually limiting progress of some of those in the low ability groups.

  56. David Perkins said,

    August 2, 2010 at 7:07 pm

    @oldandrew It is true that it can’t be properly blinded and that’s a major factor, including feedback effects from parents and teachers during the study, but there are also issues of perceived agency. People who consent to being part of a trial are more likely to accept the consequences of that decision than someone who doesn’t feel they were given a choice. Also, people in medical trails are often not connected beyond them, but students and parents within classes, schools, even districts and beyond tend to be highly connected so their organizing potential is high and their potential sympathetic audience (parents and people who plan to one day be parents) is enormous.

  57. Happy Camper said,

    August 2, 2010 at 9:55 pm

    Project ‘Follow Through’ was the biggest, longest, and most expensive study of instructional methods in education. It lasted several decades, included thousands of students across many schools, and cost 1 billion dollars. Of all 20 or so methods looked at, there was one clear winner: Direct Instruction. This has not been taken up in the US for political reasons, and no-one here in the UK is in any hurry to either.

    The evidence is in. It is time to change over to Direct Instruction, whatever your intuition may tell you.

  58. oldandrew said,

    August 2, 2010 at 10:12 pm

    “This has not been taken up in the US for political reasons, and no-one here in the UK is in any hurry to either.”

    Actually it was government policy (at least in core subjects) under David Blunkett (i.e. 1997-2001). It was abandoned, bit by bit, without debate or explanation, during the long period of drift after Blunkett. It may be policy again soon. Both Michael Gove and Nick Gibb seem to believe in it, although unfortunately they seem more concerned with putting time and money into setting up new types of school. There is also the related issue of how much particular methods should be forced on teachers. Blunkett and his successors were widely seen as overstepping the mark in this respect, and the infra-structure set up to enforce direct instruction methods was also used to remove them.

  59. oldandrew said,

    August 2, 2010 at 10:19 pm

    Apologies for the rogue hyphen in the word “infrastructure” there.

  60. Marcus Hill said,

    August 3, 2010 at 9:47 am

    @oldandrew: Try Boaler, Wiliam and Brown (2000) “Students’ Experiences of Ability Grouping – disaffection, polarisation and the construction of failure” British Educational Research Journal, Vol 26 No 5, and also some of the studies cited therein. I’m pretty sure there was a book by Boaler based on the whole study as well, published around 2002 after the study in the paper I cited was completed, but I don’t have the reference to hand. Also, I chose the words “tends to indicate” carefully, as the evidence is far from overwhelming.

    @raven: Mixed ability seems to be better for the lower ability groups at all ages, but with older children the negative effect on higher ability pupils may (and this is an impression, not evidence based!) increase. It’s difficult to disentangle the effects of grouping on achievement in purely academic terms from the social effects and the expectation of failure in low ability groups that becomes a self fulfilling prophecy. There is also a tendency to put the worst teachers (often not specialists in schools with a shortage of mathematicians) or strings of supply teachers with the lower sets. It is the norm to have within class ability groups in primary schools for Maths and English, though in larger schools there is a trend to set the children for Maths and sometimes English. You can find a picture of the setting practices of schools in England at eprints.ioe.ac.uk/370/1/Hallam2003Ability69.pdf

  61. oldandrew said,

    August 3, 2010 at 1:15 pm

    “Boaler, Wiliam and Brown (2000)”

    This is qualitative research that doesn’t measure academic achievement.

    My question was about quantitative research. Is there any that fits your claim about maths teaching?

  62. oldandrew said,

    August 3, 2010 at 1:44 pm

    Can I take back what I said about Direct Instruction earlier? I thought Happy Camper meant instruction which was direct, I didn’t realise it referred to a specific educational programme entitled “Direct Instruction”.

  63. raven said,

    August 3, 2010 at 3:25 pm

    Marcus -thanks for the link on ability groups in primary. Somebody needs to tell the CPS that !

    Happy Camper – I read some stuff about Project Follow Through last night. There’s a good article here if anyone is interested.
    darkwing.uoregon.edu/%7Eadiep/ft/watkins.htm
    The most pertinent thing to this discussion is why the US education establishment ignored the evidence from the study – pertinent because there’s similar factors at work in the UK ‘reading wars’.

  64. MedsVsTherapy said,

    August 3, 2010 at 3:25 pm

    Interesting discussion. It is kind of ridiculous, though, to argue about which method best teaches children to read. Nearly all of us have learned to read, as do most people, billions, barring significant dire social circumstances.

    We commentors here are all on the high end of ability. So, the scientists should study how we learned to read, and copy that. Well, if you interview our parents, or survey our various early-childhood learning contexts, or survey our kindergartens, you will find a general core of features, and then a great range of variability.

    So, like walking, talking, or potty-training, for nearly all people, nearly all roads lead to Rome.

    Evaluating the group of us high-achievers also will certainly mis-lead, since there might be many of us who received unhelpful methods, yet prevailed due to the amazing forces of natural intellectual development. Some of us were spanked, and some not; what if a huge trial quirks one way or the other with regard to spanking? Would we ban, or institute spanking? No: the “effect” is certainly to be modest, and we know that significant portions of people learn to read either way.

    What does this tell us? For the most part, kids can simply be handed over to the sausage factory and run through, and sausage will come out on the other end. No need to worry about the ugly details along the way.

    Here is the critical issue: how do we help the modest portion who do not naturally learn to read without much specialized attention and curricula?

    For schoolkids in general, it is ridiculous to trial a commercial method versus the vast expanse of dog-eared, stained, spine-broken basic primers. Unless you hold the license and stand to benefit. What benefit might we expect? Kids on average are a half-grade early in reading? A half grade level? Are you kidding me?

    Can’t we forgo the cost (license, retraining, and so on), and just be a bit patient with that second half of a grade level?

    Here is the issue worth studying: 1. identifying those with noted difficulty learning to read, 2. categorizing the types of problems that constitute the majority of those with difficulty (dyslexia, low IQ, low motivation, poor vision, parents fighting too much at home, attention deficit, and so on), 3. addressing each of these challenges, possibly with case study trials, conparator-group trials, and so on. Someone mentioned Bradford Hill: sure, for each type of learning difficulty, work out various means for developing theories about how the reading process fails to happen as it does for most, and marshal differnt types of evidence to test various theories, try some intervetions to test theories, and maybe at some point carry out a controlled trial on some detail, with a theoretical model as a basis.

    Each of those: 1, 2, 3 includes a full research agenda. The only one we might be good at presently is detecting poor/slow/behind-norm readers.

    As a psychologist with some training in assessing IQ and learning problems, plus a lot of tutoring experience, I believe I am good at detecting the psychosocial problems – low motivation, anxiety from family conflict, ant-education attitudes in the family, hunger, lack of discpline in the home, and so on.

    But I am not as good when it comes to neurocognitive problems such as dyslexia or learning disorders. Each of us experts may be strong in our areas, but delineating reading difficulties into types, reliably, for all kids within a school system, is quite a tall order.

    This is no easy matter, and one size does not fit all; “phonics” de rigueur will not solve the problem of all with reading problems – some but not all – and I am pretty sure that the evidence of the billions of readers who never took phonics is proof enough that at best, for the population generally, phonics will deliver at best a slender advantage.

  65. paddyfool said,

    August 3, 2010 at 4:25 pm

    @oldandrew,

    Could you link to some of this overwhelming evidence for phonics?

    @Everyone who knows more about education than me,

    How hard would it really be to design two encouragement and support packages for literacy etc., one that specifically endorses and supports phonics and a control package that doesn’t mention phonics but is otherwise similar? Naturally, the teachers already have favoured methods anyway; what matters is whether the support structure for schools backing any particular method makes any difference whatsoever. (Quite possibly it doesn’t).

  66. oldandrew said,

    August 3, 2010 at 6:26 pm

    “Could you link to some of this overwhelming evidence for phonics?”

    I have no idea what research is or isn’t freely available on the internet and I don’t intend to search for it. However, one thing I do know is available is this, which describes a meta-analysis:

    www.nichd.nih.gov/publications/nrp/upload/ch2-II.pdf

  67. maizie said,

    August 4, 2010 at 10:37 am

    @MedVTherapy, who said;

    “Here is the issue worth studying: 1. identifying those with noted difficulty learning to read, 2. categorizing the types of problems that constitute the majority of those with difficulty (dyslexia, low IQ, low motivation, poor vision, parents fighting too much at home, attention deficit, and so on), 3. addressing each of these challenges, possibly with case study trials, conparator-group trials, and so on. Someone mentioned Bradford Hill: sure, for each type of learning difficulty, work out various means for developing theories about how the reading process fails to happen as it does for most, and marshal differnt types of evidence to test various theories, try some intervetions to test theories, and maybe at some point carry out a controlled trial on some detail, with a theoretical model as a basis.”

    Durh, what a good idea, how extraordinary that no-one has thought of this before…. No-one, that is, apart from an army of cognitive pyschologists, neuroscientists, educationalists etc. who have spent the past 4 decades, at least, investigating every conceiveable aspect of reading and reading difficulties.

    I find it extraordinary that a vast body of work on the subject of reading is completely bypassed or marginalised by any old ‘man in the street’ who can read and who thus knows that the debate is completely pointless…

    While billions of people might be able to read, in major English speaking countries such as the UK, USA, Australia & New Zealand it is understood that up to 25% of the population are semi-literate or illiterate. This represents millions of people who are in many ways disabled by not having this essential skill.

    This is one organisation’s list of research papers on reading. It is by NO means exaustive, but is a start if anyone has the slightest interest in the research into reading (as opposed to offering their own uninformed opinion):

    www.sedl.org/reading/framework/research.html

  68. raven said,

    August 4, 2010 at 11:48 am

    @ paddyfool

    If you look at the government meta-analysis Ben linked to in the article, most of the 20 RCT used were synthetic phonics versus either ‘whole language’ or other phonics methods.

    And it reports ‘moderate’ evidence that systematic phonics (ie some sort of ordered programme of phonics) gives better reading accuracy in both normal children and those at risk of being poor readers. (Table 4)

    It concludes that systematic phonics should remain a routine part of teaching because of that benefit, although evidence is less clear on comprehension & spelling benefits. They couldn’t find RCT comparing the two main versions of phonics – analytical & synthetic. (Roughly analytical is taking a word & breaking it down into sounds, synthetic is starting from the sounds & building words. Analytic is an older more traditional version of phonics in the UK, synthetic is relatively newer.)

    They also, (like nearly every study done in the last 10 years I’ve seen !) recommend a “well-designed RCT…a large UK-based cluster randomized controlled trial to confirm the findings of this review and to investigate further..”

    I think it needs someone in government with a will to fund that & see it through.

  69. raven said,

    August 4, 2010 at 11:59 am

    I should stress again that most primary schools use phonics in their teaching of reading.
    The argument is over how much phonics is needed, is it well enough taught, if using whole word/ sight word methods cause problems for some readers and is phonics the answer to that,and is one particular system of phonics better.

    (There are also wider issues about how early we pick up on failing readers, and how targeted and individual is the help they get, do we monitor their progress enough, even how qualified are the people giving the help – but I’m trying not to sidetrack into those !)

  70. maizie said,

    August 4, 2010 at 12:37 pm

    I like the way that Torgerson et al called for an RCT “to confirm the findings of this review”! Nothing like knowing the answer before you’ve asked the question…

    raven, did you read the McGuinness critique of Torgerson et al?

    dyslexics.org.uk/comment.pdf

  71. Alan Kellogg said,

    August 4, 2010 at 1:05 pm

    Worked for me. But then the following conditions applied.

    1. I have mild dyslexia.

    2. I have Aspergers

    3. Mom was trained in phonics and applied her knowledge rigorously.

    Result, a small 5 year old boy learned to read using phonics where the whole word method had failed him.

    Far as I can see, the study mentioned above is another example of confirmation bias.

    Method bias is a subject it does not cover, method bias being a person’s bias towards or against a method of teaching. A teacher biased against phonics will tend to teach it poorly. When the teacher is not properly trained in applying the method results can be especially poor. I have to wonder, of the teachers involved in the study, how many were properly trained in phonics, and how many were anti-phonics to begin with?

    A study like this in never about the children alone, their teachers must be considered as well.

  72. Marcus Hill said,

    August 4, 2010 at 1:36 pm

    ““Boaler, Wiliam and Brown (2000)”
    This is qualitative research that doesn’t measure academic achievement.
    My question was about quantitative research. Is there any that fits your claim about maths teaching?”

    You missed out the vital phrase I put in following that – “and some of the studies cited therein”. As I mentioned at the outset, I have a few tangentially related references kicking around and, like you, have no intention of spending time looking for papers I read years ago to present as evidence. The issue of setting in maths is tangential to the topic in any case, I only used it as an example of politicians presenting as a panacea something which is counter to or, at best, unsupported by evidence.

  73. oldandrew said,

    August 4, 2010 at 4:53 pm

    “You missed out the vital phrase I put in following that – “and some of the studies cited therein”.”

    This is because the last time you made a vague claim like that I spent far too much time searching for journal articles only to find ones that said something else entirely. Just to ask again, do you have the details of any quantitative studies that show what you claimed?

    “As I mentioned at the outset, I have a few tangentially related references kicking around and, like you, have no intention of spending time looking for papers I read years ago to present as evidence.”

    Like me?

    If this is a reference to paddyfool’s request for evidence from me about phonics, I declined to search on the internet for links. If he’d asked more generally about how I knew what I was claiming then I could have provided more detail. (Hattie’s recent book surveying meta-analyses covers this question and is the source I have most immediately to hand.)

    This is quite different to what you have done. You talked about the importance of quantitative rather than qualitative evidence, then come up with an example of politicians ignoring evidence, only to then repeatedly fail to identify the evidence they are ignoring.

    Needless, to say, I have severe doubts about whether the evidence you mentioned exists (because I have looked into this issue in the past). If I am mistaken I would like to know, and read the evidence I have missed. If you are mistaken I would be grateful if you would admit it instead of sending me on a wild goose chase.

    So I’ll ask again, do you know of any quantitative studies that show what you claimed?

  74. maizie said,

    August 4, 2010 at 5:04 pm

    @Alan Kellog

    You say:”Far as I can see, the study mentioned above is another example of confirmation bias.”

    There have been several studies mentioned on here. Which particular one are you talking about? Can you name it please?

  75. Alan Kellogg said,

    August 4, 2010 at 6:34 pm

    The very first one, mentioned in the original post.

  76. raven said,

    August 4, 2010 at 7:53 pm

    Hi maizie -in reply to #70.
    I read that ‘confirm our findings’ as saying their findings are pretty tentative & need confirming. Plus as Ben said in the article the studies they looked at are small and only one was in the UK.

    Yes I have read that opinion piece. She makes a fair point about studies on normal class teaching versus ones on interventions for poor readers. In that perhaps we out to look at those separately, as it seems reasonable different things might work in those two situations. And that the meta analysis could do with a tighter definition of what ‘systematic’ phonics means.

    I’m off to look up West Dumbartonshire – I read what they put in place wasn’t just phonics, but a whole system to support literacy & that sounds interesting.

  77. maizie said,

    August 4, 2010 at 8:20 pm

    @Alan Kellog
    I still don’t know which ‘study’ you are talking about. Are you talking about Miriam Gross’s ‘report’, which isn’t a ‘study’ at all in the scientific sense, merely and article about the teaching of reading, or are you talikng about the so called ‘meta-analysis’, which isn’t a ‘study’ either?

    What on earth does ‘confirmation bias’ (in its statistical sense) have to do with either of them?

  78. nancy brownlee said,

    August 4, 2010 at 9:47 pm

    Hey, umm, MedVSTherapy- here in the States, the vast majority of people under 40 can’t read. No, really. They can puzzle out a sentence, given a little time, but they can’t really read. They can’t read for information, can’t read a news story and tell you what it said- can’t look up a number in the phone book or a word in the dictionary without help. They’ve been through the see’n’say sausage factories, and they can’t read. I worked for a community college for 7 recent years, and in processing thousands of financial aid applications, I had very few that didn’t need decoding by the applicants. I had 6 on which the kid had misspelled his (or her) name. They can’t read. They all got in to college…

  79. Alan Kellogg said,

    August 5, 2010 at 2:49 am

    What part of “The report examines why one third of children have reading difficulties at the age of 11, and concludes it is because of a lack of discipline, and the absence of a teaching system called “synthetic phonics”. The report contains lots of anecdotes, but barely mentions the evidence. ” don’t you understand? Or are you just being mulish?

  80. Alan Kellogg said,

    August 5, 2010 at 2:51 am

    Confirmation Bias: Looking for what confirms your beliefs in preference to contrary data.

  81. maizie said,

    August 5, 2010 at 10:20 am

    @Alan Kellog

    Oh, sorry. I thought you were using the term more ‘technically’, as of as a critique of a scientific study.

    As a matter of interest, what bl**dy ‘contradictory data’ is there in favour of not teaching synthetic phonics (quantative, not qualitative, please)? o

    I can’t say whether or not Miriam Gross had any preferences one way or the other before she started researching her report. She’s a journalist, not an educational theorist. I doubt you can, either, unless you happen to know her personally or professionally and know that she is a died in the wool synthetic phonics adherent. Accusations of ‘confirmation bias’ would only be valid if you knew for certain that she was deliberately looking for an endorsement of phonics teaching.

    There are parts of her ‘report’, such as the comments on streaming etc. which are completely irrelevant to the main premis; which is that schools should be better implementing the official guidance on the initial teaching of reading.

    You could, if you like, try to tie ‘phonics’ to ‘right wing’but in doing so you would just be repeating a mindless and damaging myth. The education of children should not be ‘political’and it certainly is not for those who are actually delivering it. Academics, theorists and loonies can attach whatever political connotations they like to various pedagogies but those of us who actually work with children are only ‘political’ in that we want all children to achieve to their potential and to ‘do’ in life, rather than be done to. I know SP practitioners of of all political colours.

  82. Marcus Hill said,

    August 5, 2010 at 11:38 am

    @oldandrew: I really didn’t want to get drawn into this, and have alredy spent way too much time on justifying a passing comment, but…

    Two of the references in Boaler et.al. (2000):

    Boaler, J (1997) “Setting, Social Class and Survival of the Quickest”, British Educational Research Journal, Vol. 23, No. 5 (Dec., 1997), pp. 575-595 has a quantitative analysis comparing the results of two similar schools, one of which teaches in sets and one which does not, showing significant advantages in terms of improvements in attainment for mixed ability teaching.

    Boaler, J (1997) “When even the winners are losers: evaluating the experiences of `top set’ students” Journal of Curriculum Studies, vol. 29, no. 2, 165–182 includes a little data showing that long term recall of learning by students in top sets is worse than that of students of similar ability in mixed groups.

    I also happened to remember another paper due to its distinctive title:

    Linchevski, L and Kutscher, B (1998) “Tell Me with Whom You’re Learning, and I’ll Tell You How Much You’ve Learned: Mixed-Ability versus Same-Ability Grouping in Mathematics” Journal for Research in Mathematics Education, Vol. 29, No. 5 (Nov., 1998), pp. 533-554 contains a study in which pupils in an Israeli school were randomly allocated to groups being taught in sets and in mixed ability classes. The results after two years were analysed and showed the lower ability pupils performed significantly better in the mixed ability classes and that the higher ability pupils performed slighly worse, but not to a degree of statistical significance.

    Since you claim to have looked into it, I’d be genuinely interested to see any studies which demonstrate anything contrary to the impression I’ve formed of what the research tends to suggest – that is, that ability grouping significantly disadvantages those in lower ability groups and slightly but less markedly advantages those in higher groups in terms of their academic progress; and that it has marked and generally (though not universally) negative impacts in social and affect terms.

    Incidentally, looking back on my original comment, I never claimed the research evidence in relation to setting was largely quantitative – you conflated my earlier comments on a preference for quantitative research with my final comments on politicians ignoring research in general, this was not my intention, and I apologise if I wasn’t sufficiently clear on this. As with most education research, this subfield is composed of a surfeit of qualitative or small scale quantitative studies with only an occasional larger quantitative study thrown in.

  83. oldandrew said,

    August 5, 2010 at 2:34 pm

    “Boaler, J (1997) “Setting, Social Class and Survival of the Quickest””

    This is largely qualitative, and what hard data there is doesn’t fit your description.

    “Boaler, J (1997) “When even the winners are losers: evaluating the experiences of `top set’ students””

    Again, this is qualitative research.

    “Linchevski, L and Kutscher, B (1998) “Tell Me with Whom You’re Learning, and I’ll Tell You How Much You’ve Learned: Mixed-Ability versus Same-Ability Grouping in Mathematics””

    Well I suppose this is quantitative, but done to such an appallingly low standard as to be utterly worthless.

    Can I take it then that your entire statement on what the evidence shows refers only to qualitative research, and this one, dire, quantitative study?

    “As with most education research, this subfield is composed of a surfeit of qualitative or small scale quantitative studies with only an occasional larger quantitative study thrown in.”

    Do you know of any larger, quantitative studies in this subfield?

  84. Alan Kellogg said,

    August 5, 2010 at 4:05 pm

    Maizie,

    I refuse to answer because you talk mean to me.

  85. Alan Kellogg said,

    August 5, 2010 at 4:06 pm

    Maizie also jumps to conclusions with no supporting evidence, but that’s another story.

  86. msjhaffey said,

    August 5, 2010 at 4:51 pm

    Ben

    You write (quoting Boris) ‘ “It is surely time for the Government to organise a competition, a shoot-out between the two methods, to see which is the most effective for children of all abilities.” Both expand on this idea.’

    But no, he doesn’t expand on it. You’re reading w-a-y too much into his words “shootout” and “competition”. I am quite sure that if you spoke to him and explained the advantage of a rigorous scientific test, he’d endorse it. What he wants is for children to learn to read better and while he might huff and puff at the rigour, I have no doubt he’d support it.

  87. Ben Goldacre said,

    August 5, 2010 at 5:03 pm

    @msjhaffey

    You’re wrong. It is very clear indeed, looking at the full quotes from Boris, and the report whose suggestion he is explicitly endorsing, that both are talking about a competition, and not a meaningful trial.

    Boris in the Telegraph:
    www.telegraph.co.uk/education/7897687/Illiteracy-is-bad-for-us-so-why-dont-we-do-something-about-it.html
    Are they right? It is time to end this culture war, and to try to settle once and for all, in the minds of the teachers, whether synthetic phonics is the complete answer or not. We have in Nick Gibb, the admirable new schools minister, one of the world’s great militants for synthetic phonics. Indeed, you can have a meeting with Nick on almost any subject, and I can guarantee he will have mentioned it within five minutes. I am almost 100 per cent sure he is right.
    And yet I have also met London kids on Reading Recovery programmes who are obviously benefiting hugely from a mixture of phonics and word recognition. It is surely time for the Government to organise a competition, a shoot-out between the two methods, to see which is the most effective for children of all abilities.
    And don’t tell me children are averse to competition. Look at me and my sister.

    Boris in the intro to the Report:
    publications.education.gov.uk/eOrderingDownload/RR711_.pdf
    This is a controversy that has been raging for so long, and with
    such theological intensity, that it is surely time to resolve it once
    and for all. If, as the Centre for Policy Studies suggests, an
    annual competition can be devised to discover which schools
    are best at teaching children how to read – with adequate
    controls – then I would certainly give the venture my full
    support.

    The report itself:
    publications.education.gov.uk/eOrderingDownload/RR711_.pdf
    One step towards achieving this might be to initiate an annual
    contest among London primary schools – a kind of Booker Prize
    for literacy, perhaps sponsored by one of the large corporations
    which have been so vehement in complaining about the poor
    skills of school leavers. The competing schools would be
    independently assessed culminating in three winners and 10
    runners-up. Every child and all the relevant teachers in the
    winning schools would then be given an award at a large prizegiving
    party. The winning schools would get a substantial cash
    award to be spent entirely at the head teachers discretion. The
    teaching methods of the successful schools – as well as the
    conduct and enthusiasm of children – would be analysed so that
    teachers and parents alike can see which approach works best.

    #############################

    Nobody has been misrepresented. These people are very clearly proposing a competition, not a trial, as I explained, and their suggestion is stupid, for the reasons I explained. I see from your link that you’re Sean Haffey, a conservative councillor. I hope you do your best to ensure that there is a good evidence base for policy where possible. If Boris would like to talk about the best way to work out if something works and revise his suggestion then obviously I’d be happy to make the time. In the meantime this was a disappointingly ignorant suggestion about how to develop evidence for policy from a senior politician and a respected think tank.

  88. msjhaffey said,

    August 6, 2010 at 9:12 am

    Dear Ben,

    You’re one of the good guys, as I’m sure most people who follow your blog realise.

    However, we also need to realise that people think and work in different ways. A good politician needs to know a little about a lot. A scientist is typically different, knowing a lot about a little. You have the honesty to quote Boris, including a key phrase “– with adequate controls –”. Boris clearly states his belief that phonics is better, but he’s prepared to be convinced and that phrase is as close as most non-scientists will come to understanding scientific method.

    I genuinely suggest you challenge him to institute or campaign for a fair test. You have the profile to do so. I may be wrong and he might duck the issue, but that’s not how I understand what he wrote.

    On a separate issue, you also wrote “I see from your link that you’re Sean Haffey, a conservative councillor. I hope you do your best to ensure that there is a good evidence base for policy where possible”. I agree completely: too often policy is driven by ideology and not facts. In my limited experience some politicians across the political spectrum get this wrong, and across the political spectrum others get it right. I would love to see the Local Government Association spend some money on educating councillors about how to make policy based on fact. An easy start would be to send every councillor your book. That’s not fawning: I just can’t think os a quicker and easier way (I’m open to suggestions)

    Let’s not assume that scientists are immune from mistakes either. The Sense about Science annual lecture a few weeks ago showed how often scientists get things wrong, as you know (you were just a few yards to my left). As I blogged the next day, we need to ensure the scientific method is less vulnerable to manipulation.

    I expect on occasion, if you continue to have the grace to accept my commenst, that I may disagree with you. After all, healthy debate is what advances knowledge.

  89. Geoff332 said,

    August 6, 2010 at 1:19 pm

    There are a lot of reasons why randomised trials don’t make it into the social sciences to the extent that they do in other sciences. Some of these reasons are good, some are not. I can cherry pick a few (for the record, my own background is Organisation Theory – which tends to draw on multiple disciplines).

    One reason is the complexity involved in isolating the treatment’s effect from everything else. Randomising is meant to do this by converting everything else to noise. When dealing with social sciences, everything else can be too noisy to drown out. I suspect one important factor in this case is the teacher’s commitment to each approach, which will have a major impact on that approach’s efficacy. If, as you suggest, most teachers doubt that the phonetic approach will work, then the results will be strongly biased against that and randomisation won’t remove that bias.

    A second issue is reflexivity – the act of studying behaviour tends to modify it (and often quite radically). This is tied into all sorts of very human things like self-awareness.

    Both of these issues are compounded by the fact that different people respond very differently to different treatments. Some students might learn better from phonetics, while others learn better from more traditional approaches (I have no idea on the evidence around this – but it’s a reasonable hypothesis). However, if the differences are of different magnitude, then the overall impact can be confusing. If phonetics is better at getting more people to read at a basic level, but more traditional approaches lay foundations that improve the long-term ability to read and comprehend at a higher level, then which approach is actually better? In short: neither approach may actually be better in any universal sense.

    Finally, it’s often very, very difficult to quantify outcomes. I suggested a difference between basic literacy and highly skilled readers/writers? The former is relatively easy to measure; the latter is both much harder and occurs over a longer time-scale. The drive to quantify tends to reduce any complex phenomena to those that are easy to measure. In some cases, this is probably a very good thing. In others, it is a very bad thing. I suspect that the measurable impact of Shakespeare or Mozart in their lifetimes was negligible. Their long-term impact is somewhat larger.

    That’s a quick (and very selective) survey of some of the reasons why strict scientific methodology isn’t applied in social sciences. Of course, many of these issues apply in all sciences, including medicine (particularly if you get into public health or placebo type effects).

    Of course I agree with you that a competition is a very bad idea and a randomised trial would be vastly superior. But I’m not sure it would actually answer the question of which approach is better.

  90. dschofi said,

    August 6, 2010 at 4:44 pm

    As some entering teaching as a second career, I was amazed at the amount of guff talked and the lack of an evidence-based approach to education. For an area as important as education it is unbelievable to have teacher-training courses spouting nonsense about learning styles (e.g. VAK learners) and reading-list texts containing absolute rubbish (e.g. one book I flicked through referenced a certain trial in Durham demonstrating the positive effects of Omega-3 oil…). The fact that cognitive scientists have made significant progress in the understanding of thinking, memory and learning in the past 20 years does not appear to impact a significant chunk of the educational establishment. Sadly, education has a long way to go before it reaches the evidence-based approach of medicine and in the meantime we’ll have papers such as that written by Miriam Gros.

    But the comments here shows that there is hope and contained some interesting debate (aside from the tu quoque p1ssing contest). And there is material out there using evidence as a basis for teaching teachers about how we learn (e.g. Daniel Willingham’s excellent book ‘Why don’t students like school?’ – not on any of my reading lists but easily the best book I’ve read on the subject). As teachers we have to strive to ensure that this evidence-based approach is championed and ideas such as running a competition between two schools on reading methods are seen by all to be as ridiculous as running a competition between two hospitals on a certain medical treatment.

  91. mrnam said,

    August 8, 2010 at 10:29 am

    Education and education research would certainly benefit hugely from a more ‘scientific’ approach. Teachers of my vintage grow weary of the latest fad being presented as the only way to success – it is (unfortunately) in the interests of the younger more ambitious teachers to ride the latest hobby horse for all its worth, even when they themselves may not have a complete belief in it (definitely a case of the ‘emperor’s new clothes’). The move to treat schools and their workforces more like competitive commercial enterprises that thrive on individual competition between workers undermines the collaborative approach essential to a successful school, and suppresses critical analysis of new initiatives.

    There is a tendency to start from an intuitive feeling about what works well, and then work back from there to find the evidence to back it up – this to me seems entirely the wrong approach. Some of the proposed new approaches may well be effective and a common sense way of structuring teaching methods. One example is the way we have had (from about 5 – 6 years ago) an approach called ‘Assessment for Learning’ pushed on us (from a booklet “Working Inside the Black Box” – Paul Black at Institute of Education). To me the actual research behind this seemed very limited and unscientific – however there are certainly SOME parts of it that I would say do seem effective – but in no way ALL of it. At the same time as this was being pushed we were also being told that ‘Brain Gym’ was a great idea. Of course, despite strong misgivings among many staff at the time about some of the content, no-one really challenged it directly. Resistance I suppose tended to be more ‘passive’ – people just didn’t bother with the bits they didn’t believe in, or only trotted out the expected methodology during observed lessons and OFSTED inspections!

    Further, given the extremely ‘dubious’ teaching methods (by modern standards) that I (and I am sure many others of you reading this forum) have been subjected to in their youth, it is a wonder that we managed to learn anything at all when at School, but somehow we did!

  92. kim said,

    August 16, 2010 at 11:30 am

    @Ian: “Every time you make some meaningless change everybody’s enthusiasm is renewed for a while because it’s different. ”

    It’s called the Hawthorne effect, and you’re right, it’s why doing research in education is particularly fraught with problems. Every so often you’ll read about some kids who were given Latin lessons or yoga lessons or meditation lessons and there’s a dramatic improvement in their performance or behaviour. Of course there is. If everybody’s suddenly interested in you and making a fuss about you, you’re inevitably going to do better for a while.

    It’s very difficult in education to know what’s a fad and what is based on sound research. The big thing in recent years has been learning styles and the idea that students are visual, auditory or kinaesthetic learners. That will probably pass. In the 80s I was a governor at a primary school and was told by the head that they didn’t teach times tables any more because the current thinking was that understanding mathematical concepts was more important than rote learning.

  93. oldandrew said,

    August 16, 2010 at 4:16 pm

    “The big thing in recent years has been learning styles and the idea that students are visual, auditory or kinaesthetic learners. That will probably pass.”

    It’s been getting challenged for quite a while now:

    teachingbattleground.wordpress.com/2008/08/27/a-helpful-video-on-learning-styles/

  94. Deasun said,

    September 9, 2011 at 3:42 pm

    Looks like Boris has convinced the emenent scientist and sociologist Call Me Dave about the value of synthetic phonics. And you voted for them? God help England if this the besy you have!

    www.guardian.co.uk/education/2011/sep/09/free-school-opponents-defending-failure-says-cameron

    The prime minister said the government would drive up standards by:

    • Ending “wrong-headed methods” that have failed pupils and making sure every teacher has the resources to deliver synthetic phonics teaching. “That’s the method that’s proven to work – and that’s how we can eliminate illiteracy in our country,” he said.

  95. Deasun said,

    September 9, 2011 at 3:43 pm

    Mind you, it looks like my spelling could do with a work-out. Apologies…

You must be logged in to post a comment.