www.guardian.co.uk/education/2011/sep/09/free-school-opponents-defending-failure-says-cameron

The prime minister said the government would drive up standards by:

• Ending “wrong-headed methods” that have failed pupils and making sure every teacher has the resources to deliver synthetic phonics teaching. “That’s the method that’s proven to work – and that’s how we can eliminate illiteracy in our country,” he said.

It’s been getting challenged for quite a while now:

teachingbattleground.wordpress.com/2008/08/27/a-helpful-video-on-learning-styles/

It’s called the Hawthorne effect, and you’re right, it’s why doing research in education is particularly fraught with problems. Every so often you’ll read about some kids who were given Latin lessons or yoga lessons or meditation lessons and there’s a dramatic improvement in their performance or behaviour. Of course there is. If everybody’s suddenly interested in you and making a fuss about you, you’re inevitably going to do better for a while.

It’s very difficult in education to know what’s a fad and what is based on sound research. The big thing in recent years has been learning styles and the idea that students are visual, auditory or kinaesthetic learners. That will probably pass. In the 80s I was a governor at a primary school and was told by the head that they didn’t teach times tables any more because the current thinking was that understanding mathematical concepts was more important than rote learning.

There is a tendency to start from an intuitive feeling about what works well, and then work back from there to find the evidence to back it up – this to me seems entirely the wrong approach. Some of the proposed new approaches may well be effective and a common sense way of structuring teaching methods. One example is the way we have had (from about 5 – 6 years ago) an approach called ‘Assessment for Learning’ pushed on us (from a booklet “Working Inside the Black Box” – Paul Black at Institute of Education). To me the actual research behind this seemed very limited and unscientific – however there are certainly SOME parts of it that I would say do seem effective – but in no way ALL of it. At the same time as this was being pushed we were also being told that ‘Brain Gym’ was a great idea. Of course, despite strong misgivings among many staff at the time about some of the content, no-one really challenged it directly. Resistance I suppose tended to be more ‘passive’ – people just didn’t bother with the bits they didn’t believe in, or only trotted out the expected methodology during observed lessons and OFSTED inspections!

Further, given the extremely ‘dubious’ teaching methods (by modern standards) that I (and I am sure many others of you reading this forum) have been subjected to in their youth, it is a wonder that we managed to learn anything at all when at School, but somehow we did!

But the comments here shows that there is hope and contained some interesting debate (aside from the tu quoque p1ssing contest). And there is material out there using evidence as a basis for teaching teachers about how we learn (e.g. Daniel Willingham’s excellent book ‘Why don’t students like school?’ – not on any of my reading lists but easily the best book I’ve read on the subject). As teachers we have to strive to ensure that this evidence-based approach is championed and ideas such as running a competition between two schools on reading methods are seen by all to be as ridiculous as running a competition between two hospitals on a certain medical treatment.

One reason is the complexity involved in isolating the treatment’s effect from everything else. Randomising is meant to do this by converting everything else to noise. When dealing with social sciences, everything else can be too noisy to drown out. I suspect one important factor in this case is the teacher’s commitment to each approach, which will have a major impact on that approach’s efficacy. If, as you suggest, most teachers doubt that the phonetic approach will work, then the results will be strongly biased against that and randomisation won’t remove that bias.

A second issue is reflexivity – the act of studying behaviour tends to modify it (and often quite radically). This is tied into all sorts of very human things like self-awareness.

Both of these issues are compounded by the fact that different people respond very differently to different treatments. Some students might learn better from phonetics, while others learn better from more traditional approaches (I have no idea on the evidence around this – but it’s a reasonable hypothesis). However, if the differences are of different magnitude, then the overall impact can be confusing. If phonetics is better at getting more people to read at a basic level, but more traditional approaches lay foundations that improve the long-term ability to read and comprehend at a higher level, then which approach is actually better? In short: neither approach may actually be better in any universal sense.

Finally, it’s often very, very difficult to quantify outcomes. I suggested a difference between basic literacy and highly skilled readers/writers? The former is relatively easy to measure; the latter is both much harder and occurs over a longer time-scale. The drive to quantify tends to reduce any complex phenomena to those that are easy to measure. In some cases, this is probably a very good thing. In others, it is a very bad thing. I suspect that the measurable impact of Shakespeare or Mozart in their lifetimes was negligible. Their long-term impact is somewhat larger.

That’s a quick (and very selective) survey of some of the reasons why strict scientific methodology isn’t applied in social sciences. Of course, many of these issues apply in all sciences, including medicine (particularly if you get into public health or placebo type effects).

Of course I agree with you that a competition is a very bad idea and a randomised trial would be vastly superior. But I’m not sure it would actually answer the question of which approach is better.

You’re one of the good guys, as I’m sure most people who follow your blog realise.

However, we also need to realise that people think and work in different ways. A good politician needs to know a little about a lot. A scientist is typically different, knowing a lot about a little. You have the honesty to quote Boris, including a key phrase “– with adequate controls –”. Boris clearly states his belief that phonics is better, but he’s prepared to be convinced and that phrase is as close as most non-scientists will come to understanding scientific method.

I genuinely suggest you challenge him to institute or campaign for a fair test. You have the profile to do so. I may be wrong and he might duck the issue, but that’s not how I understand what he wrote.

On a separate issue, you also wrote “I see from your link that you’re Sean Haffey, a conservative councillor. I hope you do your best to ensure that there is a good evidence base for policy where possible”. I agree completely: too often policy is driven by ideology and not facts. In my limited experience some politicians across the political spectrum get this wrong, and across the political spectrum others get it right. I would love to see the Local Government Association spend some money on educating councillors about how to make policy based on fact. An easy start would be to send every councillor your book. That’s not fawning: I just can’t think os a quicker and easier way (I’m open to suggestions)

Let’s not assume that scientists are immune from mistakes either. The Sense about Science annual lecture a few weeks ago showed how often scientists get things wrong, as you know (you were just a few yards to my left). As I blogged the next day, we need to ensure the scientific method is less vulnerable to manipulation.

I expect on occasion, if you continue to have the grace to accept my commenst, that I may disagree with you. After all, healthy debate is what advances knowledge.

You’re wrong. It is very clear indeed, looking at the full quotes from Boris, and the report whose suggestion he is explicitly endorsing, that both are talking about a competition, and not a meaningful trial.

Boris in the Telegraph:
www.telegraph.co.uk/education/7897687/Illiteracy-is-bad-for-us-so-why-dont-we-do-something-about-it.html
Are they right? It is time to end this culture war, and to try to settle once and for all, in the minds of the teachers, whether synthetic phonics is the complete answer or not. We have in Nick Gibb, the admirable new schools minister, one of the world’s great militants for synthetic phonics. Indeed, you can have a meeting with Nick on almost any subject, and I can guarantee he will have mentioned it within five minutes. I am almost 100 per cent sure he is right.
And yet I have also met London kids on Reading Recovery programmes who are obviously benefiting hugely from a mixture of phonics and word recognition. It is surely time for the Government to organise a competition, a shoot-out between the two methods, to see which is the most effective for children of all abilities.
And don’t tell me children are averse to competition. Look at me and my sister.

Boris in the intro to the Report:
publications.education.gov.uk/eOrderingDownload/RR711_.pdf
This is a controversy that has been raging for so long, and with
such theological intensity, that it is surely time to resolve it once
and for all. If, as the Centre for Policy Studies suggests, an
annual competition can be devised to discover which schools
are best at teaching children how to read – with adequate
controls – then I would certainly give the venture my full
support.

The report itself:
publications.education.gov.uk/eOrderingDownload/RR711_.pdf
One step towards achieving this might be to initiate an annual
contest among London primary schools – a kind of Booker Prize
for literacy, perhaps sponsored by one of the large corporations
which have been so vehement in complaining about the poor
skills of school leavers. The competing schools would be
independently assessed culminating in three winners and 10
runners-up. Every child and all the relevant teachers in the
winning schools would then be given an award at a large prizegiving
party. The winning schools would get a substantial cash
award to be spent entirely at the head teachers discretion. The
teaching methods of the successful schools – as well as the
conduct and enthusiasm of children – would be analysed so that
teachers and parents alike can see which approach works best.

#############################

Nobody has been misrepresented. These people are very clearly proposing a competition, not a trial, as I explained, and their suggestion is stupid, for the reasons I explained. I see from your link that you’re Sean Haffey, a conservative councillor. I hope you do your best to ensure that there is a good evidence base for policy where possible. If Boris would like to talk about the best way to work out if something works and revise his suggestion then obviously I’d be happy to make the time. In the meantime this was a disappointingly ignorant suggestion about how to develop evidence for policy from a senior politician and a respected think tank.

You write (quoting Boris) ‘ “It is surely time for the Government to organise a competition, a shoot-out between the two methods, to see which is the most effective for children of all abilities.” Both expand on this idea.’

But no, he doesn’t expand on it. You’re reading w-a-y too much into his words “shootout” and “competition”. I am quite sure that if you spoke to him and explained the advantage of a rigorous scientific test, he’d endorse it. What he wants is for children to learn to read better and while he might huff and puff at the rigour, I have no doubt he’d support it.

I refuse to answer because you talk mean to me.

This is largely qualitative, and what hard data there is doesn’t fit your description.

“Boaler, J (1997) “When even the winners are losers: evaluating the experiences of `top set’ students””

Again, this is qualitative research.

“Linchevski, L and Kutscher, B (1998) “Tell Me with Whom You’re Learning, and I’ll Tell You How Much You’ve Learned: Mixed-Ability versus Same-Ability Grouping in Mathematics””

Well I suppose this is quantitative, but done to such an appallingly low standard as to be utterly worthless.

Can I take it then that your entire statement on what the evidence shows refers only to qualitative research, and this one, dire, quantitative study?

“As with most education research, this subfield is composed of a surfeit of qualitative or small scale quantitative studies with only an occasional larger quantitative study thrown in.”

Do you know of any larger, quantitative studies in this subfield?

Two of the references in Boaler et.al. (2000):

Boaler, J (1997) “Setting, Social Class and Survival of the Quickest”, British Educational Research Journal, Vol. 23, No. 5 (Dec., 1997), pp. 575-595 has a quantitative analysis comparing the results of two similar schools, one of which teaches in sets and one which does not, showing significant advantages in terms of improvements in attainment for mixed ability teaching.

Boaler, J (1997) “When even the winners are losers: evaluating the experiences of `top set’ students” Journal of Curriculum Studies, vol. 29, no. 2, 165–182 includes a little data showing that long term recall of learning by students in top sets is worse than that of students of similar ability in mixed groups.

I also happened to remember another paper due to its distinctive title:

Linchevski, L and Kutscher, B (1998) “Tell Me with Whom You’re Learning, and I’ll Tell You How Much You’ve Learned: Mixed-Ability versus Same-Ability Grouping in Mathematics” Journal for Research in Mathematics Education, Vol. 29, No. 5 (Nov., 1998), pp. 533-554 contains a study in which pupils in an Israeli school were randomly allocated to groups being taught in sets and in mixed ability classes. The results after two years were analysed and showed the lower ability pupils performed significantly better in the mixed ability classes and that the higher ability pupils performed slighly worse, but not to a degree of statistical significance.

Since you claim to have looked into it, I’d be genuinely interested to see any studies which demonstrate anything contrary to the impression I’ve formed of what the research tends to suggest – that is, that ability grouping significantly disadvantages those in lower ability groups and slightly but less markedly advantages those in higher groups in terms of their academic progress; and that it has marked and generally (though not universally) negative impacts in social and affect terms.

Incidentally, looking back on my original comment, I never claimed the research evidence in relation to setting was largely quantitative – you conflated my earlier comments on a preference for quantitative research with my final comments on politicians ignoring research in general, this was not my intention, and I apologise if I wasn’t sufficiently clear on this. As with most education research, this subfield is composed of a surfeit of qualitative or small scale quantitative studies with only an occasional larger quantitative study thrown in.

Oh, sorry. I thought you were using the term more ‘technically’, as of as a critique of a scientific study.

As a matter of interest, what bl**dy ‘contradictory data’ is there in favour of not teaching synthetic phonics (quantative, not qualitative, please)? o

I can’t say whether or not Miriam Gross had any preferences one way or the other before she started researching her report. She’s a journalist, not an educational theorist. I doubt you can, either, unless you happen to know her personally or professionally and know that she is a died in the wool synthetic phonics adherent. Accusations of ‘confirmation bias’ would only be valid if you knew for certain that she was deliberately looking for an endorsement of phonics teaching.

There are parts of her ‘report’, such as the comments on streaming etc. which are completely irrelevant to the main premis; which is that schools should be better implementing the official guidance on the initial teaching of reading.

You could, if you like, try to tie ‘phonics’ to ‘right wing’but in doing so you would just be repeating a mindless and damaging myth. The education of children should not be ‘political’and it certainly is not for those who are actually delivering it. Academics, theorists and loonies can attach whatever political connotations they like to various pedagogies but those of us who actually work with children are only ‘political’ in that we want all children to achieve to their potential and to ‘do’ in life, rather than be done to. I know SP practitioners of of all political colours.

What on earth does ‘confirmation bias’ (in its statistical sense) have to do with either of them?

Yes I have read that opinion piece. She makes a fair point about studies on normal class teaching versus ones on interventions for poor readers. In that perhaps we out to look at those separately, as it seems reasonable different things might work in those two situations. And that the meta analysis could do with a tighter definition of what ‘systematic’ phonics means.

I’m off to look up West Dumbartonshire – I read what they put in place wasn’t just phonics, but a whole system to support literacy & that sounds interesting.