“Exams are getting easier”

August 21st, 2010 by Ben Goldacre in bad science, evidence, numerical context, politics, schools | 124 Comments »

Ben Goldacre, The Guardian, Saturday 21 August 2010

Pass rates are at 98%. A quarter of grades are higher than an A. This week every newspaper in the country was filled with people asserting that exams are definitely getting easier, and then other people asserting that exams are definitely not getting easier. The question for me is always simple: how do you know?

Firstly, the idea of kids getting cleverer is not ludicrous. The Flynn Effect is a term coined to describe the gradual improvement in IQ scores. This has been an important problem for IQ researchers, since IQ tests are peer referenced: that is, your performance is compared against everyone else, and the scores are rejigged so that the average IQ is always 100. Because of the trend to greater scores, year on year, you have to be careful not to use older tests on current populations, or their scores come out spuriously high, by the standards of the weaker average population of the past. Regardless of what you think about IQ tests, the tasks in them are at least relatively consistent. That said, there’s also some evidence that the Flynn effect has slowed in developed countries more recently.

But ideally, we want to address the exams directly. One research approach would be to measure current kids’ performance on the exams of the past. This is what the Royal Society of Chemistry did in their report “The Five Decade Challenge” in 2008, running the project as a competition for 16 year olds, which netted them 1,300 self-selecting higher-ability kids. They sat tests taken from the numerical and analytical components of O-level and GCSE exams over the past half century, and performance against each decade rose over time: the average score for the 1960s questions was 15%, rising to 35% for the current exams (though with a giant leap around the introduction of GCSEs, after which score remained fairly stable).

There are often many possible explanations for a finding. Their result could mean that exams have got easier, but it could be that syllabuses have changed, and so modern kids are less prepared for old style questions. When the researchers delved into specific questions, they do say they found some things that were removed from the GCSE syllabus because they’d moved up to A level, but that’s drifting – unwelcomely – into anecdote.

Another approach would be to compare performance on a consistent test, over the years, against performance on A levels. Robert Coe at Durham University produced a study of just this for the Office of National Statistics in 2007. Every year since 1988 they’ve given a few thousand children the Test of Developed Abilities, a consistent test (with a blip in 2002) of general maths and verbal reasoning skills. Scores on this saw a modest decline over the 1990s, and have been fairly flat for the past decade. But the clever thing is what they did next: they worked out the A level scores for children, accounting for their TDA scores, and found that children with the same TDA score were getting higher and higher exam results. From 1988 to 2006, for the same TDA score, levels rose by an average of 2 grades in each subject.

It could be that exams are easier. It could be that teaching and learning have improved, or teaching is more exam focused, so kids at the same TDA level do better in A levels: this is hard to measure. It could be that TDA scores are as irrelevant as shoe size, so the finding is spurious.

Alternatively, it could be that exams are different, and so easier with respect to verbal reasoning and maths, but harder with respect to something else: this, again, is hard to quantify. If the content and goals of your exams change, then that poses difficulties for measuring their consistency over time, and it might be something to declare loudly (or consult employers and the public over).

Our last study thinks more along those lines: some people do have clear goals from education, and they can measure students against this yardstick over time. “Measuring the Mathematics Problem” is a report done for the Engineering Council and other august bodies in 2000, analysing data from 60 departments of maths, physics and engineering who gave diagnostic tests on basic maths skills to their new undergraduates each year. They found strong evidence of a steady decline in scores on these tests, over the preceeing decade, among students accepted onto degree courses where they would naturally need good maths.

Sadly they didn’t control for A level grade, so we can’t be sure how far they were comparing like with like, but there are a few explanations. Maybe maths syllabuses changed and were less useful for maths and engineering degrees. Maybe the clever kids were doing English to become lawyers instead. Or maybe exams got easier.

If you know of more research, I’d be interested to see it, but the main thing that strikes me here is the paucity of work in the field. There’s a man called Rupert Sheldrake who believes that pets are psychic: they know when their owners are coming home, that sort of thing. Obviously we disagree on a lot, but we chat, and are friendly, and once when we were talking he came out with an excellent suggestion: maybe 1%, maybe 0.01% of the total UK research budget could be given to the public, so that they could decide what their research obsessions were. Obviously most of the money would get spent on psychic pets, and which vegetables cure cancer, but I’d like to think that some of it, possibly, might get spent on good quality robust research to find out whether exams are getting easier.

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

124 Responses

  1. jamesc said,

    August 29, 2010 at 11:28 am

    The London Mathematical Society produced this in 1995 and is very relevant to the current discussion.


  2. eddwilson said,

    August 29, 2010 at 7:58 pm

    I could believe dumbing-down skulduggery if the results soared for year n compared with n-1, but 20 years of incremental advance? That would require an amazing amount of careful planning and implementation which could not stay hidden.

  3. penglish said,

    August 30, 2010 at 2:23 am

    There seem to be two purposes for exam results. A secondary purpose is to monitor the quality of teaching, learning, and testing. This latter one is harder, as we have observed grade inflation that is probably not really because people are becoming genuinely cleverer.

    The main purpose of exam results, however, is to distinguish between the achievements of individual students, mainly within the same year. I can see no reason why the grades given could not be given as centiles, with the best 1% of students being in the 99th centile, and so on. Or even, why the actual marks could not be published. If the student’s actual mark, and the average mark for the year were also published, this would go some way to comparing students in different years.

    In reality, if we make the (possibly not exactly true, but likely to be very close to the truth) assumption that people on average are equally intelligent year on year, then knowing which centile a student was in would be a pretty good discriminator between results.

    This all presumes, of course, that the exams accurately measure something important, and which they are setting out to measure, of course – another matter again.

  4. irishaxeman said,

    August 30, 2010 at 8:11 pm

    Probably the most useful course I’ve ever taught on, moving Post 16 students on to uni, was BTEC National Health Science. It was largely project-based. Skills were paramount.

    A-levels currently are possibly fit for purpose for a few subjects (sciences most usefully) but far better tests and a far better education system eludes the blinkers of the political set. The Bacca is better but not perfect. Frankly I cannot for the life of me see why a foreign language is not mandatory from primary school onwards – most other countries manage at least one if not more, for all levels of accomplishment.

    All citizens should be literate, numerate, scientifically literate, have a foreign language, geographical nous and a grasp of history beyond the Oxbridge Greek lovers or the post-modernists that have dominated C20 specifications. I teach a short and very popular general studies topic which traces the development of what is now Britain from first inhabitants 1/2mya. Most students ask why this is not taught in history. The curiosity is there, the educational system isn’t.

  5. rhymos said,

    August 30, 2010 at 8:43 pm

    The main reason why research is so limited in this field relates to the way teachers are taught to research on higher level qualifications at national and international scale. The scientific model has been replaced by action research. This methodology is a ‘flexible spiral process which allows action (change, improvement) and research (understanding, knowledge) to be achieved at the same time’ (Dick 2002 p. 2). This is a methodology takes out many of those pesky things like pre-protocols, blinding, control groups and statistical analysis. Check out the National College of School Leadership website (www.ncsl.org.uk) to look at the standards of research presently being promoted. Alternatively a good way into the educational establishments views on research methodology try www.ngfl-cymru.org.uk/eng/pedagogy-conf-o8.
    DICK, B. Action Research: Action and Research. www.sci.edu.au/schools.html2002

  6. rhymos said,

    August 30, 2010 at 9:09 pm

    Any UK based research has to take into account the great divide between private and public sectors. Any meaningful study has to separate pupils into these discrete groups and then examine how their achievements have changed. I suspect grade inflation amongst pupils from private schools have risen at a much slower rate than those from public schools. Our kids are catching up. Societies response – move the goalposts The new A* is a marketing tool for the private sector to differentiate their product.

  7. NorthernBoy said,

    August 31, 2010 at 12:20 pm

    rhymos, what is this great divide between private and public sectors which you mention, and what justification do you have for proposing this as the important divide? Would it not make more sense to consider, for example, selective against nonselective schools, or those with streaming against those which put all abilities into the same class (not to be confused with comprehensive schooling, where all abilities are in the same school, but where there can still be different classes for different abilities)?

  8. rhymos said,

    August 31, 2010 at 4:49 pm

    NorthernBoy – 7% of UK pupils go to private schools yet these pupils dominate the top universities and professions. Remember if the new cabinet reflected the UK’s population as a whole then 4 million of us would have gone to Eton. The first set of A* results are dominated by pupils from private schools. The divide is central to understanding what has happened to UK schools results. All your other suggestions for investigation are sensible and should be considered in explaining differences within the public sector. You never get far in research when you compare chalk to cheese and pretend they are the same.
    I am an experienced teacher and my view is the main reason the data has changed is state teachers are much more effective at getting pupils to pass exams (refer to comment 91). I am not suggesting for a moment that state teachers are better educators than past teachers only better at “playing the game”.

  9. irishaxeman said,

    September 1, 2010 at 5:48 pm

    The NCSL is a nest of vipers dedicated to working teachers until they crack. The only interest is in preserving their salaries by hitting the statistical targets. Real research is of no interest – they are ideologues not scientists.
    Some heads stand out with their dedication to real education. The number gets smaller every year.
    Bitter and twisted? Biased by 30+ years experience?

  10. grumpy bloke said,

    September 1, 2010 at 9:25 pm

    In John Pilger’s excellent book,Heroes, there is a telling(standby for really bad paraphrasing) statement where a senior Minister or Civil Servant is quoted as saying that it is bad enough having 4 million people unemployed and even worse if 4 million well educated people were unemployed. This was a pre-cursor to the GCSE system.
    After last years GCSE results I followed a few threads on the exams getting easier debate. One of note was that research by Norwich? University indicated that todays A* grade was equivalent to the old GCE grade 6 or F.
    As for private schools, results mean income. My ex wife taught English in a state school and the GCSE was 100% coursework with controlled submissions by the pupils to enable the examining board to check the standard of grading by the teacher. On a day exchange with a local private school these controlled submissions were marked and corrected by the teachers and handed back to the pupils in order to re-submit them.
    I agree with rhymos (108) that teachers are more effective at getting pupils to pass exams and the curriculum material is tailored to the exams e.g. when looking at a French GCSE paper in my step-daughter’s school I recognised many of the questions from the coursework completed that year. In other words pupils are being coached to pass exams, whether this is the right approach or not I don’t know.
    It would be interesting to see how our education system compares to our European partners.

  11. Ted Ainsworth said,

    September 3, 2010 at 10:42 pm

    Here’s why:

    1. Schools compete to get the best league table results.
    2. Schools are able to select which exam board syllabus they teach
    3. Some exam boards set easier exams than others
    4. The exam boards therefore compete for business by setting the easiest exams/courses they can get away with, including methods such as modular courses and assessed project rather than 100% exam.

    Government has turned a blind eye to this because the improving grades make good, but now unconvincing headlines. The remedy is a single, national exam board, that is independent of government influence. The conservatives included this in their manifesto, so we’ll see…

    When I did A levels back in 1995, we did past papers from the previous 5 years as preparation. The older ones were much harder than the more recent, and there was significant content that had been cut from the syllabus just in those 5 years.

  12. Bonsai said,

    September 4, 2010 at 12:20 pm

    An older female academic, successful professor, hod, permanent employee but post-menopausal and pre-retirement – informed a group of younger female researchers that ‘these days a Bachelor of Honours is the equivalent of what A’levels used to be (back in the 1600s or something?!).

  13. Bonsai said,

    September 4, 2010 at 12:20 pm

    oops missed the apostrophe – do you have an edit function?

  14. Bonsai said,

    September 4, 2010 at 12:21 pm

    err I mean quotation marks. Maybe she had a point?

  15. ferguskane said,

    September 5, 2010 at 11:23 am

    The ‘Flynn Effect’ and ‘the idea of kids getting cleverer is not ludicrous’.

    IQ tests are used to obtain an estimate of intellectual abilities. As morph says, what they actually do is test one’s abilities on a selected range of specific tasks. This makes IQ tests pretty good at finding intellectual deficits that may result from trauma or developmental difficulties – although they have many well known flaws even here (problems for instance with ecological validity). What they don’t really do well is measure any kind of ‘innate intelligence’. That’s a tad harder!

    One’s scores on such a test are clearly and demonstrably affected by cultural/educational factors, as well as by one’s ‘innate’ intelligence (although I guess ‘innate’ is rather hard to define here). Education and culture clearly change over time and will affect average IQ scores.

    So: ‘the idea of kids getting cleverer is not ludicrous’. No it’s not. Especially if you consider ‘clever’ to mean ‘good at IQ tests’. But if you mean clever to mean something else, some evidence would be nice.

  16. Chenneth said,

    September 5, 2010 at 11:42 pm

    Comment 111 is pretty spot on. I work in futther education and from my unscientific expereince there is a dramatic decline in standards.
    1. Where as a student would previously need 5 GCSE grade C and above, including English and Maths to go onto A levels the college I work at at(biggest one in the country) this has been reduce to 4 without English or maths, all a student needs now is to pass a level 2 literacy and numeracy exam (www.keyskills4u.com/tests/) have a go an you tell me if you think this is the equal to a GCSE, becauase it’s considered equally when taking on students. So as an example I was forced to take on a student who has an E in English and a D in maths but managed to pass the basic skills exams… is it realistic to expect this student to pass A levels?
    2. This has a futher effect of lowering standards because lecturers are expected to get certain pass rates or face the sack therefore staff will do ANYTHING to get these student through their exams… hence why many courses at the college have a 100% pass rate.
    3. The college is paid for the number of students it takes and the number who pass… end product eveyone pass and the qualification is worthless as it is too easy and does not seperate the good from the bad.
    4. The government is able to porduce glowing reports of improvements in education and continues to roll out even lower standards and the cycle of decline begins again.
    5. All the lecturers have taken level 2 numeracy tests… why? it increases the number of passes and so the amount of money that can be claimed from the government.
    6. The college is a private business that sells qualifications… the more it sells the more money it makes… anyone see the confilct of interest?
    I am far more worried about the privatisation of education and dubious business models being used within the sector.

  17. grumpy bloke said,

    September 6, 2010 at 9:37 pm

    Re 116, your first sentence alone is evidence of a decline in standards. I hope your subject is not english.

  18. davevincent said,

    September 7, 2010 at 7:41 am

    This article and the ridiculous discussion reminds me of an amusing paper: ‘Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials’ (see www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/).

    Of course exams are getting easier, you fools! Doesn’t mean kids are any cleverer or more stupid than they used to be, though.

  19. NorthernBoy said,

    September 9, 2010 at 12:26 pm

    Chenneth, I have to agree with grumpy bloke, your writing is absolutely execrable, and it’s clearly not only typographic errors. If you are seeing standards even lower than those which you demonstrate, then we are indeed in a bad state. I know that it’s impolite to mention someone’s poor writing skills, but it seems relevant when you are questioning the literacy of others. This, I suppose, though, is a separate issue (although a related one), to the question of grade inflation.

    As we see more and more people gaining the higher grades, recruiters are going to need to start placing more emphasis on higher qualifications, and hopefully the most able students are still going to rise to the top eventually, and show their ability in other ways. Gade inflation is an issue, but as long as discrimination occurs eventually, it’s not a fatal problem for people seeking to rank candidates.

    I never crunched the statistics when I was a recruiter, to compare across years, but as we started looking globally, one of the metrics that turned out to be a good guide was quality of university degree. The guy with the engineering degree from Imperial may have similar GCSEs to the one who studied at a poorer institution, but it was very rare indeed to find a greater level of ability in the latter.

  20. sockpuppet said,

    September 10, 2010 at 12:09 pm

    1-Most of the moaners about educational standards were educated many years ago.
    2-Science is better understood now. If you take the view that science is only a hundred or so years old(as an average)then thirty years of progress may create a significant change in the field leading to better understanding and improved teaching. Particularly if in your purview of the last thirty or forty years you consider the contribution of IT. So it may also be that the subjects just aren’t as hard as they used to be but that doesn’t excuse them from being easier. If that is the case perhaps we should expect more from our students rather than less. To make up the shortfall if you will.
    In the meantime rpbt1 will have to deploy his own maths and verbal reasoning tests if he is not satisfied with the current lot from the exam board.

  21. Su_e said,

    September 12, 2010 at 8:32 pm

    Exams have got much much easier – I still have some of my old O-level papers, and picking one that is not subjectively marked (maths); I can breeze through a modern GCSE paper, but I would have to do some revision to do well on my late 1970s paper.
    Yes, I know I fall into the anecdotal camp, but we could test this out by scanning O-level papers and giving them to people of their vintage along with today’s GCSE papers. This gets rid of the problem of a trial of teenagers not being fair as they have only studied for the modern exams

  22. Ely said,

    October 2, 2010 at 2:54 pm

    Well, I both dumbing down and students becoming better at it have to do with it.

    Most schools stick to the same exam boards because the more years the teacher has spent teaching the specification the better he can actually “teach the exam”. I’ve seen it. My teachers tell me that so openly (because I’m not native to the UK), is frightening. Nowadays, books have harder questions than the exam (specially in science subjects, which I’m taking).

    Also, students are actually quite smart. If the only thing that you’ve to do to learn is “box ticking” and experience doing exams, they’re going to tick the boxes AND look for someone who already went through it and pester that person for how to give the perfect exam answer. It becomes even easier if they insist on dumbing down the papers.

    It would be easier if there was a draft of a National curriculum and the school are free to teach more (not less) if so they wish and schools were in control of their own exams, made by each department, with a government watch-dog body to maintain the exams’ standards and to ensure the kids are learning what they’re supposed to be learning, if not more.

  23. Phil said,

    October 7, 2010 at 5:18 pm

    Unfortunately I know from my own personal experiance that exams are indeed getting easier. I took my GCSE’s in 1989 (the first year that they were introduced) and then subsequently A-Levels in 1991. For A-level maths exam practice we were give O-Level papers form the 1960’s/70s that contained, amoungst other things Calculus which was’nt even on the GCSE sylabus. Also in my first year of Uni (Science degree) they had to lay on extra maths courses to get the students up to an acceptable level.

  24. Edward said,

    May 12, 2015 at 3:23 pm

    The consensus seems to be that they are getting easier, but my personal experience from doing past papers says otherwise (only in AS level Maths (AQA)). When I started revising I was not doing well because I struggled to complete the papers in the time allowed, content wise I was largley fine though, it was just figuring out how to apply the knowledge. In past papers I did from 2006, 2007 and 2008 there was a noticble difference in camparison to papers from 2011, 2012 and 2013. In the earlier papers the questions where very direct naming individual process’ however in the newer papers the approach is much vaguer and requires you to have a higher understanding to apply the concepts. Despite this however the grade boundries are very similar. Perhaps this is just me, has anyone else experienced this trend with other exam boards/subjects? If so it seems like a backwards way to go about things, slowing people down by making the questions harder to understand whilst keeping the content the same.