Calling all science teachers

August 23rd, 2007 by Ben Goldacre in bad science | 285 Comments »

Send me your old O level, CSE, GCSE and A level science and maths papers, any format, any condition, email ben@badscience.net, fax 020 7117 3593, snailmail 119 Farringdon Rd EC1R 3ER. Anything more than ten years old gratefully received. Ideally I’d like the accompanying marking schemes, where possible, but I’ll take anything you’ve got!

I’m cracking on at work stuff today, but the GCSE results are out, and the inevitable squabbles are ensuing over the dumbing down of education.

It occurs to me that this is a very simple and familiar problem: on the reliability and validity of a test.

Test reliablility refers to the degree to which a test is consistent and stable in measuring what it is intended to measure. Most simply put, a test is reliable if it is consistent within itself and across time.

Test validity refers to the degree to which the test actually measures what it claims to measure. Test validity is also the extent to which inferences, conclusions, and decisions made on the basis of test scores are appropriate and meaningful.

Now clearly any thoughts or data you might have addressing the question of reliability, of consistency across time, is going to be confounded by the fact that the mission has changed: examiners are quite free about the fact that they aim to measure “different stuff” these days. Science education has been made into a bit of a joke, for example, and in many schoools they’ve stopped doing proper science GCSE’s in favour of “double science”.

However there are interesting things that can still be collected, and I need your help.

Firstly, we can look at whether, even when the mission has remained the same over time, the questions have remained similar.

But more than that, we can construct a narrative that people can simply experience for themselves. Can you believe I just said that? What I mean is: can people – and I know a lot of you are science teachers – send me examples of old science and maths papers which I can post, in the public interest? Either whole papers, or silly examples? Is there a good archive of old O Level science papers we can compare them to?

My fax number is 020 7117 3593, you can bung stuff at me anonymously or in the interests of your own microfame, at your choice, and if other people have blogged on this, do let me know. I’m happy to hold off writing on it until next week if it means doing it properly, and hang timeliness in the name of having something really interesting to say, but it would be good if I could bash it out tonight.

If we get a lot I’ll bung them all up in a nice database for everyone to see.

And remember: Positive Internet are gods.

Update:

Almost nothing more than ten years old coming in, if you can help, or know someone who can, please do get in touch, or just bung the papers over. There’s almost nothing online more than five or ten years old, except for this..

www.cambridgeassessment.org.uk/downloads//exampapers/maths_html


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

285 Responses



  1. jackpt said,

    August 23, 2007 at 12:48 pm

    Good idea because all I’m hearing is anecdotes. Science and mathematics are perfect subjects for historical comparison. Taking into account different devices like slide-rules and calculators.

  2. flange said,

    August 23, 2007 at 12:54 pm

    What are you hoping to show though? All an exam is supposed to show is that you’ve absorbed a certain level of knowledge in a subject. i would say that at GCSE level all you’re really showing is a vague understanding of the basics i.e. enough to get you through day to day life

  3. Matt J said,

    August 23, 2007 at 12:55 pm

    They’ve stopped doing science GCSE’s? That’s news to me, I must have hallucinated the three results I just got in Biology, Chemistry and Physics.

  4. megachicken(b) said,

    August 23, 2007 at 1:01 pm

    There’s some 2006 double science papers free to download here:

    www.aqa.org.uk/qual/gcse/sci_da_m_assess.html

    Example question: Which organ on this diagram of a racoon allows it to see fish? Its fur, its ears, its nose, or its eyes.

    Mind you, it’s unfair to pick out the easiest questions to mock – if you’re going to guard against a ‘floor’ effect, you do need to include some pretty easy stuff in the exam.

    I shudder to think what grade I’d get in biology now if I retook it!

  5. manigen said,

    August 23, 2007 at 1:02 pm

    It is irritating that most of the people insulting the current crop of GCSE graduates – and the A-level lot before that – haven’t even looked at an exam paper in three decades.

  6. Ben Goldacre said,

    August 23, 2007 at 1:13 pm

    well let’s look at some. i’ve got some gcse physics ones from three years ago and they’re a bit of a joke, frankly. if exams are getting easier then it is the examiners who are insulting children.

  7. nohassel said,

    August 23, 2007 at 1:39 pm

    Ben – post the physics paper. I’m still peeved that I had to take “Double Science” GCSE in 1992, but the timetable (a double and a single lesson per week in EACH of physics, chemistry and maths) and curriculum were based on the separate subjects previously used. I was done out of an additional GCSE.

    I went on to do physics at university.

  8. nohassel said,

    August 23, 2007 at 1:40 pm

    I meant physics chemistry and biology – NOT maths.

    Doh!

  9. Ross Burton said,

    August 23, 2007 at 1:57 pm

    Back in my day (I’m only 28, so we’re talking mid-90s) our school generally did double GCSE science, but the brighter students could take the double science in November and if they did well, go on to take the separate sciences in the summer.

  10. edd said,

    August 23, 2007 at 2:14 pm

    Like Despard, I resent any implication that double science does not make for proper GCSEs. There’s less content for sure, but it certainly made absolutely no difference when it came to me becoming a professional physicist.

  11. wilsontown said,

    August 23, 2007 at 2:49 pm

    I too did dual award science, and I now have a PhD and work in research. But I think this might miss the point somewhat. After all, I’ve done A levels and two degrees since my GCSEs. The question is, is the science education given at dual award GCSE adequate as a general science education for people who won’t go on to do further study?

  12. julian said,

    August 23, 2007 at 3:09 pm

    I’ve found that AQA are generally the revising student’s best friend when it comes to past papers and mark schemes, take your pick from this lot for GCSE www.aqa.org.uk/admin/qp-ms_gcse.php and www.aqa.org.uk/admin/qp-ms_gce.php for AS/A-Level Science (I know you didn’t ask, but I figured you might be interested nonetheless).

    Edexcel are somewhat less good in terms of the numbers of past papers being online (at least in my non-sciencey Edexcel experience), but they they still have some past papers for GCSE Science here www.edexcel.org.uk/quals/gcse/science/ .

  13. cat said,

    August 23, 2007 at 3:13 pm

    Bear in mind that these discussions have been going on for 20 years. I was one of the last groups to do O’levels(1985) and even then all three of my science teachers used to tell us how hard exams used to be and produce past O’level papers from ten or fifteen years prior that used to make us poo our pants. The bastards.

    However, I agree with anyone who says we had to be much cleverer then to get an A than these young whippersnappers do now with their piss-easy exams. Just as our teachers used to tell us that they’d had to be much cleverer to get their C than we would have to be to get our A.

  14. waster said,

    August 23, 2007 at 4:49 pm

    Long time reader, first time poster.

    Jamie Whyte wrote an excellent piece on this subject (Sunday Times?) following the A-Level results. His argument was that the universities are the organisations with an interest in the validity of the results. By analogy with credit rating, he concludes that instead of schools (debtors) buying (ever easier) exams from competing exam boards (credit rating agencies), universities (banks) should be paying.

    This seems magnificently simple, but it doesn’t quite fit for GCSE. And it doesn’t include leaving education at 18 as a goal of A Levels in themselves. Universities ultimately have an interest in quality of examinations at 16.

    PS Nice example of opposite effect: women’s clothes sizes in the US, I’m told, go into negative numbers, and yet…

  15. alsodug said,

    August 23, 2007 at 4:55 pm

    Just been trying this one:
    www.aqa.org.uk/qual/gcse/qp-ms/AQA-3451H-W-QP-JUN06.PDF
    and I thought it was actually alright, if you do well in this then you probably know a lot about some useful things.

    The calculation questions are too easy, didn’t they used to have it so the last few marks required some, unprompted, creative thinking? Could be the difference.

  16. manigen said,

    August 23, 2007 at 5:10 pm

    Ben said: “well let’s look at some. i’ve got some gcse physics ones from three years ago and they’re a bit of a joke, frankly.”

    Don’t get me wrong, I’m not saying they aren’t. I’m just generally peeved at various opinions being expressed by people who haven’t bothered to check. Which is why the collection/comparison thing is a good idea, is what I was saying.

  17. RS said,

    August 23, 2007 at 5:18 pm

    alsodug – I seem to remember going to University having got nigh on 100% in most A Level Maths papers (a few years ago now) and having real trouble remembering lots of stuff (like how to solve various differential equations) that I had previously breezed through.

    So I think you may well get a Summer holiday amnesia effect when you encounter your students.

  18. RS said,

    August 23, 2007 at 5:59 pm

    I imagine research has been done on this topic, I found this for A Levels:

    66.102.1.104/scholar?hl=en&lr=&q=cache:Z5zLNw-5SacJ:www.ucles-red.cam.ac.uk/conferencepapers/BERA1997JBTBNR.pdf+related:Z5zLNw-5SacJ:scholar.google.com/

    An interesting question is how much easier the A Level examination became under the norm-referencing system used from the ’50s to the ’80s. As participation rates increased we would assume that more of the less able took the exam – yet pass rates and grade proportions remained roughly the same.

    I’m sure there are some other arguments we could make about how demographic changes (changing % middle class kids),
    and proportion of kids hived off into less difficult (e.g. GNVQ?) exams could impact on pass rates – anyone know of any modelling done?

  19. profnick said,

    August 23, 2007 at 5:59 pm

    Ben,
    For O-level papers how about:
    www.mel-lambert.com/Ruskin/Images/JRGS11E_KB_Images.htm

  20. RS said,

    August 23, 2007 at 6:49 pm

    alex1 – that tells you nothing without knowledge of pass marks and how the syllabus has changed.

  21. Quixotematic said,

    August 23, 2007 at 7:04 pm

    It began quite a while ago. Back in 1983, the comprehensive I was at tried to schedule O-Level biology and physics in the same time slot, as they could not imagine anyone wanting to do both.

    Later, in about 1988, I remember my chemistry lecturer at tertiary college telling us that the nitration of benzene was to be cut from the syllabus since it was bringing grades down.

    (saddles hobby horse and mounts up)
    I think that the only fair solution would be to use ‘double-barrelled’ exam grades, where one figure is their mark relative to the marking scheme and the other figure is their mark relative to their cohort.

    It will never catch on, because it would bereave the politicos of levers to distort the reality.

  22. RS said,

    August 23, 2007 at 7:21 pm

    Haven’t A Levels been explicitly made easier? The new A2/AS system was designed with a 4-subject AS first year, and a 3-subject A2 second year; thus it has been made easier to fit in that extra AS – as previously 3 A-Levels was the norm.

  23. RS said,

    August 23, 2007 at 7:43 pm

    While I’m talking to myself, I’d also point out that for GCSEs dropping compulsory languages has also had an inevitable effect in allowing people crap at languages to do something else.

    I was amused to see that the media continued its ‘shock, horror, X% of students still fail to meet the minimum standard of 5 A*-C grades at GCSE’ whilst simultanously complaining about grade increases – which clearly cannot have any connection with A*-C pass rates.

  24. alex1 said,

    August 23, 2007 at 7:49 pm

    RS – I can’t speak from a position of knowledge here, but I just can’t see the grade boundaries going up, can you? Blatant politicization of the education system would appear to be culpable here. Correct me if I’m wrong, but is it not the case that GCSE grade boundaries are calculated after the papers are marked, using a basic distribution curve? I think I read this somewhere. I know GCE A levels conform to a uniform mark scheme, but GCSEs certainly don’t do that.
    If this is true, it surely shows all grading except pure percentages is totally arbitrary.

  25. RS said,

    August 23, 2007 at 8:05 pm

    less than 40% is a pass/fail system – which is an utterly meaningless use of percentages.

  26. jessina said,

    August 23, 2007 at 8:16 pm

    I did GCSE triple science in 2005 and it was bloody hard. I got three A*s in it – but the exam papers were very difficult. I remember having to learn the colours that different chemicals go when they react the various isomers (? forgotten the term) of bromine, chlorine and some other chemicals in order to identify the reactant. And having to do long and confusing calculations with moles of gas and volumes. I have forgotten almost all of it now but at the time it was not easy!

  27. nd said,

    August 23, 2007 at 10:01 pm

    So far, no-one has mentioned CSEs! Remember, GCSEs have to cover an ability range previously covered by two separate qualifications. It might be fair to compare a GCSE Higher Tier paper with an O-level paper, but it’s not clear to me without further information that their target audiences are the same.

    Also, don’t forget coursework! It’s horribly open to abuse, but at its best it can be genuinely impressive. Certainly a lot of effort goes into it, which means that there is less teaching time available for the written papers.

    I did O-levels myself; I currently teach Physics at secondary level. Off the top of my head, I’d say that the papers are easier now. However, getting an A* in a Science GCSE is genuinely difficult. The grade boundaries are high. Which is/was harder – an “A” at O-level, or an A* at GCSE? This question can’t be answered merely by looking at the papers.

    A further point: mere “difficulty” is surely not an end in itself. If the purpose of GCSEs is to produce scientists, then GCSEs (or O-levels) are only one step in the process. It might be the case that making GCSEs easier is a good thing, if it encourages more people to go on to study science, so long as degrees aren’t being dumbed down to accommodate this. Or it might be a bad thing, if intelligent people are turned off science because it isn’t challenging enough. Once again, looking at the papers doesn’t answer the interesting question. More work is needed!

    Currently, it is compulsory for every student to be taught at least some science to GCSE. This clearly puts some constraints on what the examinations can be like. So far as I can remember, science wasn’t compulsory for everyone in the old O-level/CSE days. Bear this in mind as well before getting too nostalgic.

    Some of my best GCSE students go on to do A-levels, and some then go on to do science degrees or medicine. Such students tend to get A*s at GCSE. Nearly all of them have to work hard to achieve this grade. Unless the students I am teaching are rather dim, it seems that GCSEs can still stretch the best.

    Of course, this year and next the science GCSE syllabus is changing completely. How well the new syllabus will work is still an open question.

  28. alex1 said,

    August 23, 2007 at 10:18 pm

    There’s a real cracker I remember from Summer 2005 physics which is the social aspects of security cameras, a question
    along the lines of:

    “Discuss the benefits and disadvantages of CCTV (closed-circuit television cameras)”

    The mark scheme indicated points such as “reduces vandalism” and “only displaces crime” were worthy answers. I’m only paraphrasing since I can’t remember the exact details, but it was roughly so.

    Whatever it may be, it’s not science.

  29. JQH said,

    August 23, 2007 at 10:29 pm

    I haven’t forgotten the old CSE.

    We’ve got a CSE physics text book from 1974 at work. It has far more diagrams and (horror) equations in it than modern A level text books.

    The rot started IMO when “General Science” was introduced to cover up the shortage of physics and chemistry teachers. There seems to have ben a cycle of dumbing down ever since. The current GCSE papers are a joke. CSE’s from 30 years ago were far harder.

    Pleaase run with this project Ben. We can but hope it’ll make a difference.

  30. Ben Goldacre said,

    August 23, 2007 at 10:35 pm

    ok, i think the trick is to collect together resources so they can be seen side by side.

    jqh, are you sure about that book? can you fax choice pages that i can post? whole papers? send it?

  31. Ben Goldacre said,

    August 23, 2007 at 10:55 pm

    i realise i’m being lazy/busy but the motherlode is to find a bunch of old o level and cse papers online, is there such a thing? ig not then posting JQH’s and others will be a serious resource.

    please, send me old o level papers, maths and science.

  32. marcdraco said,

    August 23, 2007 at 11:01 pm

    While it’s depressing me, I might as well add that the way things are in science these days, it’s little wonder that “scientfic” creationists are making such an effective mockery of basic biology (not to mention related sciences). Without an effective science base, we might as well descend back into the dark ages.

    I was heartened to read this today:

    news.bbc.co.uk/1/hi/health/6960612.stm

    Another “inexplicable phenomenon” linked to unreason bites the dust. Pity that the telly is full to brimming of cold readers and other assorted charletans rolled out routinely as if they actually have some supernatural ability.

  33. Neil Desperandum said,

    August 23, 2007 at 11:18 pm

    Lots of issues here…

    Exam papers for the last few years can be downloaded from exam board websites
    www.aqa.org.uk/
    www.edexcel.org.uk/home/
    www.ocr.org.uk/
    www.wjec.co.uk/
    www.sqa.org.uk/

    For older ones you might try local schools.

    Science has been a compulsory subject at GCSE in England and Wales (along with maths and English) since the National Curriculum was introduced in 1989(?). Double award science has just ended after some 15 years. It was the three separate sciences (usually taught separately) with enough stuff to fill 2 GCSEs, as opposed to 3 (which was also available). Double science was by far the most popular GCSE science option and was a perfectly good basis for science A levels (at least with a good grade).

    The new science GCSEs starting this year are more consumer-oriented. They are modular and contain less science content and more “how science works”. They are probably a less good basis for science A levels.

    In Scotland the 3 science subjects are completely separate and they are not compulsory at standard grade.

    Exam grades are not norm-referenced, but criterion-referenced. The examiners know how many marks are needed to get an A grade, without reference to how many people are achieving those marks. This is how the % of A* grades can go up each year.

    Since exam performance is increasing each year that must mean that the exams are getting easier and/or the teaching is getting better and/or the pupils are getting cleverer.

    Are exams on average getting easier? That’s a very hard question. They’re certainly different from exams 20 years ago. For one thing a far bigger cohort are sitting them so there has to be a range of questions and some will be very easy (to separate the Es from the Fs). Wider accessibility also means that the appearance of the exam papers has changed. They look easier, with lots of pictures and spaces for answers. There has been a swing from knowledge to understanding, so questions may be set in an easy context. An apparently hard question about the loop of Henle may actually be simple factual recall, while an apparently easy question about Mars bars may require some high level analytical skills.

    Are teachers on average getting better? Yes, beyond any doubt. Teachers today are far more reflective and responsive professionals than they were 20 years ago and this is likely to be reflected in increased exam performance. That’s not to say that there aren’t some rubbish teachers today and great teachers in the past.

    Are pupils on average getting brighter? No idea (not even sure what the question means), but it seems unlikely.

    My personal guess as an experienced teacher and head of department is that improved exam performance is a mixture of better teaching and easier exams. Today’s science exams are trying to be all things to all people and inevitably failing. Perhaps a return to a 2-tier system of a content-centred route for future professional scientists and a consumer-centred route for a science-literate population.

    As for comparing old and new exams, you’ll have to think long and hard about what you’re going to measure and what questions you’re addressing. I’m not sure anything useful will come of it.

  34. steveh said,

    August 23, 2007 at 11:20 pm

    Anecdotal evidence for sure but 14 years ago I did a maths degree and then ended up tutoring the first year undergrads 6 years later as a postgrad. By the time I left, the mathematical background of the new intake was so poor the entire first year of the degree course had to be restructured. Still amazes me now that we were getting 18 year olds with A level maths who had never come across a vector.

    As steevl suggested, teenagers should be more appalled at the state of their education. I’m also equally amazed that for A level physics back then, I studied O level physics exam papers from a decade earlier … and they were harder than my exams.

  35. Neil Desperandum said,

    August 23, 2007 at 11:32 pm

    Isn’t that a bit woo?

    What about validity and reliability?

  36. Osman said,

    August 23, 2007 at 11:46 pm

    I just did my chemistry A level a couple of months ago and I found it quite tough. I found the past papers I did from about 2000-2006 were all of about the same difficulty.

  37. NickConnolly said,

    August 23, 2007 at 11:55 pm

    Your plan is a bit confused. Whether individual questions are harder or easier doesn’t tell you whether the tests are ‘easier’ or not. Nor is that even the right question to be asking.
    The key issue is not so much ease as *discrimination*. Discrimination is that quality in a test that means people with different amounts of the trait in question get different scores. Two tests one with a mean of (say) 70% and one with a mean of (say) 40% could, if they differed in other regards, be equally good at splitting up the test group.

    A better peice of data to get your hands on would be the scores needed to get particular O-Level grades in 1980, equivalent GCSE grades in 1990 and then again in 2000. You’d also need individual item statistics on each of those papers. Once you had that you could do some meaningful comparison. Anything less really is just Daily Mail style woo were we hunt down things to bolster our prejudgement.

  38. Ben Goldacre said,

    August 24, 2007 at 12:01 am

    sigh. there is nothing confused about my plan. as i say at the top, i want past papers and marking schemes. youre clearly upset that i’m not, in a casual blog post, setting out a clear methodology for a published quantitative paper. please write such a methodology and do it, we’d all be delighted to read it, i think you’ll find the issues are even more complex than you describe.

    personally right now i’m interested in sticking some papers side by side and seeing what we think from there. i think that will be interesting. if you don’t find that an interesting prospect then i apologise profusely for your disappointment and wasted brain energy.

  39. marcdraco said,

    August 24, 2007 at 12:07 am

    Being a bear with little brain, I’m all for the side-by-side and see how they compare.

  40. NickConnolly said,

    August 24, 2007 at 12:56 am

    The confusion I’m refering to is what you would hope to achieve. As an exercise is looks like the gathering of rhetorical ammunition. If that isn’t what you are intending too (and I’ll happily take you at your word if you say that it isn’t) then the exercise is a bit pointless.

    As for being ‘clearly upset’, I’m afraid you’ve misread the tone of my message. I was trying to be helpful. You don’t need a fully fledged method but you needn’t set off down a blind alley at the start – thats all.

  41. amoebic vodka said,

    August 24, 2007 at 1:48 am

    When I took A level biology, the *syllabus* at least was 20 years out of date in places…

    No-one has brought up that there’s a difference between a course being hard work because of the quantity of course material and being hard work because of the difficulty of the content.

    This particularly irritates me, because exams could be getting less challenging while at the same time the people taking them could be having to work harder. There’s a general implication – elsewhere anyway, that suggesting that exams are less challenging implies that the people taking them didn’t need to do any work for their high grade and they’re therefore stupid and lazy.

  42. plasecaha said,

    August 24, 2007 at 3:43 am

    Looking at papers will be interesting, but we already know the answer, if it’s true that things like algebra are being gradually excised from the curriculum.

    Oh, but some teachers – and some who have undergone the new system – will instantly point out that nowadays, the exams test DIFFERENT things. Things like “research” (i.e. looking up the hard bit you used to have to figure out yourself on the web), and “teamwork” (i.e. getting the brighter person in your group to do the work to make sure he doesn’t suffer a poor grade for the lack of industry of others).

    Sorry, but something like understanding calculus is a lot harder than all this research and teamwork stuff. In PRINCIPLE, something like research itself can be tricky, but it’s not like they’re discussing the ins-and-outs of methodologies in these GCSEs. What passes for research in many cases in no way bares relation to the difficulty of something like calculus. Algebra is an abstraction, and abstractions are tricky stuff.

    So, let’s get rid of algebra, replace it with easier stuff, and pretend that easier stuff is just as hard Problem solved!! We see the same kind of approach when it comes to ditching the requirement to remember things. Sorry again, but the sad fact is that if we are testing capability, a good memory is going to be an asset, as is the ability to learn things QUICKLY, and the ability to do that often requires intelligence in terms of how you structure and go about learning the information. You also have to remember how best to USE the information, you have to decide WHAT’s worth remembering etc., etc….

    Often enough, the memory component is ESSENTIAL, say in the case of learning French. Sorry, but you’re not going to be good at French without having memorised a good vocabulary. With languages, there is no way around this… if you want to converse with someone, they are not going to wait while you look up every response on the web. The demise in the number taking languages is therefore hardly a surprise. Perhaps they should replace the French GCSE with a course where you don’t actually learn French, but about the IMPLICATIONS of being French, or learning French, even. Like they’ve done with science.

    BTW… sure, if INDIVIDUAL questions are more difficult, that may or may not determine whether a paper is harder overall. One could, for example, envisage two papers, both of which have ten questions, nine of which are of equal difficulty, with the tenth question in the first paper being a lot more difficult. If all questions are worth, say, ten percent, and the pass-mark in both cases is 70 percent, that tenth question may make little difference. Even those who couldn’t do it, would still have access to the top grade.

    If MORE of the questions are more difficult, however, then it’s a different story. So yeah, this IS a worthwhile exercise. Gauging comparative difficulty can be tricky, but I doubt it’s going to prove so in this case. Calculus is much harder than looking up some stuff on the web and cobbling something together on the back of it, and the same will apply to many of the difficult things that have been excised.

  43. plasecaha said,

    August 24, 2007 at 3:53 am

    Sorry, when I said this “If all questions are worth, say, ten percent, and the pass-mark in both cases is 70 percent”

    …I of course meant to say if you had to get 70 percent to get the top grade.

    Also, they are supposedly ditching “coursework” now because, shock-horror, it’s open to cheating by staff and students. Who’da thunk. What are they going to replace it with, is the question, I suppose. Will the questions get more obviously easier?

  44. NickConnolly said,

    August 24, 2007 at 4:28 am

    plasecaha,
    Sorry one more note.
    You say “Calculus is much harder than looking up some stuff on the web and cobbling something together on the back of it, and the same will apply to many of the difficult things that have been excised” – you have a point. Note though that you can already settle that question by looking at the syllabus. Your point about things execised is one about whether the students are studying a different (and easier) subject. That is to some extent a subjective judgement but not one we shouldn’t make. However the exam papers tell you nothing you wouldn’t already know from the content of the syllabus.

  45. plasecaha said,

    August 24, 2007 at 5:30 am

    “The point about the futility of looking at two papers *without* knowing what grades are allocated for which marks is best exemplified by thinking about two *identical* papers with different grade boundaries. You could stare at the papers all day and not discern which of the two was easier because the information you needed simply isn’t in the content of the questions.
    Now imagine two papers A and B. Imagine every question in paper A had a similar, but tougher, version in paper B. Now a side by side look at A and B may result in you declaring B to be ‘harder’ – but that decleration is still meaningless if you get the same grade for fewer marks on B.”

    You are entirely correct, but isn’t that rather obvious? Obviously, if the marking schemes or grade boundaries differ, one might get differing grades on two papers, even if one performs the same in both cases. It’s worth saying I suppose, but I think many are aware that in PRACTICE, that is unlikely to be a concern. Has anyone ever gotten an A at “O” level with a score of just 40 percent? Generally speaking, the grade boundaries etc. have remained ROUGHLY consistent. I would definitely imagine there is some variation (with the trend being towards making things easier over time), but the effect is going to be DWARFED by the difference in the difficulty of the questions, if they are doing things like dropping algebra and calculus.

    If there wasn’t such a stark difference in the difficulty of the questions, then your concern would be of much greater impact, but then no one would be that concerned in the first place. If the exams remained BROADLY the same in terms of difficulty, there’d be less need to debate.

    We do have to keep it in mind, but I think your concern applies to a much greater extent in Higher Education, where it really IS possible for a good student to perform pretty good in one institution and come out with a 2.1, while a poorer student elsewhere gets a first. Clearly, when it comes to HE, grade boundaries and what you get for your 40 percent differ WILDLY.

  46. NickConnolly said,

    August 24, 2007 at 6:10 am

    Plasecha,
    Yes it is rather obvious which is rather the point and why I’m surprised Ben would even bother. The thing is the grade boundaries *are* going to change. Firstly with the old norm-referenced O-Levels that process was effectively built in. With the change to GCSEs the manner in which the grade boundaries were determined changed completely. So the issue isn’t a trivial one or just some subsidiary noise in the data. A difficult O-level question which next to nobody ever got right and which played no statistically significant role in determining who got a grade A is essentially irrelevant in determing whether the papers are easier or harder.

  47. plasecaha said,

    August 24, 2007 at 6:22 am

    I already dealt with the point about a single difficult question. Agreed, that may make no difference. As I pointed out before if a goodly number of the questions are more difficult, rather than just a couple, then it will have an impact. All we need to do is look at whole papers, rather than isolated questions.

    (Someone else has already made this point when talking about the impact of having a few easy questions in a paper to help differentiate between an E and an F, and how we shouldn’t just focus on those isolated questions. I think we’re all aware of this point).

    Agreed, grade boundaries may change. Whether an A requires seventy percent or seventy five percent however will be dwarfed by the impact of doing something like ditching calculus.

  48. plasecaha said,

    August 24, 2007 at 6:58 am

    “You are right, the tests also contain information about the syllabus. My point was that the issue you raised was primarily a syllabus question rather than a testing question: a case of a course with easier content.”

    Eh? We’re talking about the exam papers. My contention is that a paper without much algebra in it is going to be a lot easier than one with stuff like calculus in it, if the algebra has been replaced by questions like those CCTV ones above.

    You made the point we could determine such things from the syllabus, I made the point that wasn’t necessarily so, because the exams don’t always relate that well to the syllabus, and maybe if we want to test ability at the top end, it’s possibly a good thing if they go beyond the syllabus. I’m focusing on the tests, here. How is the syllabus relevant?

  49. plasecaha said,

    August 24, 2007 at 6:59 am

    “That in itself is a worthy issue for discussion but it brings in other issues. For example A-level maths has always had a reputation as being hard. A similar issue occurs for the IB Diploma Higher course in Mathematics. In either case a student who doesn’t need the specific content would be poorly advised from a pragmatic p.o.v. to take maths at 16-19 simply because they will have a greater chance of failing and will (in most cases) ended up with a lower grade. That creates a dilemma: dumb maths down or have a ‘gold-standard’ course which nobody takes.”

    Well, in principle, it’s easy to solve. You still teach maths to everyone, because everyone needs maths, but you stretch the people who are better at it in the teaching, and award them higher grades when they do better in the tests. A bit like what used to happen.

    Nowadays, it’s not enough to teach everyone maths, now everyone has to get better and better grades. For this to happen, either the teaching has to improve, or the kids suddenly become brighter, or we dumb down the exams. Since we’re doing things like excising algebra, guess what’s really happening?

    When it comes to maths post-16, it’s obviously not a bad thing if we get more people doing it. And to get more people doing it, it may be necessary to dumb it down for some. It’s not the dumbing down per se, that’s necessarly the issue. It’s the dumbing down while pretending in fact results are genuinely improving.

    And of course, the dumbing down can mask poor teaching. To be fair to secondary teachers trying to get people their GCSEs, it’s a pretty thankless task given the levels of literacy and numeracy they frequently inherit from the primary sector. And we can of course point out how that in itself isn’t by any means entirely the fault of those teachers. There is, to some considerable extent, a failure in the system – and outside it – which dumbing down tries to hide.

  50. Gimpy said,

    August 24, 2007 at 7:34 am

    It would be interesting to see how the English system compares with the Scottish system by comparing science and maths papers from equivalent levels. The Scottish system has one less year at school and one more year at University. Is this reflected in a lower level of knowledge at school in the Scottish system?

  51. NickConnolly said,

    August 24, 2007 at 7:36 am

    My point was that you are raising questions primarily about the syllabus. Your counter-point doesn’t change that; you are just describing how you would use the test papers to discover syllabus content.

  52. Cassander said,

    August 24, 2007 at 9:52 am

    This is a rather depressing exercise. You can’t just take examinations out of their proper context and try to make comparisons, as it just doesn’t work. For anyone who knows anything about education it’s like watching Gillian McKeith rabbit on about chlorophyll…

    Anyway, if you really were to measure whether examinations were getting easier, you’d have to take into account all kinds of issues, such as the switch from referencing by norm to referencing by criteria, and the move from differentiation by task to differentiation by outcome.

    And one little last point, quoting from post … something or other (I can’t read the post numbers on my computer):

    “Are pupils on average getting brighter? No idea (not even sure what the question means), but it seems unlikely.”

    Well, it seems that they may well be. A large scale study of IQs in Scotland during the 1950s and 1960s found that this measure of intelligence was increasing over time. No-one was quite sure why and both hereditarian and non-hereditarian arguments were advanced in an effort to explain this.

    Of course, whether this is relevant, and whether it is fair to equate IQ with intelligence are other matters, but I still think that it’s a point worth raising.

  53. plasecaha said,

    August 24, 2007 at 10:09 am

    “Anyway, if you really were to measure whether examinations were getting easier, you’d have to take into account all kinds of issues, such as the switch from referencing by norm to referencing by criteria, and the move from differentiation by task to differentiation by outcome.”

    We’ve just been through all that. Everyone is aware of those issues. The fact such issues exist does not automatically invalidate the exercise of comparing papers from different eras. Because…

    1) As Ben pointed out, it may reveal interesting stuff, even if one can’t make a definitive statement about precisely how much things may or may not have gotten easier

    2) Such effects may be DWARFED by the drop in difficulty of the questions.

    If the difference in difficulty isn’t all that great, then issues such as fluidity in grade boundaries or the adoption of criterion-based referencing loom larger.

    But if that’s the case, there’s less of a debate to be had. We will have established that, whether or not standards have fallen, or even risen, in terms of the papers, it’s not significant compared to other effects.

    If, on the other hand, there IS a much greater disparity in difficulty, then the papers are easier now DESPITE distorting effects such as grade boundaries, and there’s a real problem.

    BTW, I wouldn’t be surprised if IQs have risen a bit… might be diet-related, the same way people tend to be taller on average now…

    But that doesn’t mean exams haven’t been dumbed down…

  54. Dr Aust said,

    August 24, 2007 at 10:27 am

    Not science, of course, but I thought the comments by the language teacher / examiner in the Telegraph (link on the Miniblog) were quite enlightening.

    What they did was explain exactly how the exam could be dumbed down without exactly making the questions easier. It took a kind of “connivance” from both directions – parts of the exam were re-jigged to make the questions much more predictable, and then teachers “taught to the test” (e.g. by teaching kids a “framework story” that they could use to answer the questions but could just customise e.g. a holiday story with “customising” by adding where exactly they had holidayed.

    I remember a friend of mine who teaches EFL (English as a Foreign Lang) telling me once that this was how he got all his pupils through the TOEFL tests (widely used by Univs for foreign students to assess English proficiency) – give them a well-rehearsed outline and they just have to add bells and whistles (names, places).

    Not sure what the science analogy is, but as a long-standing exam cynic I would suspect there is one.

    Would be interested to see if grades went down again if all course work were to be abolished.

    Standard view of all Univ science lecturers I know about coursework:

    – a good way of teaching people how to do useful stuff (though labour intensive) …BUT…

    – bugger all use for grading / assessing / ranking people.

  55. plasecaha said,

    August 24, 2007 at 10:43 am

    Here’s some more in the same vein, Dr Aust…

    news.bbc.co.uk/1/hi/education/6918805.stm

    news.bbc.co.uk/1/hi/education/6653533.stm

    Of course, by the time they get to HE it’s even easier to cheat, when the lecturers set and mark the exams as well…

  56. megachicken(b) said,

    August 24, 2007 at 11:13 am

    Now that would be an interesting test – get some averagely bright kids, and ask them to sit GCSEs for subjects they’ve never been taught. If you can get a good pass just through general knowledge and educated guess-work, it can hardly be a decent test of subject specific learning.

  57. misterjohn said,

    August 24, 2007 at 12:26 pm

    Thank you Richas for injecting some sense into this discourse.
    I had wondered whether Ben had been kidnapped by aliens from the Daily Mail when I read hi original comments.

  58. David Mingay said,

    August 24, 2007 at 12:44 pm

    You think the current A Levels are bad – wait till you see next year’s new lot.

    Psychology (and this is now “officially” a science – what it’s been for the last 133 years, I don’t know) will have no coursework. A science, with no assessed lab work. These aren’t being replaced by extra written exams (and will make boys artificially “better”, I expect), so that’s 1/6th easier for a start.

    At the moment, students have to, for example, know about schizophrenia and depression and anxiety disorders, in order to answer the one question which could be on any of them. As of next year, they have to know schizophrenia or depression or anxiety disorders, and there will be a question on each, from which they have to answer one. So that’s another 2/3rds easier.

    The need to know anything about parametric statistical tests was removed a few years ago, at the same time as the number of assessed experimental reports was reduced from four to one.

    Re #79, I don’t think today’s youth are ignorant, callow and lazy. I think they’re being treated as if they were though.

  59. nekomatic said,

    August 24, 2007 at 1:05 pm

    Well I had some anecdotal evidence to offer about how chemistry A-level hasn’t significantly dumbed down in the last 18 years but I’ll keep quiet now ;-)

    >I remember my chemistry lecturer at tertiary college telling us that the nitration of benzene was to be cut from the syllabus since it was bringing grades down.

    Use of benzene was already deprecated on safety grounds when I was doing A-level in the late 80’s (though not banned as I did get to demo brominating it when I did a talk to the science society!). Are you sure it wasn’t bringing healthy student numbers down?

    >I understand that in GCE arts exams, ABRSM music exams, etc. that percantages are added after the grade is decided upon.

    In ABRSM grade exams the number of marks needed for pass, merit and distinction is fixed. I don’t know if there’s a moderation process for the marks actually awarded.

  60. Mark Frank said,

    August 24, 2007 at 1:13 pm

    At the beginning of this Ben wrote:

    “It occurs to me that this is a very simple and familiar problem:”

    I wonder if, 80 responses later, he still believes that?

  61. richas said,

    August 24, 2007 at 1:38 pm

    Instead of a random collection of old papers why not look at the QCA’s published research on the 1996-2001 exams?

    The next 5 year batch must be due soon too.

    www.qca.org.uk/libraryAssets/media/6301_compare_nat_tests_96-01.pdf

  62. David Mingay said,

    August 24, 2007 at 1:50 pm

    Ben – that certainly meets the criterion of parsimony.

  63. Ben Goldacre said,

    August 24, 2007 at 1:55 pm

    thanks.

    i am not offering to re-do this:

    66.102.1.104/scholar?hl=en&lr=&q=cache:Z5zLNw-5SacJ:www.ucles-red.cam.ac.uk/conferencepapers/BERA1997JBTBNR.pdf+related:Z5zLNw-5SacJ:scholar.google.com/

    ten years later and for GCSE exams, but it would be interesting if someone did, given how much guff is written on the subject.

  64. pv said,

    August 24, 2007 at 2:47 pm

    Ben Goldacre said,

    August 24, 2007 at 1:43 pm

    “i have to say i am amazed that people are so terrified and outraged by my suggestion that i simply plonk some old O level papers next to some new GCSE ones.”

    I think some people are terrified, and with good reason. In an education system where the most important aspects are league tables and consumer choice (i.e. the numbers of students going on to university and universities creating courses according to public demand), it’s pretty clear to an outsider that the primary purpose of the UK education system is numbers and statistics. Academic attainment and intellectual rigour would appear to come slightly further down the list of educational priorities.

    I’m damn sure many students work hard to get their results, as do many teachers. But that’s no reason to to refrain from investigation or criticism – just because someone might be upset to discover that their results turn out to be slightly less grand than they thought.
    People can study homeopathy at university these days, and I’m sure they work hard too for their “qualifications”. It’s still shite though!
    And the comparison with homeopathy doesn’t end there. I feel there are those who think it is not only unfair to try and compare examination papers from different eras, but that it is impossible.
    To paraphrase Bill Shakespeare and Ambrose Bierce:
    A skunk by any other name would still stink.

  65. thekumquat said,

    August 24, 2007 at 2:52 pm

    Richas makes some excellent points about the changes in purpose of exams over the years.

    I did GCSEs in their second year (1989), so most of our revision was done using O-level papers. At first glance, the O-level questions looked much harder – curt statements followed by ‘Why?’ or ‘Discuss’, rather than indicating what types of answer would be valued. However, once our teachers showed us a few sample answers, it became clear that if your teacher told you what facts you needed to know, and you had a good memory, these questions were very easy. In contrast, GCSE introduced the concept of the real world – most obviously in physics where instead of simply answering questions on pressures, that gases can be compressed, etc, you had to explain how a fridge worked.

    People criticise a lot of these questions for having obvious answers, but just as common sense isn’t very common, they aren’t obvious to lots of 16-year-olds (hence teaching).

    At A-level, in particular for chemistry where our teachers were men on a mission, we got to do every single paper (A level, S level, Oxford Entrance…) from 1991 back to 1967 when our teacher did his. Again, if you had a good memory, it was a doddle if you were exposed to the material. Ozonolysis was not on the syllabus at any point, but for some of the papers you were ‘expected’ to deduce how it works. In reality, teachers in the know taught it to their students. Teachers who weren’t in cahoots with the examiners didn’t.

    I think I have all said chemistry papers around somewhere.

    Yes, calculus is no longer done at GCSE whereas it was introduced at O-level. On the other hand, O-level didn’t mention matrices or vectors, nor have coursework with problems such as calculating bellringing changes or number of regions created by diagonals in regular polygons, where a logical approach was needed. It was nothing but a memory test. In some subjects like languages memorisation is useful, but in others, why test ability to memorise formulae when anyone can look them up (even pre-Internet)? The useful test is the application of the right formula at the right time.

    A somewhat separate point – the standards of marking are becoming more variable, as with more papers some exam boards are recruiting almost anyone. A lab tech in my lab did lots of GCSE marking for sciences, but had poor English (not a native speaker), and marked lots of answers wrong because of not understanding a perfectly good sentence, until I explained the words used.

  66. CrackMouse said,

    August 24, 2007 at 2:59 pm

    @ #70 Gimpy:
    Yeah the scottish system is a wee bit different (or was when I left school in ’96). You have the standard grades (=GCSE) and Revised Highers instead (= slightly lower than A level). The revised Highers were below the standard of A levels. You can opt spend two years in scottish schools doing highers if you like. You can sit up to 6 highers per year, but normally 5 (up to 12 different subjects over the two years). If you’re good, you can take individual subjects further in your second senior year by moving onto Sixth Year Studies (SYS), which was equivalent to A level. If you had decent SYS grades in your chosen university course subjects, you could opt to go directly into 2nd year uni, skipping 1st year altogether. You generally find, however, that Scottish uni’s use the first of the 4 years largely as A level “catch-up”.

    On the point of exams getting easier…. My Higher exams (in 1995) were far easier than the older (non-revised) Higher past papers I used for study purposes. For example, the (non-revised) Higher physics paper from a few years before I did mine DIDN’T supply you with all the various equations you needed to answer the questions (which the Revised Higher did).
    I know through trying them myself that the older papers were harder than the ones I was examined on. Needless to say, though, the exams I sat were a comparative breeze, and I liked that at the time…

  67. marcdraco said,

    August 24, 2007 at 3:15 pm

    Surely as intelligent (many quite learned) people on here, we shouldn’t have too much trouble working out which exam paper is easier – if, indeed, one or other is.

    Sure, anecdotal observation is personal and unscientific, but it’s a start so what’s the problem? Initial observations of phenomena are what leads us to more detailed examination.

    It’s no use discussing if exams are easier if we don’t at least START looking.

  68. richas said,

    August 24, 2007 at 4:36 pm

    Plasecha – what makes you think that the exam boards and QCA have changed the course work because of “cheating”? Oh yeah the Daily Mail said so. It is true that some concerns about plagiarism or parents doing them were raised but this was not the main reason for the change.

    What happened was that they found that the course work was more socially, gender and economically biased than the exams. They looked into it and found that girls, the better off and those with educated parents who value education spent a lot more time on it, reworking it to make it better. Given there are only 190 days when they are in school they also found that schools were allocating more time to coursework rather than teaching cutting the teaching time. The new “supervised” coursework is more about applying a time limit than preventing “cheating”.

    The curriculum has changed somewhat but they are not learning something different. The difference is the purpose of the exam – it examines what standard an entrant has achieved not whether they are in the top X% of their cohort. About 3% of kids get 3 As at A level today. Now some see this as a terrible decline as A grade A levels are too common – how so? How can they be so easy when only 3% get 3 As?

    As for the system – the regulators enforce the procedures on the exam boards. They are inspected and checked. You claim they face competitive pressures to dumb down but how so? They need to ensure the quality of their exam is maintained. There is no evidence of an easier board or of boards making their exams easier. Those that set the grade boundaries include independent professionals as well as exam board employees who stake their reputation each time they sign off an exam.

    Finally your point about fairness of retakes/modules because they have more time to reach the standard. What does this matter at all in a standard based exam. They either have achieved it or not. It is like the driving test – have you met the standard or not. If you take it 20 times it does not make your driving license less valid.

  69. jcm07 said,

    August 24, 2007 at 7:17 pm

    @70 Gimpy
    Not quite true … Scottish pupils do an extra year at primary (entering secondary when 12, rather than 11), then 6 years of secondary, so do the same schooling as English pupils overall, but also an extra year at University.

    @94 Crack Mouse
    The Highers system has been changed again (since I finished in 1999).
    Now it’s a Higher (revised for a 2nd time), which is still slightly below the A level, and an Advanced Higher (instead of the Sixth Year Studies), which is a slightly higher qualification than an A level.

    Most people will sit their Highers in fifth year, then retake in sixth year as necessary or go straight into First Year of University if they got all they needed first time round. As you say, the good students usually go onto the Advanced Higher and can potentially go straight into 2nd year of uni or go to an English uni to be finished quicker. Very few people do this though: most like to have a year of doing not very much in first year of uni!

    There’s much of the same debate going on in Scotland about easier exams. I know when the new system was first put in place the new Advanced Higher in particular was looked on as being inferior to the SYS, but whether the same’s true 5+ years down the line I’m not sure.

    The Scottish system does have the benefit of only having one set of exams set and one awarding body.

  70. plasecaha said,

    August 24, 2007 at 7:38 pm

    richas

    “Plasecha – what makes you think that the exam boards and QCA have changed the course work because of “cheating”? Oh yeah the Daily Mail said so. It is true that some concerns about plagiarism or parents doing them were raised but this was not the main reason for the change.”

    I didn’t say it was because of teachers cheating. Indeed I went on to point out that since teachers are still in control of the coursework monitoring, it wouldn’t stop that anyway. But the education department isn’t likely to annoy half-a-million teachers and lecturers any time soon and their unions, so even if they are concerned about teachers cheating, they aren’t liable to rush to invite such hostility.

    In any event, it’s in the government’s interest for grades to keep rising, or appear to, however it happens.

    “What happened was that they found that the course work was more socially, gender and economically biased than the exams. They looked into it and found that girls, the better off and those with educated parents who value education spent a lot more time on it, reworking it to make it better. Given there are only 190 days when they are in school they also found that schools were allocating more time to coursework rather than teaching cutting the teaching time. The new “supervised” coursework is more about applying a time limit than preventing “cheating”.”

    Yep, that’s another way the system was compromised by coursework: the potential for teacher assistance, AND parental assistance. So, they’ve taken steps to curtail the latter, possibly because it’s somewhat elitist, as you point out, and traditionally this party is somewhat against privilege. But the tests have been terribly compromised in the meantime, and remain so because of teacher involvement. You can’t take a rise in standards too seriously while such a state of affairs persists.

    “The curriculum has changed somewhat but they are not learning something different. The difference is the purpose of the exam – it examines what standard an entrant has achieved not whether they are in the top X% of their cohort. About 3% of kids get 3 As at A level today. Now some see this as a terrible decline as A grade A levels are too common – how so? How can they be so easy when only 3% get 3 As?”

    Yes, I agree that we have switched from norm to criterion-based referencing. This is largely irrelevant, in the sense that it is possible for overall standards to slip either way. Just stating that there has been a switch, over and over as some have, doesn’t mean there hasn’t been a drop in standards. It just complicates the process of determining the extent of any change in standards.

    As for learning something different, you can say that they aren’t learning anything different, but you’re going to have to establish that a bit more rather than just saying it. There have been marked changes in the last 20 years or so, and we’ve already discussed some of them. Traditionally difficult aspects have been diminished, and there has been much more emphasis on things like coursework, “research” and “teamwork”, for example.

    But of course, a better look at the papers can establish the nature and extent of this.

    “As for the system – the regulators enforce the procedures on the exam boards. They are inspected and checked. You claim they face competitive pressures to dumb down but how so? They need to ensure the quality of their exam is maintained. There is no evidence of an easier board or of boards making their exams easier. Those that set the grade boundaries include independent professionals as well as exam board employees who stake their reputation each time they sign off an exam.”

    You write as if you have an unwavering faith in the system. This was the same system that introduced coursework in the first place, allowing parents to help their kids. Anyone who has been through a few TQAs in HE and Ofsteds in the mainstream knows how much of a joke such regulation can be. The regulators have a vested interest in appearing to improve performance. The system is fundamentally flawed – just about everyone involved has a stake in having results consistently appear to rise. ACTUALLY making them rise, however, is very difficult in practice, especially given the levels of literacy and numeracy coming up from primary.

    I KNOW schools consider how difficult the exams are because I have seen it – indeed it works the other way sometimes too. One school I attended, deliberately picked the O & C board because their papers were commonly regarded as the most difficult at the time, the most prestigious. Many public schools eschew normal GSCEs in favour of iGCSEs because the latter are more challenging. Ironically, in our “flawless” system the government won’t give them recognition for these exams so very good schools drop down the tables. It’s hardly a stretch of the imagination that schools who struggle may pick easier papers. No turkey is liable to vote for Xmas by admitting to such practices. But there’s plenty opportunity to do it, and pressure to do so. Without proof tt hasn’t taken place, the assessments are fatally flawed.

    (Not too many teachers who cheat are going to rush to admit it either, but occasionally evidence leaks out, if you check my links above).

    “Finally your point about fairness of retakes/modules because they have more time to reach the standard. What does this matter at all in a standard based exam. They either have achieved it or not. It is like the driving test – have you met the standard or not. If you take it 20 times it does not make your driving license less valid.”

    I agreed with you that, on balance, your take on how this part of the system should be is a good one. I am just pointing out that there is a difference between someone who masters something quickly, or more slowly. If institutions or employers want to pick students who have more potential/learn more quickly, then they may prefer those who picked stuff up in less time. Quite often, it’s apparent exams were retook, so they can make an enlightened choice, but there can be an economic/privilege effect at work. I knew some people who failed their Oxbridge exams the first time around, and got sent to a crammer for another year to retake by supportive and more wealthy parents. It’s not a flawless approach, but on balance I think it’s preferred.

  71. Robert Carnegie said,

    August 24, 2007 at 8:04 pm

    #78, David Mingay: the indelicate question is, how many more years have you spent in the University of Life, getting an extended general education? You can pick up a lot of biology from watching the likes of David Attenborough, or what would be the point of having him?

  72. richas said,

    August 24, 2007 at 8:17 pm

    plasecaha – you have accused me of just aserting that standards have been maintained. My evidence was the 249page PDF from the QCA that checked every paper in every subject for 5 years. You however are asserting that a conspiracy between the government, the independent regulator, the independent exam boards and leading educatonalists exist without any real evidence at all. Indeed the same phenomenom you believe so odd in the UK (it happens in Scotland too) is a global one.

    Assessment is a profession, it requires expertise and experience. It uses sophisticated sampling and statistical techniques and these are independently reviewed. It does not lend itself to a conspiracy.

    The iGCSE is nor more challenging. It is more similar in format and style to the A level. The independent schools who choose it are as guilty as anyone of teaching to the test, it is just they are judged on the A levels and they want their kids used to that style. Fair enough but not more challenging and not an indictment of the broader GCSE which also asseses lower ability pupils.

    I am confident that the nesxt PISA intenational test will show the UK as doing well – the last one we had enough entrants for validity showed us in a good light, the latest one we had too few entrants for statistical validity ut the core data published in the appendix was again positive (we only missed validity by a small sample).

    we seem to have a philosophical difference on the role of assessment. You want a time pressured exam, without retakes. Essentially a test of short term retention for the purose of selecting the few for work or university. I want a system that allows for extended work projects over a period that asseses that a curriculum has been covered and used to a given standard. Coursework assessment has some clear advantages over a solely exam system, hey they even use it at Uni.

    Frankly I don’t care if Oxbridge now struggles to sort the 3% with 3 As – they are bright enough to do something like interview them or set them a group exerise/problem/seminar style interview if they need to – the whole A level and hence the whole education system should not be skewed for these two colleges and the 3%. Besides the 3% have reahed a standard that Oxbridge should be able to build upon for any of them.

  73. DTM said,

    August 24, 2007 at 8:38 pm

    I have points to make:

    1. Surely the question is, does GCSE or A-Level better prepare students for whatever they do next now, or did it in the past?

    The answer that I would propose is no, it is a far inferior preparation these days. I base this on the appalling numeracy and literacy skills of students entering university (based on my own observations). Back when I did my engineering degree the entire first year maths syllabus was basically everyone should have known.

    Whatever the grades boundaries are, surely this is THE fundamental question we should be asking.

    2. The school examinations should put more emphasis (than at present) on the best students, but this is discouraged by the structure of league tables and grade boundaries. I don’t want to come across as being elitist, but it is these people who are going to make Britain Great in 20 years time. We need to invest in them better now in order to reap the rewards in the future. This involves having a set of exams which differentiate among these students such that there is incentive (for schools and students) to be among the best. This seems to be completely lacking in the current system.

  74. thescientist said,

    August 24, 2007 at 8:54 pm

    I got 2 A* for dual award science which i think is a waste of time. For me it wasn’t really a challange and since i am doing science maths and further maths for A-level doesn’t really help but at my (private) school there is no option.

    In preparation for chemistry because we had finished the syllabus a term early we did a 20ish year old o-level paper as p[ractice. you actually had to think rather than just “learn the facts” and jump through hoops.

    In many subjects it is virtually expected to get full marks in coursework. in dual award, for a top set, anything less isn’t really good enough. so already you have 25% of your marks with not much effort.

  75. marcdraco said,

    August 24, 2007 at 9:31 pm

    @DTM – at least you can *spell* literacy! How many of these smart kids come out with A+ grades still unable to punctuate correctly? I should know – 30-odd years ago I was one of them!

  76. NickConnolly said,

    August 25, 2007 at 1:17 am

    “i have to say
    i am amazed that people are so terrified and outraged by my suggestion that i simply plonk some old O level papers next to some new GCSE ones.”

    I am more amazed that your internet system allows you to determine unexpressed emotions in your correspondents. That certainly is a substantial upgrade from your previous web-host. As for me, I am niether terrified nor outraged – I’m not very impressed either but that is a different question. I’d like you to continue with your project if only because I’ve got a hypothesis to test now. I’m pretty sure that once you have those papers side by side you won’t know anymore about the relative difficulty than you do now. The reason for that are given above, but I’ll summarise:
    If NewTest looks subjective easier than OldTest you still won’t know if the grade boundaries were more generous – and that will make all the difference. If NewTest looks just as difficult as OldTest then the same issue hold: NewTest could still be substantially easier. The third posibility is that the two test differ so much in content and style that the argument becomes one not so much about the tests as about the syllabus.

    Perhaps you’d might like to comment – one with some insight would be good but heck, if speculation about emotional states is all you can manage I’ll even take that [g].

  77. plasecaha said,

    August 25, 2007 at 1:38 am

    richas

    That is a dramatic, ludicrous and frankly ridiculous overstatement of my position. More than once you accused me of asserting a conspiracy between the government etc. This is abject. I am not suggesting these people get together to cook up some scheme together. I am saying there is a PRESSURE on all of them in a certain direction.

    But I forgive you, because it made me laugh. Meanwhile, more corrections…

    I never said the iGCSE was any kind of indictment of the GCSE. I cited it as an example of schools selecting exam boards to suit their agenda. And regardless of whether you happen to think the iGCSE is harder or not, it happened in the past when they deliberately picked the O & C board papers because they were harder. Somehow you seem unable to accept the possibility that there may be differences and difficulty between papers set by different boards, or that schools may use that to their advantage.

    Those QCA reports you cite only go back five years, whereas we are more interested in previous eras, before all the coursework distortions, and the gradual erosion of difficult topics like algebra. I don’t especially trust the QCA report because it is self-interested, and you have an astonshing faith in these things.

    But OK, I took a look for myself at the QCA site, waded through the self-congratulatory PR, and eventually found a document that describes their approach to reviewing standards. Hilarious stuff. Some choice excerpts…

    “All appointments are made by a mixture of public advertisement and drawing on a list of experienced QCA consultants. The lead consultant is normally an existing QCA consultant. A member of QCA staff manages the work. ”

    Oh yeah, real independent.

    “The awarding bodies provide 15 examples of candidates’ work at the defined boundaries for the syllabus under review… This is difficult for awarding bodies to provide, since it involves identifying suitable candidates”

    Great, so the exam boards actually get to choose the sample themselves????

    “In general, judgements are based on only what is common to all syllabuses in the review.”

    Right, so they ignore what’s changed!! Fantastic. If calculus gets dropped from a test, to be replaced by some noddy coursework, that just gets passed by?!! But wait, there’s more good stuff in the same paragraph…

    “However, current examinations normally provide much more by way of support than used to be the case and this aspect is not taken into account in these reviews. It should be noted that while such support does much to improve teaching and learning for an examination, it does not inherently change the difficulty of the tasks involved.”

    Right. Let’s note that. It doesn’t matter how much help and support they get these days, they are going to explicitly “not take this into account”.

    The PISA thing is more interesting, international comparisons. First thing to note is there was low participation because not enough schools could be persuaded to take part, which some have regarded it as telling. They showed improvement in the past but that’s from a pretty low base in maths in particular.

    To be honest, I don’t think the rapid improvement a few years back was entirely illusory. In primary education, for example, I think that was genuine, a consequence of the introduction of the literacy and numeracy hours, among other things.

    Your faith in coursework is also amazing. In PRINCIPLE, I would agree that it has a number of advantages. There is no philosophical difference. The problem is that In PRACTICE, these benefits are frequently undermined by the cheating that goes on. I guess in your world, that doesn’t happen either. I’ve seen it happen too many times now to take coursework seriously unless properly managed. And it rarely is because it’s real bad news when colleges get funded on a bums-on-seats basis to lose students through failing modules.

    Finally, even when I agree with you, you seem determined not to see it. I AGREE with you about about the retakes thing. I pointed out one potential concern with it, but conceded that, on balance, I agreed with you. Jeez…

    Meanwhile, another correction: I didn’t say Oxbridge struggled to sort out the people with 3As. Many have more As than that, and Oxbridge set their own exams on top of that, in addition to interviews packed with trick questions, or at least that’s how it used to be. Oxford and Cambridge really have to put the effort into sorting out who’s going to stand the pace, which can be considerable as Rosy pointed out above. Everything gets crammed into three, eight week terms a year. I mentioned the Oxbridge thing in connection with the issue of PRIVILEGE – wealthier people getting a second bite at the cherry…

  78. NickConnolly said,

    August 25, 2007 at 1:39 am

    Dr Aust said: I thought the French examiner’s tale was interesting because it showed exactly how you could have an exam that retained the appearance of the kids achieving “competencies”

    I’d suggest you go back and read it more carefully. The article is purposely misleading. Look at this bit again:
    “The first role play – for the **foundation** level – is often a very simple scenario, in which the examiner plays the part of a shopkeeper and the student a customer.
    This “test” simply requires the knowledge of four words that every 11-year-old should know, such as “pommes”, “deux”, “limonade” and “combien”. No verbs, no prepositions necessary. Et voilà! You’re on your way to a grade C. ”

    The accompanying news story is even worse as it doesn’t even mention that the question is at FOUNDATION level – i.e. the LOWEST tier at GCSE. Note the weasel words “on your way to a grade C” – of course you are “on your way” to a Grade C, just as flying to Kathmandu means you are “on your way” to climbing Mount Everest. The reality is a massively distorted comparison: old O-Level standards with the lower grades available at GCSE – grades obtained by students who in the good-old-days wouldn’t have taken O-Levels and would have been lucky to have any teaching in a foriegn language (and in the good-old-old-days would have left school at fifteen or earlier with little or no formal qualifications).

  79. NickConnolly said,

    August 25, 2007 at 1:58 am

    Richas wrote:”Instead of a random collection of old papers why not look at the QCA’s published research on the 1996-2001 exams?”

    The research you cite is on the key statge tests (SATs). Now those tests do actually undergo a formal equating process – that is a statistical, experimental and mathematical modelling process which allows (up to a point) numerical comparison of difficulty between related tests.
    en.wikipedia.org/wiki/Test_equating
    The problem is that you need to build that process into your tests – its not something you can do with much validity 10 years later. Consequently because the O-Levels and GCSEs haven’t been systematically equated over the years the data to that kind of comparison simply doesn’t exist.

    You could get kids now to sit a test from 25 years ago *but* the kids now 1. wouldn’t have been prepared for the test in the same way. eg. teachers then would have known what style of question was going to be asked and rehearsed likely answers
    2. the *stakes* would be different. ie. the kids now would be taking part in an experiment which had little or no bearing on their future whereas the kids *then* would have been taking a fairly high stakes test and would be (presumably) more motivated.

  80. mattieb said,

    August 25, 2007 at 3:34 am

    Selina Morse, Post 34:

    “”

    I believe GCSE’s and the like fundamentally changed when coursework was included in the overall assessment. From being a test of “how much you can remember and apply” it became a case of “how much can you reproduce from textbooks (and subsequently the internet)”.

    “”

    The assumption in this criticism is that recall of memorised facts is somehow more worthy a skill than the ability to research issues from a range of sources, filter that information, construct an evidence based argument on the back of that, and then write it all up coherently in an extended essay. I’m not sure I would be able to agree with that assumption.

    RE: Retests. I am from the GCSE class of 2000, the great year of experimentation (first year group to sit SATs, new science GCSEs, AS-A2-level).

    We were informed that, with the AS-AS level, there would be a *restriction* on re-tests, in that were we limited to only *one* re-sit for a particular entry, in contrast to the situation for earlier years, who could resit and resit to their hearts content. Over the course of my four A-levels (yes, one was general studies) and one AS level, that came to 30 exam papers over two years. Now, I could have resit all of them, but there are only two (?) examable periods in each year (May-ish is the main one).

    Suppose someone resat everything.

    Is someone who got A grades by sitting 30 resits in the space of a month more or less capable than someone who got A grades by sitting half a dozen ‘finals’ papers?

    One last thing: The AS was supposed to be easier than half and A-level, thus you can do lots of them in the first year, but the full two-year AS/A2 level is supposed to be equivalent to a full A-level (ie, the A2 half is supposed to be harder than the AS).

  81. Don Cox said,

    August 25, 2007 at 10:50 am

    “If B needs to re-sit modules in her A-levels repeatedly, how well will she cope with the much more intense learning environment of university? In Cambridge (I’m afraid I’ve no other grounds for comparison) she’d go under in weeks.”

    In local universities (mostly former Polytechnics), there is more help for weaker students. People can do resits, take a year out to catch up on modules they failed, and so on. It would be possible for a student to take five years to complete a three year degree course.

    But these are in most cases courses in professional skills, not tests of innate ability, so the driving license comparison is valid. Also, local universities have part time degree students.

    It would be wrong to say that such degrees are only training; there is a good deal of education too. But the emphasis does tend to be on training for some specific career, which is what the students and their parents want.

    So, are the A levels testing students for suitability for Cambridge, or for local universities, or for the many in between?

    ++++++++++++++++++++++++++

    One point I can settle, having been teaching since 1961. Students haven’t changed much, if at all.

  82. jjbp said,

    August 25, 2007 at 10:50 am

    I dont understood why GCSE/A level grades are not defined in terms of where you come in the distribution of marks (i.e. your marks were in the top 10 per cent – you’re an A* grade or whatever) rather than setting a hurdle at a certain mark. That would separate subject performance in a reliable banding year on year.

  83. roGER said,

    August 25, 2007 at 12:25 pm

    I’ll never forget the first question of the very first chemistry exam I ever sat (aged 11 I think):

    “Who invented the bunsen burner?”

    a) Mr. Bunsen

    b) Mr. Burner

    c) Mr. Bunsen Burner

    That was back in 1977, doubtless the midst of some golden age now…

  84. marcdraco said,

    August 25, 2007 at 2:40 pm

    Bad Science? What about bad English – and stories that it has been awarded with excellent grades?

    Exams are murder these days.

    news.bbc.co.uk/1/hi/education/6963646.stm

    Seems like it’s not just science papers that are getting less challenging.

  85. plasecaha said,

    August 25, 2007 at 7:44 pm

    roGER, that question is stlll too hard by today’s standards. Nowadays it would have to be:

    a) Mr Bunsen

    b) Jade Goody

    c) a bucket of water

  86. Robert Carnegie said,

    August 25, 2007 at 9:05 pm

    Having re-consulted Wikipedia, it would seem to be Professor Robert Wilhelm Bunsen, except that it actually was his laboratory assistant Peter Desaga improving a previous design by Michael Faraday, and the thing in its common form is properly called a Tirrill burner for some reason. But that is in the hands of cultural relativists. It’s also Wikipedia so… you really don’t know.

    Wikipedia also thinks that “Bunsen burners have largely been supplanted by hot plates, heating mantles, and other similar electric heating elements as sources of heat in laboratories”, but this may not apply to schools.

    Did you have to draw a diagram of the thing upon making its acquaintance? I did. Maybe 1979 or later, not sure when we were actually given the matches.

  87. plasecaha said,

    August 25, 2007 at 9:21 pm

    Wish I’d read the article by the French examiner earlier:

    www.telegraph.co.uk/news/main.jhtml;jsessionid=LCDNVTZ4ET2MZQFIQMGSFFOAVCBQWIV0?xml=/news/2007/08/23/ngcse223.xml

    Because the discussion had been about the easiest questions, and some people had pointed out that one cannot simply go by the easiest questions in a paper, which are there to differentiate performance at the lower end, I thought “well, fair enough, they’re right about that” and paid no further heed to it.

    But that article goes on to examine the modern French paper as a whole, and tells us all we need to know about how things are nowadays. The modern French paper is SO much easier, that reservations are academic, and concerns about grade boundaries and so on are not just marginal, they are entirely irrelevant, utterly moot.

    For it doesn’t matter whether or not on the new paper one gets an A for scoring 60 percent, 70, 80 or even 90 percent… one could could get all the questions right on that paper and still not do enough to get a C under the old system. Just as in Physics… it doesn’t matter how many questions about CCTV cameras you answer, you are never going to do OK on old papers with all that calculus. People who studied under the old system, however, would have no trouble talking about the future implications of CCTV or doing the modern French paper.

  88. NickConnolly said,

    August 25, 2007 at 9:59 pm

    jjbp said, “I dont understood why GCSE/A level grades are not defined in terms of where you come in the distribution of marks (i.e. your marks were in the top 10 per cent – you’re an A* grade or whatever”

    That is norm-referencing – it works only if you have a stable test population and the test produces similar distributions (but possibly with different means). If the test population changes then ‘standards’ on the test change.
    Standards referenced test set the difficulty by assuming some sort of pre-determined hierachy of skills. Grades (or levels in the case of SATs) to cover bands of skills in that hierachy. The problem with that is that question difficulty is not entirely determined by the skill assumed to be tested.
    Equated tests use comparitive data to put all test items on a common numerical scale of difficulty. For example a sample of students may sit this years test AND last years test: ie common person equating. Alternatively some test questions from previous years are included in this years test: ie common item equating. Grades are then set on the basis of the common scale rather than the students raw mark.
    The SATs (key stage tests at KS2 & KS3) are equated standards referenced tests. Doing the same for GCSEs is difficult because multiple exam boards means this isn’t just one form of any one subject paper.

  89. NickConnolly said,

    August 25, 2007 at 10:07 pm

    Plasecha;
    Well you seem to have made your mind up already. Again that rather points to the futility of the exercise Ben started.

    “But that article goes on to examine the modern French paper as a whole, and tells us all we need to know about how things are nowadays.”

    Odd then that they spent so much of the article focusing on a fallacious comparison. People tend to lead with their strongest argument – notably the ‘French Examiner’ led with comparing *FOUNDATION* level French with old O-Level. Why would she do that? As an examiner she could hardly be ignorant of the silliness of such a comparison. Her points about the rest of the exam where more hand waving than details and her objection that people ‘teach to the test’ for French is 1. hardly surprising, 2. inevitable for any established French test *including* the old O-Level (see if you can devise a consistent & valid French test that DOESN’T involve memorising a whole bunch of stuff).

  90. plasecaha said,

    August 26, 2007 at 2:49 am

    It’s not fallacious, she made it clear it’s a foundation question. You take it as part of the exam. And it makes no difference anyway, because the KEY POINT is that the more difficult questions are way easier than they used to be. That is the telling thing.

    “I note finally your post still is struggling to grasp that a harder task with an easier mark is not neccesarily a “harder test” than an easier task with a harder mark. The issue remains one of discrimination. Your current comparison is as daft as saying that the Olympic 100 metres race is ‘easier’ than the 1500 metres because the athletes have fewer metres to run.”

    No, I have grasped that and said so: if the questions were closer together in difficulty, grade boundaries etc. WOULD be significant. You are right about that and I have accepted it.

    What YOU don’t understand, is that when the difference in difficulty between the papers is sufficiently large, the grade boundary thing etc. becomes irrelevant, because it doesn’t matter HOW well you do on the easier paper, or how the marking is set up, someone could get a perfect score on the easier paper and still not be able to do the questions on the harder one.

    The easier paper is so easy, that even the BEST candidates, who get a perfect score, will still not have done anything like as hard as on the old paper. The marking scheme is irrelevant, because whether you get an A for 40 percent or 90 percent, with a perfect score, you are getting an A whatsoever.

    And yet, they still would struggle to get a C on the old paper, so wide is the gulf. Anyone doing WORSE than this perfect candidate would be even worse off, so marking schemes are even more irrelevant.

    And that example is nothing like my argument at all. If you want a more appropriate analogy, it is like you saying that someone might get a decent grade at A level French if they do well in the GCSE level, if the grade boundaries are arranged sufficiently. But that could never happen, because the A level paper is so much harder, it doesn’t matter HOW well you do on the GCSE, or how the marking is arranged, you will never do good enough on the GCSE paper to match the requirements of the A level.

    You seem to think that no matter HOW big the difference in difficulty of the papers, this can always be accommodated by the marking scheme. And this is so wrong.

  91. plasecaha said,

    August 26, 2007 at 3:26 am

    Nick

    “that is the claim being made. A claim isn’t true by virtue of it being claimed – the question is what eveidence supports the claim. The examiner’s evidence can be split into
    1. the vague
    2. the fallacious
    Perhaps her claim is correct but she has failed to support it – and indeed has given good reason to question the ingerity of the evidence she did offer.”

    Basically, you are trying an ad hominem attack – try and avoid the uncomfortable evidence of the author by discrediting them. Try and claim something is fallacious when it isn’t and use that to ignore the rest.

    She wasn’t vague, she pointed out quite clearly how students are given canned responses and cue cards these days. This is DRAMATICALLY easier than before.

    You don’t really want proof Nick, do you? Ben’s idea of getting papers to actually compare would prove that this actually DOES take place, but you were down on him doing that. You could check the evidence for yourself, like I did when someone mentioned the QCA earlier, and you will see she’s telling the truth.

    How do I know? I took the original French O Level. More recently, I took German classes in the evenings… I decided in the end to switch to the business German exam instead, but saw enough before then to easily see it’s all about canned responses these days, and nothing like it was before.

    But don’t take my word for it Nick – an examiner – who doesn’t have the vested interest that exam boards or teachers may have – is a good source. Or, you can support Ben’s endeavour. Or, you can check it yourself… it doesn’t bother me if you want to keep your head in the sand…

  92. NickConnolly said,

    August 26, 2007 at 4:05 am

    Comparing Foundation level with O-level is fallacious. I appreciate that you may not be fully aware of why but the examiner certainly would have been.

    “No, I have grasped that and said so”, then I have no explanation as to why your posts repeat the same error of reasoning. All I can do is point out the error.

    “The easier paper is so easy, that even the BEST candidates, who get a perfect score” – and how many is that then? You are starting to argue my point. You can’t determine that figure by looking at the papers.

    “You seem to think that no matter HOW big the difference in difficulty of the papers, this can always be accommodated by the marking scheme.”
    Not at all, in fact the exact opposite. Read my FIRST message on this thread where I explained to Ben the mistakes he was making. The most important question is DISCRIMINATION.
    Honestly this stuff isn’t that difficult but people seem to insist that naive ideas of testing and education are naturally superior to actually learning stuff.

  93. plasecaha said,

    August 26, 2007 at 4:28 am

    She just worked through the exam, starting at the beginning and didn’t leave anything out. The important point, is how easy the subsequent stuff is. I rather doubt a national broadsheet would have published such an article, if indeed it was talking complete balls about what was in the exam, something which we can be sure would not have gone unchallenged. All papers publish contentious stuff, but not something that easy to expose. I know you don’t want to accept that the exams are really like that Nick, but sadly they are, and of they weren’t none of us would be bothered about it.

    “No, I have grasped that and said so”, then I have no explanation as to why your posts repeat the same error of reasoning. All I can do is point out the error”

    Actually, it’s not necessarily a trivial point for me to explain where you are going wrong…
    “The easier paper is so easy, that even the BEST candidates, who get a perfect score” – and how many is that then?”

    Ah, now I can see one misconception on your part. In order to show the problem with the grade boundary et al theory, I talked about a perfect score, by which I mean a perfect set of answers. A perfect score means getting every question as correct as is possible. Such that, no matter what marks or grades they decide to give, you couldn’t actually have PERFORMED any better.

    Since that perfect performance is way below even just an OK performance on the old O level, IT DOESN’T MATTER what marks they give, or how they arrange grades etc. – you still wouldn’t be good enough to get a C on the old paper.

    Basically, there are limits to the influence marking and grades etc. have. If you were correct, and one could always make one paper harder than another by rejigging the marking, you could in practice make a GCSE harder than a postgraduate qualification by adjusting the marking. Clearly, this is absurd, so there are limits to how big an effect the marking plays. You cannot adjust the marking in the new GCSE sufficiently to make it comparable in difficulty in practice to the old exam, because the latter’s difficulty eclipses anything you can do with the marking.

    The case of the perfect performance highlights this. If even someone getting a perfect score and top grade cannot do the old exam, then it is impossible to use the marking to make the new exam harder than the old in practice.

    “Not at all, in fact the exact opposite. Read my FIRST message on this thread where I explained to Ben the mistakes he was making. The most important question is DISCRIMINATION.
    Honestly this stuff isn’t that difficult but people seem to insist that naive ideas of testing and education are naturally superior to actually learning stuff.”

    Nick, you keep claiming you are right, but don’t actually provide a proper explanation of why the way I have, again, above. I read your stuff, and have explained in detail why you are wrong, and you keep going “nah, you just don’t get it”, “read it again” etc.

    I keep reading it, and your stuff doesn’t get any more right. Explain how you could rejig the marking to make any exam more difficult than another, even if the other actually contains questions that are much more difficult. Explain how you could, through messing with grade boundaries, make a GCSE in practice more difficult than an A level.

    Can’t be done.

  94. NickConnolly said,

    August 26, 2007 at 5:33 am

    ” I know you don’t want to accept that the exams are really like that Nick,”
    Apparently you “know” many things based on zero evidence. I haven’t expressed an opinion as to what exams are “really” like. Apparently Ben isn’t the only one with a psychic internet connection.

    “Ah, now I can see one misconception on your part.”
    hmm no. I know what a perfect score means. I was asking you how many students get perfect scores. It was primarily a rhetorical question as I’m pretty sure you don’t know. After that you didn’t manage to follow the rest of my point. You maybe need to read it again.

    ” I read your stuff, and have explained in detail why you are wrong, and you keep going “nah, you just don’t get it”, “read it again” etc.”

    There isn’t much more I can say, is there. You obviously didn’t understand my point about perfect scores for example.

    “If you were correct, and one could always make one paper harder than another by rejigging the marking”
    That is what is called a ‘straw man’ argument. Again it is a fallacious argument. We could conduct the discussion that way – I could exaggerate and then misreprsent your position for example – but I got tired of that mode of argument at least a decade ago.

    ” Explain how you could rejig the marking to make any exam more difficult than another” – the same strawman repeated. I didn’t claim ANY pair of exams – we are talking about roughly comparable exams. For the kind of differences you are talking about there wouldn’t be any question. Or are you ACTUALLY claiming that the diffwerence between O-Level and GCSE is equivalent to the difference between a PHd and

    “If even someone getting a perfect score and top grade cannot do the old exam,”
    And can you show me where, anywhere, the examiner provided ANY evidence that somebody with a perfect score on the modern GCSE couldn’t do an O-level? This is a whole new claim you are making made out of whole cloth.

  95. plasecaha said,

    August 26, 2007 at 6:56 am

    “hmm no. I know what a perfect score means. I was asking you how many students get perfect scores. It was primarily a rhetorical question as I’m pretty sure you don’t know. After that you didn’t manage to follow the rest of my point. You maybe need to read it again.”

    It’s absolutely clear now you don’t understand my point Nick, because the whole point of my example is that it doesn’t matter how many do a perfect performance. If all of them do, or none of them, or any number in between, my argument holds.

    That’s why I expressed it that way. To demonstrate your approach is impossible. You are not actually working through my argument, what you are doing is just looking out and hoping to see if there is anything you can kinda leap on. I actually understood your point Nick, and expressed it back to you in ways to make it clear I did, and agreed it works in certain situations.

    “There isn’t much more I can say, is there. You obviously didn’t understand my point about perfect scores for example.”

    Actually, you didn’t understand mine. It doesn’t matter how many get perfect scores, the point is that EVEN with a perfect performance and the most favourable marking approach, the new paper cannot be harder than the old O level.

    “That is what is called a ’straw man’ argument. Again it is a fallacious argument. We could conduct the discussion that way – I could exaggerate and then misreprsent your position for example – but I got tired of that mode of argument at least a decade ago.”

    No, it’s a standard logical device called reductio ad absurdem – taking your argument to its logical conclusion to demonstrate the absurdity of it. I know what a straw man is and it’s something different. A straw man is a misrepresentation of someone’s position in order to make it easy to attack, and I didn’t do that. If you want an example of straw men in action, check my last response to Ricas, because in that post I had to deal with a litany of them. Or you could check your own misrepresentation of the examiner’s point about drilling, earlier.

    And anyway, you started it with your Olympic example, which didn’t even apply. At least mine works…

    “the same strawman repeated. I didn’t claim ANY pair of exams – we are talking about roughly comparable exams. For the kind of differences you are talking about there wouldn’t be any question. Or are you ACTUALLY claiming that the diffwerence between O-Level and GCSE is equivalent to the difference between a PHd and”

    No of course I’m not claiming that. I made clear that I didn’t. I am pointing out how ridiculous your argument is. You seemed unable to see that there is a limit to how much of an impact the marking and grade boundaries have. You kept repeating your argument as if it always held, and therefore was unchallengeable, in effect arguing as if there isn’t a limit, and I am pointing out how ludicrous it is to do that by illustrating what happens if we follow your logic.

    I had to accept that there are conditions in which your argument holds, Nick. I didn’t have any trouble doing that. What I am doing now is showing that you have to accept there are conditions under which MY argument holds: you cannot use marking etc. to make ANY paper more difficult than another.

    If you accept that, all we have to do is show you that the old O level is so difficult that it does in fact eclipse even the most favourable use of marking to compensate. Read on…

    “If even someone getting a perfect score and top grade cannot do the old exam,”
    And can you show me where, anywhere, the examiner provided ANY evidence that somebody with a perfect score on the modern GCSE couldn’t do an O-level? This is a whole new claim you are making made out of whole cloth.”

    She gave examples of the difference in difficulty of the exams. Clearly you are not aware of just how much harder the O level was in comparison. That’s the point: the difference is so huge, reservations are academic. I was kinda hoping you would see the dramatic difference between being given canned responses with cue cards, and told when to say them, to being able to spontaneously improvise a conversation without such aids. This isn’t marginal, Nick. It is, in effect, the difference between actually being able to speak French, and being able to just do little more than memorise and trot out a few lines from a phrase book. It’s huge.

  96. RS said,

    August 26, 2007 at 10:52 am

    This thread is generating lot more heat than light.

  97. richas said,

    August 26, 2007 at 1:12 pm

    plasecaha – you seem to have added a bit to the anecdote from the French examiner. Your claims from it go beyond what they said.

    From te old O level all we have is an optional essay question which is contrasted to a spoken French test. This does not demonstrate that the 1959 pupil could say anything. It is perfectly possible to be able to write French and yet be completely unable to make yourself understood verbally. This example just shows how difficult and pointless it is to look at very different exam papers without acces to the mark scheme and what they are assessing.

    With a simple norm referenced exam less thought is required in creating the paper. Essentially you ask a set of questions, mark them and rank the candidates. Job done. A standard based test is more difficult you need to test all the different aspects of the curriculum in a way that is comparable to your previous papers. The tendeny is to need shorter compulsory questions to get the coverage you need. The questions also need to be clear so that the candidate is treated fairly in being given an opportunity to demonstrate the skill or knowledge being assessed.

    Standards based tests because of their format compared to the simpler norm referenced paper (typically a choice of essay questions). The questions may look simpler but the knowledge/skill is being assessed in a more systematic way. The design of the paper is far more imporant to maintaining the standard than a norm referenced paper where all you are doing is comparing them to their peers.

  98. richas said,

    August 26, 2007 at 1:38 pm

    1. Surely the question is, does GCSE or A-Level better prepare students for whatever they do next now, or did it in the past?

    The answer that I would propose is no, it is a far inferior preparation these days. I base this on the appalling numeracy and literacy skills of students entering university (based on my own observations). Back when I did my engineering degree the entire first year maths syllabus was basically everyone should have known.

    A good question spoilt somewhat by falling back on the anecdote of the old viewing the young and judging them by their current standards not how ignorant and callow they were at the same stage.

    Let us look at the past. Back in the late 1950s (golden era LOL)more than half of all school leavers left with no qualifications whatsoever. Nothing to show employers and no prospect of continued education. About 5% went to university.

    By the seventies early eighties about 10% went to university and “only” a third left with nothing. This period saw the heyday of the CSE in metalwork or home economics as the only bit of paper to show your employer (or the job centre).

    Now about 45% go to university, about 3% leave with no qualifications and about 45% get 5 “good” GCSEs including maths and English – roughly the proportion that managed to get anything at all in the good old days.

    Now I would no be surprised if the uni student on the 45th percentile has lower literacy and numeracy skills than the fifth percentile university student in the sixties but this tels us nothing at all about the performance of education in this country. Your anecdote just does not stack up. The change is not the exams it is a seismic shift in terms of educational attainment for the majority.

    Many find the terrible performance of education in the good old days a bit shocking. It is, the proof we have of this is the appalling levels of adult literacy and numeracy. Millions of adults do not meet the KS2 benchmarks we expect of 11 year olds today (and 75% achieve).

    If we want to judge the exams of today against the exams of yesteryear this legacy of past failure shows why today’s setup is far superior.

  99. RS said,

    August 26, 2007 at 1:43 pm

    www.timesonline.co.uk/tol/news/uk/education/article1499956.ece

    “Pressure will grow for GCSE language exams to be made easier, after a comprehensive study by Durham University disclosed that they are at least a grade harder than many other GCSEs.

    The findings are likely to be endorsed by Lord Dearing, who is expected to recommend that language GCSEs should be overhauled and marked more generously in a radical shake-up of language teaching and assessment in schools.”

  100. plasecaha said,

    August 26, 2007 at 2:12 pm

    Ricas,

    “From te old O level all we have is an optional essay question which is contrasted to a spoken French test. This does not demonstrate that the 1959 pupil could say anything. It is perfectly possible to be able to write French and yet be completely unable to make yourself understood verbally.”

    Actually, Ricas, she moved ON to the written portion of the exam, citing the 1959 example, AFTER having dealt with the oral part, in which she DID contrast explicitly with the O level.

    For example, from the article:

    “Did someone mention the subjunctive? The third role play, attempted by higher candidates only, used to be a difficult task: a photo to be described in detail with no helpful vocabulary included.

    I remember one of these being of a man with a tennis ball stuck in his mouth.

    Most of them are now reduced to a series of straightforward scenarios, such as a day trip to France.

    Candidates have to narrate a story outlined for them in picture form, with key words and phrases under each picture. Only putting the verbs into the perfect tense and a little padding are required for another 16 per cent of the mark.”

    Here, Ricas, you can see clearly she contrasts the current state of things in the oral exam in which the role play is about something straightforward and with absurd amounts of assistance- the day trip to France – with the stuff they USED to have to talk about: the tennis ball in the mouth.

    Another example: the presentation part of the exam in which you used to have to deal with the “poker-faced complete stranger visited the school to fire unprepared questions at you.”

    (Ricas – he did not fire questions at us poor O level students by writing notes on a piece of paper and handing them to us. He actually SPOKE them!!)

    As opposed to now, when “the pupil makes a short, pre-prepared presentation on any easy topic – normally their school or a family holiday. Straightforward questions are then asked by the pupils’ own teacher….”

    followed by the bit with the cue cards…

    “candidates are also allowed to take cue cards with up to five short headings into the examination room with them, just in case they forget what they have memorised a hundred times over.”

    So yeah, she directly notes that you used to have to do an oral in days gone by, and indeed, she makes clear from the start she is comparing with O level…

    “Then you are no doubt over 40 and spent hours slaving over these moods and tenses in order to secure your O-level pass.

    To equate a GCSE Grade C with such a demanding exam would be laughable if it were not so depressing.”

    I mean, how could anyone reading the article miss all this? It’s as bad as Nick saying the examiner described the thing about the foundation question in most detail, when in fact she only wrote three sentences on that, while she wrote four on the section that followed, and six on the third of the three role plays.

    This is a new level of illiteracy I have hitherto not experienced.

    The oral had long been a part of the O level… I didn’t take the O level in 1959, but when I took it, there was definitely an oral, because it is not something I will ever forget. I remember the poker-faced man firing questions, can still remember what he looked like, though to be honest, he was more pitying than poker-faced. It was far and away the most difficult part of any O level I ever took, Latin, not exactly trivial, was a doddle in comparison: translating Caesar’s Gallic Wars felt like a walk in the park in comparison, and I consider myself lucky to have scraped a C in the French.

    I’d have had no trouble with things the way they are now, though, as I discovered when I returned more recently to take German.

  101. plasecaha said,

    August 26, 2007 at 2:20 pm

    Ricas,

    “With a simple norm referenced exam less thought is required in creating the paper. Essentially you ask a set of questions, mark them and rank the candidates. Job done. A standard based test is more difficult you need to test all the different aspects of the curriculum in a way that is comparable to your previous papers. The tendeny is to need shorter compulsory questions to get the coverage you need. The questions also need to be clear so that the candidate is treated fairly in being given an opportunity to demonstrate the skill or knowledge being assessed.

    Standards based tests because of their format compared to the simpler norm referenced paper (typically a choice of essay questions). The questions may look simpler but the knowledge/skill is being assessed in a more systematic way. The design of the paper is far more imporant to maintaining the standard than a norm referenced paper where all you are doing is comparing them to their peers.”

    I can accept all of that, Ricas. I can accept that it may be helpful in a standards-based system to break things down into smaller units. And I can agree it’s got something going for it, and that it can make it APPEAR as though things are easier when they are not.

    However, that isn’t what is happening with the French. I mean, they may break it down, but the stuff they are breaking down is intrinsically much simpler. The students don’t actually have to be able to speak French freely in order to be able to do that exam, whereas they did used to for the O level, and that is a difference which transcends any concerns about breaking down questions.

  102. RS said,

    August 26, 2007 at 2:24 pm

    plasecaha, while the examiners view is interesting – it is still anecdotal – you seem to have taken the examiners comments as proof that grade C GCSE students couldn’t even attempt O Level French, which the article neither says, nor gives any evidence to establish.

    For instance, I presume that your O Level oral was not free to roam into any territory and vocabulary that the examiner saw fit.

  103. RS said,

    August 26, 2007 at 2:26 pm

    “Ricas – he did not fire questions at us poor O level students by writing notes on a piece of paper and handing them to us. He actually SPOKE them!!”

    Where do you derive the idea that GCSE oral examinations do not have spoken questions? The only example where anything is written down is the story narrative section, and here it is contrasted with describing a picture.

  104. plasecaha said,

    August 26, 2007 at 2:33 pm

    RS

    “plasecaha – your point that a C candidate on the modern GCSE couldn’t attempt the old O level is just your opinion and backed up by very little. You are quite entitled to that opinion but then others are perfectly entitled to disagree.”

    It’s not just my opinion, RS. The examiner’s article makes it pretty clear. But I accept others can disagree, sure. The thing that’s rather bemusing is when someone says there isn’t enough evidence, then maintains it wouldn’t necessarily be a good idea to post past papers, when in fact that is the one thing which would prove the point.

    On balance, at least my point has SOME evidence going for it – the examiner’s article – whereas no one is providing any evidence to counter.

    “but to pretend that old O level students did not memorise key phrases likely to come up is disingenuous to say the least. (incidentally, why is no one focusing on the CSE which would be similar to a foundation level GCSE?)”

    Disingenuous? It wouldn’t have helped much, because the oral was more free-flowing, wasn’t as prescribed as the GCSE. You couldn’t predict as much what was going to happen, you didn’t know what the other guy was going to say. Sure, you thought about it, but there were no canned responses or cue-cards. They wouldn’t have been much use.

    (If we COULD have memorised key phrases like that, I would have found it a great deal easier, believe me).

    “The article establishes very little so harping on about it doesn’t win any arguments. The examiner doesn’t present enough details about O Level oral tests to make any comparison.”

    She establishes the point that nowadays you can get by with canned responses and cue cards, where in the past, you couldn’t. You had to be able to speak more freely. Which is a huge difference.

    “Seems pretty unlikely to me – the main reason kids don’t do language GCSEs is because they are optional and hard because languages are badly taught, and taught too late.”

    I wouldn’t necessarily disagree with you there.

  105. plasecaha said,

    August 26, 2007 at 2:41 pm

    RS

    “I’d point out again – that no one here seems concerned that given the rising participation rate and thus increased numbers taking A Levels and O Levels, the performance of a grade C candidate in the mid-80s must have been much lower than that of a 1950s/1960s candidate – given that grades were explicitly given out as a fixed percentage.”

    Yes, good point. It’s tempting for all of us to try and defend the qualifications we took as being at least as hard as those previous, ego and all that, but I’m in no doubt that in many cases, the O levels and A levels in the years before I took mine were even harder. Our teachers quite often reminded us of this, and I was naturally a touch cynical, until…

    … I entered the world of work, in which quite often I was required to submit written proposals, and quite often they would be handed back with various corrections of my English. This despite the fact that amongst my peers, I’d always done fine with English, and even attended more prestigious academic institutions than the older guys critiquing my work. Truth is, the standards WERE higher in their day. Higher in maths, too.

  106. RS said,

    August 26, 2007 at 2:41 pm

    Re: cue cards I believe that she says that they can have five short topic headings – to help them remember their presentation – which is slightly different to what you’re implying.

  107. RS said,

    August 26, 2007 at 2:48 pm

    I’m also not too sure about the stock phrases thing – I guess I took French more recently – but I certainly recall that stock phrases make up quite a large part of French – grammar, verbs, vocab sure, but stock phrases are right up there, they often help to provide a structure to what you’re trying to say and help you to avoid the various pitfalls of just freestyling a sentence in French where you can often use the correct tense and verb and vocab, and still have it be wrong.

  108. plasecaha said,

    August 26, 2007 at 2:52 pm

    RS

    “For instance, I presume that your O Level oral was not free to roam into any territory and vocabulary that the examiner saw fit.”

    You sat there, and had a conversation, basically. It wasn’t prescribed the way it is now. Why do you find that so hard to believe? All our lessons were conducted in French, we spoke French all the time. We read French magazines regularly, most went on exchanges in the holidays to live in France…. we didn’t need canned responses or cue cards. It was a different league, that’s why the examiner makes a point of the canned responses and cue cards. If it had been the same for the O level, there would be no point mentioning it.

    “Where do you derive the idea that GCSE oral examinations do not have spoken questions? The only example where anything is written down is the story narrative section, and here it is contrasted with describing a picture.”

    God, not you as well. There was nothing in the comment you quoted to suggest that I thought GCSE does not have spoken questions. Obviously they do, and that’s what the canned responses are for.

    I made that point as part of the answer to Ricas who seemed to think the article gave no indication that there was an oral element in the O level.

  109. plasecaha said,

    August 26, 2007 at 3:05 pm

    RS

    “I’m also not too sure about the stock phrases thing – I guess I took French more recently – but I certainly recall that stock phrases make up quite a large part of French – grammar, verbs, vocab sure, but stock phrases are right up there, they often help to provide a structure to what you’re trying to say and help you to avoid the various pitfalls of just freestyling a sentence in French where you can often use the correct tense and verb and vocab, and still have it be wrong.”

    Maybe it’s hard to believe. The class I was in had gone beyond all that, leaving me struggling somewhat in their wake, which is why I think I was the only one to only get a C. Certainly, when one is beginning to master French, stock phrases are a very necessary thing to structure things at the start. But beyond a certain point…

  110. RS said,

    August 26, 2007 at 3:09 pm

    plasecaha – if one masters French at O Level, what does one do at A Level, or University?

    Maybe it was all better in your day – certainly I wouldn’t say that I, or anyone I was at school with, ‘mastered’ French.

  111. RS said,

    August 26, 2007 at 3:32 pm

    “you presumably did GCSE”

    I did GCSE when it was fairly young – and we were taught in French, read French magazines, and went on a French exchange. I wouldn’t say I easily conversed in French mind you.

    I also did Latin GCSE, and that involved translating straight from original texts too, as I recall, although it was an odd exam, part-foreign language, part-literature, part-history.

  112. RS said,

    August 26, 2007 at 3:36 pm

    I suppose if you want an anecdotal case study – I studied French GCSE as a normal all-exam at the end of two years subject, and German GCSE as a trendy new modular subject. I was ok at French, but never really got on with German, and being put in a lower set for German (for obscure timetabling reasons) I got pretty poor teaching – yet I managed to get an A in both subjects, despite my German not really being good enough to carry on even a rudimentary conversation, certainly compared to my French.

  113. plasecaha said,

    August 26, 2007 at 3:43 pm

    Lol, RS… you beat me to it!!!

    So cool…

    OK, so we had twice as long as you. That makes it a bit easier to understand…

    Given it was only two years, an A sounds pretty good, better than my C, lol…

  114. RS said,

    August 26, 2007 at 3:48 pm

    Started at 11, took it at 15 – so same time frame, German and Latin started at 12.

    I think I said above that the oral sounded a bit like ours in the sense that it was generally role played rather than straight conversation although I think there might have been some general conversation right at the end – can’t really remember to be honest – and it was our teacher that did the oral, with it taped and sent off for marking – the writing sounded a bit like what we had, often things like ‘write a letter to X, talking about Y’ but fairly free-form in terms of content.

  115. RS said,

    August 26, 2007 at 3:50 pm

    Oops, no, by end of two years I mean that the GCSE course is taught for two-years, we studied French from the start of secondary school so three more before that.

    And I use ‘taught’ in the loosest possible sense.

  116. plasecaha said,

    August 26, 2007 at 4:08 pm

    Role-play’s OK… We didn’t have as much role play in the O level, but I found it handy stuff with the German more recently.

    It’s the identikit answers, every member of class saying the exact same thing, drilled by the teacher… ironically, this is supposed to be more progressive education, but it’s more didactic and drilled than in the more formal education of the past.

    Ah, so we had roughly the same amount of time, then. Though it also depends how many lessons a week, and so on. I can’t remember how many lessons we had, just that there seemed to be quite a lot of them.

    I should have had five years or more really, but I got “accelerated”, and then they made me take French and Maths early so they could cram some more O Levels into my sorry life. The norm for our school was to start at ten and do O levels at 16. Which would give rather more time, hence you could give more demanding exams.

  117. richas said,

    August 26, 2007 at 9:42 pm

    I just don’t remember any of my mates from comprehensives doing languages back then…

    Well 1959 pre dates widespread comprehensive education so the question is about the secondary moderns. Many did not do any O Levels just CSEs, languages were indeed less common but as most left with no qualifications at all I guess any sort of meaningful assessment was rare.

    In this debate we seem to be contrasting the education of about 20% of the school population with the national curriculum for all pupils today.

    Plasecha – as you accept tha standards based exams do ask smaller, simpler but more systematic questions than with norm referenced papers just how do you think comparing the two can tell you anything but the subjective appearance of difficulty inherent in the question type? Looking at just the questions not the mark scheme or the curriculum would not allow you to assess the difficulty of an exam.

  118. plasecaha said,

    August 26, 2007 at 11:16 pm

    Well Ricas, there’s the thing.

    That’s one of the things that makes this thread interesting. The challenge of gauging comparative difficulty.

    It’s rather frustrating, even annoying, because you can compare things and see that one of them is obviously VASTLY more difficult than the other, and yet still not be able to even have any kind of stab at saying by how much.

    Witness, the calculus/CCTV comparison I keep going on about. It’s pretty obvious that calculus is a lot harder than just listing a few pros-and-cons of CCTV, but is there an easy way of even proving that is the case at all, never mind by how much?

    Yet who would dispute it? Almost anyone could list a few concerns regarding CCTV, even without studying the issue properly, but the same can not be said for calculus.

    That, of course, can be one gauge, though. How many people find something difficult. One could approach it another way, and try and gauge things like the degree of abstraction involved, or the complexity of the task, both of which highlight the difficulty of calculus. Or the amount of knowledge necessary to even have a go, or maybe how much accuracy is required, etc. etc.

    Something like calculus scores pretty highly on all these measures.

    The difficulty of this problem, is one reason why I focused on the issue of accessibility. On whether or not a GCSE candidate could even reasonably have a go at the paper. If something is so much harder that it is obvious that candidates with GCSE-level sklls couldn’t even access it, it kinda absolves us of having to do more detailed comparisons.

    I suggested this approach, in order to deal with the problem of grade boundaries, but it isn’t automatically necessary that we have to have such a challenging yardstick. Yes, grade boundaries are an issue, but being realistic, not as much as has necessarily been made out.

    For example, if I recall correctly, typically one used to have to get 70 percent to secure an A at O level. If you want to make an easier paper harder in practice, then you can do so by shifting the boundaries forwards or upwards. Such that it would require 80 or even 90 percent to get a top grade.

    I rather doubt that in reality, they would have pushed the boundary past 90 percent for the top grade, so at worst we are probably dealing with a shift from 70 to 90 percent… significant, but not necessarily earth shattering.

    In order to render concerns moot, I suggested the more dramatic standard of accessibility, but it’s almost certainly overkill in practice. It just avoids the need to quibble over grade boundaries. But it ALSO allows us to avoid quibbling over comparative difficulty of questions too.

    I mean, to be honest, if the differences aren’t so stark, then there’s less of an issue anyway.

    Meanwhile, regarding the breaking-things-down-into-smaller-chusks thing… I accepted it may not affect the difficulty of the task. however, this does not mean this is always the case. You CAN make a task easier this way, because you CAN be doing some of the hard work for them.

    For example, I did Nuffield Physics a school, which was a bit progressive at the time, but had a lot going for it. One of the traits of the programme, was to lead you through derivations of some formula, by breaking it down into a series of questions which you answered, thus leading you to the answer.

    This was a handy way of helping people with a more complex derivation. However, it DID make things easier, because in breaking things down like that, it suggested each next step to the pupil, rather than the pupil having to figure out that part for themselves. What to do next, can be the hard part in a question.

    So yeah, it’s a difficult issue all roand. And because it’s difficult, it’s easy to dismiss the task. But the difficulty means it’s more of a challenge, and even though some things are hard, like gauging comparative difficulty, it doesn’t mean we can’t have a go, or even obviate the problem in some way, which is always something I like to try and do, because it has a certain elegance about it.

    The difficulty is no doubt why the QCA basically avoided the problem by simply declining to compare things in a paper which were significantly different. I don’t have a lot of time for the QCA because it is abundantly clear to me that as an organisation, it is not very “intelligent”.

    The national curriculum, as an idea, has a lot going for it. But in practice, it obliges teachers to come up with the meat of the curriculum themselves in some way… the curriculum mostly just sets the objectives. This is a big ask, to expect say a Primary teacher to come up with an entire year’s activities to meet all those objectives in all subjects, and then do it all over again when they move year groups.

    EVENTUALLY, the QCA came up with a barebones scheme, bless ‘em. By then, teachers were now being obliged to come up with their own assessment programmes for mid-term assessments, another onerous task… now, at last, it seems the QCA are going to be doing something, years later. Having left the teachers to struggle on for years.

    Too little, too late, basically.

    The assessment thing, can actually give us another angle into the issue of gauging difficulty. Back when I was 23, I had the task of designing a chunk of a BTEC diploma for national validation, ond of course that meant designing the assessments. It involved various media technologies, and thus contained the challenge of deciding how, for example, one might assess someone’s video or audio recording.

    I looked around, and discovered that one way of approaching it was to differentiate between the basic skills needed to complete a task, the degree of abstraction involved, and then the use of creativity/problem-solving.

    With abstraction being the use of a skill in a different context, and creativity the creation of something unique and useful – the more unique and more useful, the more creative.

    Thus a student who just applies a skill gets some marks, if they use that skill in a different context, a few more, and if they use it creatively or to solve a problem, still more. We could, for example, apply the same method in reverse, by gauging the amount of abstraction or creativity necessary to do an O level or GCSE question.

    There are probably quite a few ways into the problem…

  119. plasecaha said,

    August 26, 2007 at 11:47 pm

    Ricas,

    Regarding the languages thing… yeah, I know it was secondary moderns back in 1959, that back then you either went to one of those, or a grammar school. It changed to comprehensives by the time I was doing O levels, though. Indeed, I think if I remember correctly, I was ten when they abandoned the 11-plus in our neck of the woods, much to the chagrin of my mother, since I was going to school on a council estate and the standards left something to be desired.

  120. NickConnolly said,

    August 26, 2007 at 11:49 pm

    Apolgies: I’ve ignored your previous reply to me as it was getting us nowhere. Agree to differ at that point. I’ll move on.

  121. plasecaha said,

    August 26, 2007 at 11:59 pm

    OK Nick…

  122. Ben Goldacre said,

    August 27, 2007 at 12:22 am

    as fascinating as i find this, nick will be relieved to hear that so far i only have an o level maths paper from 30 years ago.

    anybody want to send me more?

    c’mon…

  123. plasecaha said,

    August 27, 2007 at 1:50 am

    I don’t… it was the fashion, back in my day, to burn in celebration all notes, papers, and pretty much anything to do with exams the instant you were done with them.

    He says ruefully.

    I’m going to ask around though…

  124. plasecaha said,

    August 27, 2007 at 2:28 am

    For Nick’s benefit, here’s a discussion between students about current grade boundaries and the differences between them in recent examinations.

    www.thestudentroom.co.uk/showthread.php?t=220846

    They’re very enterprising, it must be said. We never gave a toss about grade boundaries in my day…

    They have even set up a Wiki for past papers, though the furthest it goes back is 1990…

    thestudentroom.co.uk/wiki/Mathematics_Past_Papers

  125. NickConnolly said,

    August 27, 2007 at 3:16 am

    “nick will be relieved to hear that so far i only have an o level maths paper from 30 years ago”

    [grin] I did used to have access to some A-Level maths papers going back to 1966 but that was over 15 years ago. Whilst I’m not convinced about the merits of your project it is disturbing that all that material may have vanished, probably in periodic cleaning-outs of cupboards and filing cabinets over the years. Which, in retrospect, lends a bit more validity to your project. Old test papers becoming mythologised in vauge anecdotes about the good old days without any grounding in their actual content is hardly a good basis for debate.

    Still I think there are better discussions to be having about the flaws and strengths in the current exams. For example the ability of the tests to distinguish between the top performing students is inadequate. Given that the marks are likely to follow a normal(ish) distribution if a hefty percentage are getting a top grade then that grade will represent a much wider range of achievment than other grades.
    Secondly rather than struggling with inadequate data to make comparisons that probably won’t hold up, a *better* argument to be had is what can be done now to exams so that meaningful comparisons of data can be made in the future.
    I apreciate that you weren’t claiming to have some watertight methodlogy to your project – but if there is anywhere that shouldn’t shy way from questions of methodology its a place like this. Its not just that electrosensitivity or homeopathy aren’t true in their claims it is also the underlying reasons they give for their claims. The sad fact is that education policy often rests, quite respectably, on the basis of claims with little more standing than those for alternative medicences et-al {in some cases the cross-over is overt as with fish-oils or the Dore claims for ‘cures’ for ADHD, dyslexia et al news.bbc.co.uk/2/hi/health/4715782.stm }

  126. NickConnolly said,

    August 27, 2007 at 3:34 am

    Ben,
    a good place to look for some better emprical data is from the people behind the Yelis tests at Durham University. They’ve been using a variety of equated tests that place students on a common scale since (I think) the early 1990’s. I don’t know of any data that goes back prior to that.

  127. NickConnolly said,

    August 27, 2007 at 3:36 am

    And on that note…
    www.cemcentre.org/documents/News/Are%20A%20levels%20and%20GCSEs%20getting%20easier%20(Wellington%2027.6.06).ppt#10

    Cuts to the chase so to speak. The tests are ‘easier’.

  128. plasecaha said,

    August 27, 2007 at 3:42 am

    I got all excited, clicked on the link, and it said “page not found”.

    :(

    I guess that kinda does cut to the chase, though…

  129. NickConnolly said,

    August 27, 2007 at 3:44 am

    The brackets broke the link. A different link with more explanation and a rather fun graph ranking GCSE subjects from “Harder” to “Easier”:
    www.cemcentre.org/Documents/News/subjectdifficultiesbyrasch.pdf

    They are using the Rasch model to equate tests.

  130. NickConnolly said,

    August 27, 2007 at 3:50 am

    Plasecaha, if you copy and paste the text it will take you to the right page (a PowerPoint presentation). The brackets in the URL broke the link so clicking on it won’t work (sorry). The second link is more technical but covers some of the same ground.
    Or you can head to the CEM centre website:
    www.cemcentre.org/RenderPage.asp?LinkID=10010000

    Note that they aren’t neccesarily correct but they are going about it in the right way.

  131. NickConnolly said,

    August 27, 2007 at 6:31 am

    Ben,
    Have you tried the British Library? You can search their catalogue here: catalogue.bl.uk/F/?func=file&file_name=login-bl-list

  132. NickConnolly said,

    August 27, 2007 at 6:35 am

    sadly…
    “Categories exempt from Legal Deposit regulations
    Publishers are not required to deposit the following categories of material unless a written demand for them is made by the British Library:

    Internal reports
    *Examination papers*
    Local transport timetables
    Appointment diaries; wall and desk calendars; posters ”
    www.bl.uk/about/policies/legaldeposit.html

  133. NickConnolly said,

    August 27, 2007 at 7:01 am

    …the University of London will have in its archives papers dating back pre-GCE when it published matriculation exams…

  134. richas said,

    August 27, 2007 at 10:01 am

    The difficulty of this problem, is one reason why I focused on the issue of accessibility. On whether or not a GCSE candidate could even reasonably have a go at the paper. If something is so much harder that it is obvious that candidates with GCSE-level sklls couldn’t even access it, it kinda absolves us of having to do more detailed comparisons.

    But this shows us nothing. There is a different skill in understanding all the elements that are required for a full answer with the broader question but this can be and was taught. When tackling the question candidaes had a good understanding of the elements required. Today an unprepared candidate would not but such preperation and practice would not take very long nor require a great leap in ability or knowledge. It is a bit like the IQ test – a bit of practice significantly increases your score but does not increase your intelligence or ability.

    Interestingly though IQ scores (using the same test) have been rising consistently by about 3% a decade. Now I don’t believe that a single generation can lead to a six to nine percent gain in intelligence. What it suggests is that tests, yes all tests, are imperfect measures of knowledge an abilit.

    The difficulty is no doubt why the QCA basically avoided the problem by simply declining to compare things in a paper which were significantly different. I don’t have a lot of time for the QCA because it is abundantly clear to me that as an organisation, it is not very “intelligent”.

    Whereas I have met some of them, even worked with them. My impression was that they are highly intelligent, professional individuals highly motivated to maintain and enforce standards and improve assessment. In short they were highly professional (if a little anal as many intellectual professional tend to be).

  135. Phil said,

    August 27, 2007 at 10:50 am

    I know for a fact that things are getting easier…

    I did my A-levels in 1992 and we used O-level papers from the 60’s & early 70’s as test papers………

    Phil

  136. RS said,

    August 27, 2007 at 11:36 am

    “When I was involved in teaching Media a few years ago, most students had difficulties properly writing an essay. It was just an incoherent collection of ideas. It seems rather like the CCTV example: it seems to be sufficient to just mention a few things… you don’t have to make any kind of real argument.”

    I believe that trend is partly to make the marking easier – even as a kid I used to get annoyed at the GCSE mark schemes that simply seemed to reward mentioning key words without any indication of understanding.

    Then again, if you have any expertise in an area I wouldn’t recommend looking at any GCSE or A Level syllabuses since they are invariably massively out of date and quite often wrong on important points.

  137. DTM said,

    August 27, 2007 at 1:28 pm

    Richas, you said:

    “Now I would no be surprised if the uni student on the 45th percentile has lower literacy and numeracy skills than the fifth percentile university student in the sixties but this tels us nothing at all about the performance of education in this country. Your anecdote just does not stack up. The change is not the exams it is a seismic shift in terms of educational attainment for the majority.”

    I would argue that my anecdote does stack up, because the anecdote relates to students at a top university, hence it is comparing the achievements of the top 5%. I weep for the chances of the rest of the nation who did not even achieve this standard.

    Secondly, what is the value of a “seismic shift in terms of educational attainment for the majority” if the majority STILL don’t have the basic skills required either at university or in industry?

  138. plasecaha said,

    August 27, 2007 at 1:59 pm

    RS

    I haven’t formally taught either O level or A level… if I had, I’d be able to more definitively state a few things, instead, I am having to try to work with whatever evidence we all have. Not very convincingly, it seems, partly because people seem unwilling to accept any differences that may have have been the case under the old system, because they haven’t experienced them (am I the only person here who did O levels? Anyone else care to back me up, lol), and maybe, find them hard to believe. So, they set the burden of proof very high, which is why we need the papers.

    And it is a site about Bad Science, so I guess we ought to be scientific. I don’t think we need to be so down on things like anecdote, however. It’s handy for the thing which the hard evidence of papers won’t give us: which is ideas for hypotheses and underlying causes and mechanisms, and for where to look for answers.

    I DO know what it was like under the old O and A levels, because I have done them, and I know what it’s like with BTEC, and I know what it’s like in Higher Ed., because I’ve taught in various different institutions.

    That’s where the Media thing came from – it was a Media degree – people coming to College having done A level, never mind GCSE level, and being unable to put any kind of argument together.

    When you see that sort of thing again and again, you cannot help but infer something about the education they have been through. It wasn’t their ability – because it was possible to fix the problem. They clearly hadn’t been taught how to do it, and they COULD have been. But equally clearly, there hadn’t been any NEED to teach them as far as exams went, because they had all seemed to do fine at A levels.

    It isn’t necessarily a trivial thing to teach someone how to properly write a good essay… most “guides” say things like “have an introduction and a summary”, give some stylistic points, and so on. Which is OK as far as it goes, but still… truth is, many teachers themselves don’t properly know how to do it these days.

    “I believe that trend is partly to make the marking easier – even as a kid I used to get annoyed at the GCSE mark schemes that simply seemed to reward mentioning key words without any indication of understanding.”

    Yes, fair enough, that’s definitely a big part of it too. It can be done, a lot of the time, but it’s not a trivial task, and even if you do it, it’s not necessarily going to be the case that all teachers are going to be able to do it.

    To properly assess an essay, the teacher is going to have to appraise the LOGIC of it. Have they covered ALL the salient issues, or have they left anything out that’s important? Have they properly justified a point, or have they just mentioned something in its favour. Have they left their argument open to attack? Have they claimed too much? Have they made any of the many different kinds of logical error? Have they properly used the evidence?

    Now, obviously, it’s a bit much to expect a GCSE level student to do the perfect essay. But at the moment, it seems as though they don’t even have to have a stab at it. And part of the reason for that is that, well, a lot of teachers these days wouldn’t necessarily spot a logical error, etc., so it’s probably a bit much to put them in an exam.

    I know this, because I once worked at a teacher-training college…

  139. RS said,

    August 27, 2007 at 2:15 pm

    To be fair to teachers quite a few academics seem happy to commit logical errors in their written work. Human condition I’m afraid.

    I believe they have an AS now called ‘Critical Thinking’ where they learn about that sort of stuff.

  140. plasecaha said,

    August 27, 2007 at 3:02 pm

    “But this shows us nothing. There is a different skill in understanding all the elements that are required for a full answer with the broader question but this can be and was taught. When tackling the question candidaes had a good understanding of the elements required. Today an unprepared candidate would not but such preperation and practice would not take very long nor require a great leap in ability or knowledge. It is a bit like the IQ test – a bit of practice significantly increases your score but does not increase your intelligence or ability.”

    You sound like some students I have come across. Or a bit like the “I can do that!!” character on the Catherine Tate show. One gets the impression these days that people seem to think all things are equally difficult, and they can just turn their hands to anything with minimal effort, and if it’s a struggle, someone just didn’t help them enough or make it easy enough.

    It’s not enough to say it’s just a “different skill”. Take calculus, for example. It’s not just a different skill, it is conceptually much harder than what usually comes before, and is actually a COLLECTION of skills. Of course, it can be taught, because it used to be. So why isn’t it taught now: you can’t say it’s because it lacks utility, because it is one of the most useful maths skills there is, and one of THE great breakthroughs in maths, and is one of the reasons that Newton is commonly regarded as our greatest scientist, indeed as anyone’s greatest scientist in the eyes of many.

    With something like fluency in languages, it’s not a case of the candidate realising what elements are required. It’s a different LEVEL of operation. You can teach a candidate a vocabulary, and grammar, and canned phrases, and this will not automatically mean they are fluent. Fluency is a different LEVEL of difficulty, because it demands the higher-order skill of INTEGRATING all these things.

    Obviously, this too can be taught, because it used to be. The point is: why isn’t it happening now? And the REASON it isn’t happening now is because it’s harder. That pretty much HAS to be the reason, because there is no excuse for it really. Calculus is so useful, it’s almost impossible to do physics-proper without it. And surely, THE principle aim in teaching a language is to achieve fluency. Nothing they are learning instead is of a comparable utility or skill level.

  141. plasecaha said,

    August 27, 2007 at 3:06 pm

    RS

    “To be fair to teachers quite a few academics seem happy to commit logical errors in their written work. Human condition I’m afraid.

    I believe they have an AS now called ‘Critical Thinking’ where they learn about that sort of stuff.”

    Yeah, that’s true… we all make mistakes. But if one is aware of a good approach, one can at least minimise the errors, and actually have a GO at constructing a proper argument. These days, many don’t even have a go.

    Funny, I was thinking about the critical skills A level as well. Saw a piece on it the other day… fastest growing subject apparently, although from a low base of around 2000. I need to find out what’s in it…

  142. plasecaha said,

    August 27, 2007 at 3:06 pm

    “Interestingly though IQ scores (using the same test) have been rising consistently by about 3% a decade. Now I don’t believe that a single generation can lead to a six to nine percent gain in intelligence. What it suggests is that tests, yes all tests, are imperfect measures of knowledge an abilit.”

    Well, I certainly think diet can have an impact here, as can one’s experiences. But equally, I know the evidence suggests that people CAN improve on IQ tests. I don’t know enough to be able to hazard a guess as to what’s really going on, but you certainly have a point there…

    Interestingly, as an aside… one’s intellectual peak is supposed to occur in one’s early twenties. To me, this suggests that in order to make the most of it, you have to make sure you are at the cutting edge of your discipline, up-to-speed, when you reach that age, in order to leverage your peak. Many famous geniuses in the maths and science worlds had their best ideas in their early-to-mid twenties… they may have LEVERAGED them later on, but the core of it happened earlier.

    (Newton, for example, had many of HIS breakthrough ideas in his early twenties when he was sent away during the plague, but didn’t work through them and publish the Principia till he was in his forties).

    What THIS means, is that to get up to speed you need to start the harder stuff earlier. I began calculus when I was fourteen, and this meant that by the first year of university I was already up to partial differentials and grads and curls, which made life a LOT easier in comparison with a number who had had a state school education in their teens, and who hadn’t been allowed to accelerate. But had I known then what I learned later, I would not have let the teachers dictate the pace of my learning. Truth is, the earlier the better for these things, and 14 is nowhere near early enough if you want more people to make the most of whatever they have.

    The geniuses, of course, start a LOT earlier… I remember reading not so long ago Wolfram mentioning reading a book on statistical mechanics aged 12. TWELVE!!

    I just have my doubts that had I started calculus later, it would have ultimately curtailed what I was capable of, and by the same token, that I and my contemporaries should have started that stuff even earlier. PARTLY to reach the peak further along, but also to RAISE the peak. Such things can improve one’s intellectual capabilities when done while the brain is still developing.

    Reading is a good example of how one’s learning can affect one’s intellectual capability, how nurture can enhance what nature gives you in intellect. In studies of the problem of people with reading difficulties, its found that up to the age of around seven, if poor readers are given remedial help, then they can recover to become good readers, AND with no measurable effect on their IQ. If the problem is allowed to persist beyond age seven, it DOES start having a measurable effect, about five IQ points I believe, because in not reading much, they are increasingly denied exposure to more and more advanced concepts.

  143. misterjohn said,

    August 27, 2007 at 5:12 pm

    The discussion seems to have gone a long way from reliability and validity, but so be it.
    I disagree with the comments about calculus, inter alia, as I used to teach O level Maths, and it was possible to coach for the O level questions, as they were always very similar to each other (London Board). Is it not a plausible hypothesis that the same similarities hold for GCSE questions, and that any teacher who wants to get her/his students through the exam will inevitably teach to the predictable? I certainly did so, for (too) many years, although one cannot guarantee that the students will pay any attention to your advice, or be able to learn the necessary information.
    On a separate point why do we have to bee (sic) logged in to comment? Why can’t we be logged in?
    Critical Thinking sounds like a good idea for an AS. My wife is an examiner for the subject, but it does appear to be one where the recollection of a few facts, use of phrases such as “ad hominem”, etc., gets the marks, whereas any genuine critical thinking runs the risk that it’s not what was wanted by the Principal Examiner. I’ve met her and she seemed very nice.
    I could say something about the hours of work put in by me and others to ensure the comprehensibility, reliability and validity of Edexcel A Level Mathematics papers, and to ensure that standards are maintained, but there are those who would not believe it.
    May well comment on some other points if this discussion continues.

  144. plasecaha said,

    August 27, 2007 at 6:35 pm

    My comments regarding the difficulty of calculus applied more to the A level. Yes, we did a bit of rudimentary calculus for the O level when I did it, but it was far from essential to be able to do it properly to be able to do the paper. You could quite easily get an A in the O level without really understanding calculus, as I know, because I didn’t really have much of a clue about it at all, and I still got an A.

    I took my maths exam in the same year I started doing calculus… it was more or less an afterthought in terms of the exam.

    In effect, it was such a small component, it didn’t really matter whether you could do it or not, echoing Nick’s point earlier that just the odd more-difficult question need not make a paper much more difficult in practice, if one can still get an A if one ignores it.

    VERY different at A level, however, where it was much more intrinsic, and in things like physics which depended on it a lot.

    I already accepted it was possible to teach calculus, as obviously it was taught in my day. Possibly, one could “coach” it, if by coaching you mean kinda drilling people in common examples of its use without them really understanding.

    At A level, you couldn’t really do that. You needed to understand what calculus was, and how to apply it, to decide how it should be used to answer questions.

    Sometimes, coaching would go on, and it was frequently a disaster. It’s fine when the exam remains simple and very predictable, as is the case it seems with modern exams.

    But it’s rubbish when things get more difficult, because you can’t adapt. I saw that happen to a bunch of my contemporaries, when we had a set of harder-than-they-expected papers at A-level that year, harder in the sense that it required the use of the knowledge they had been given in rather less straight-jacketed ways.

    Dependent for parts of their learning on rigid coaching, not really understanding properly what they had learnt, they couldn’t adapt, and didn’t do as well as was expected.

  145. Dr T said,

    August 27, 2007 at 8:00 pm

    This has been a fascinating discussion, serving to highlight how inappropriate it is for the press to speculate ever year on the easiness or otherwise of the exams. How could they tell?
    As an HE tutor who deals with the products of our education system, a couple of points have crossed my mind while reading this discussion:
    In the bad old days (1980s) when I sat my exams, the context in which the exams were sat was very different. The system had no qualms about saying to students that they had failed. Those who did fail, were likely to go into jobs (if there were any jobs, which were very very thin on the ground in 1984 when I left school, lest we forget just how awful it was back then!) where high levels of literacy and communication skills were probably not necessary. The main purpose of school exams was to differentiate among students and determine who was ‘good enough’ for college entrance, in the days when >10% of school leavers went to Uni.

    Today, the situation is very different it seems to me, and all the above factors have changed. There is a strong culture within schools of not branding children as failures, and I think on balance, that this is a good thing. Employment rates are much better, but the nature of those jobs is such that literacy and ICT etc, are much more important. Everybody, not just the academic ‘high achievers’ (what ever that means) need to be at least conversant with such skills in order to function in the workplace. The main purpose of the school exams therefore seems to have changed from discrimination at the top end, to testing for a particular lever of attainment, i.e. determining how many students have achieved a minimum. This certainly seems to be the emphasis in the early stages (KS1 & 2). The key thing here is that where the purpose of the test shifts from finding out the spread of marks at the tail of the distribution to fine resolution in the middle, I think that is *bound* to result in a larger number of subtly different, easier questions, because that is what you are testing for.

    The conflict arises because those outside the system still want to know what the rank order is at the top end, but it isn’t immediately clear to me how you do that, or indeed if it is desirable in the context of school. I guess what this discussion has made me think is that we need a serious debate on how we address these two very different problems.

    I could be barking up a nonsensical tree here, but it seems to me that the fascinating thing about comparing papers from 20 years ago with today is that it will reflect much more about society and what it expects from educators, than about education per se…

  146. NickConnolly said,

    August 27, 2007 at 9:05 pm

    “Not very convincingly, it seems, partly because people seem unwilling to accept any differences that may have have been the case under the old system, because they haven’t experienced them (am I the only person here who did O levels? Anyone else care to back me up, lol), and maybe, find them hard to believe. ”

    I did O-Levels. I’ve also taught GCSE and A Level mathematics and IB Higher mathematics and the numeracy compnents of GNVQ courses. In my experience there were certainly *some* harder questions on O-level maths than Higher tier GCSE maths. However there was also GCSE content not covered by O-Level, particularly statistics. I can’t tell from my experience whether standards have fallen or not – but I can’t rule it out based just on the experience.

  147. NickConnolly said,

    August 27, 2007 at 9:11 pm

    “if there were any jobs, which were very very thin on the ground in 1984 when I left school, lest we forget just how awful it was back then!”

    …and lets not forget that the GCSE was introduced by the Thatcher government at the height of its powers…

  148. plasecaha said,

    August 27, 2007 at 9:13 pm

    OK Nick. This is handy to know. Earlier someone said that people were going on to Uni. to do maths and science and stuff, and had only the most rudimentary algebraic skills.

    If this is true, then clearly they won’t have been doing calculus and stuff.

    If they ARE doing that stuff to the extent we used to, then I may as well ditch that example.

    So, are they doing it nowadays, or not, and to what extent?

  149. plasecaha said,

    August 27, 2007 at 9:14 pm

    BTW, obviously I’m talking about the A level here…

  150. Dr T said,

    August 27, 2007 at 9:49 pm

    Hi plasecaha,
    you can download the AQA syllabus from www.aqa.org.uk/qual/gceasa/mathematics.php

    I’m not a maths person, so not qualified to compare what was done before. HTH

  151. plasecaha said,

    August 27, 2007 at 10:12 pm

    I have just downloaded an OCR paper from a couple of years ago.

    I picked that as it’s the old O & C board, which used to be the hardest.

    Oh. My. God.

    I am still reeling.

    One example.

    Question 4:

    Solve the simultaneous equations x 2 − 3y + 11 = 0, 2x − y + 1 = 0. 5

    It’s barely O level standard, never mind A level.

    And it isn’t even the easiest question. That would be question 1, of course. Which is…

    … wait for it…

    Evaluate 100 to the power of 3/2.

    Oh yeah, there’s a bit of calculus in there. Calculus like….

    Find dy/dx in each of the following cases: (i) y = 1/2x(cubed) − 3x,

    This takes the concept of discriminating the floor to a new level. That’s some floor. Not that there really should be a need for a floor past GCSE…

    No wonder Nick didn’t want us to see the papers, lol.

  152. plasecaha said,

    August 27, 2007 at 10:26 pm

    It’s not like there is loads of new stuff we didn’t used to do to compensate either. No, there is more of the same…

    Question 8 – The length of a rectangular children’s playground is 10 m more than its width. The width of the playground is x metres. (i) The perimeter of the playground is greater than 64 m. Write down a linear inequality in x.[1]

  153. Dr T said,

    August 27, 2007 at 11:30 pm

    I remember 1989 well – where I was, the cohort size increased from 25 or so to nearly 70 if memeory serves…. and they didn’t treble the staff numbers!

    as to standards…

    “only these students naturally couldn’t perform at the usual levels, so since they couldn’t fail them without losing the income again, they did the only thing they could: drop the standards dramatically to suit.”

    I agree – the standards have changed to suit the wider range of abilities present among the student population. And that is inevitable if you treble the numbers. I would argue that using the phrase “drop the standards” is a harsh way to put it – on a good day I would say “widen the remit” instead… :-) . Even so, the really bright ones can and do still shine, so it is not all doom and gloom..

    But I make the same point here as I did above – we need to ask ourselves what high schools and Universities are for. And that is what should define the curriculum, and the exams that test this.

    I’ve yet to decide this for myself to be honest. On one day i will argue passionately for the HE sector to lighten up and broaden out to be more accessible to individuals who wouldn’t normally access it because it is such a privilege to have a time to learn and think and discuss in relative freedom, but on others I am hugely frustrated by the ‘conveyor belt-ness’ of it all, and long for a system with fewer students and space to think outside the box – I do genuinely worry that there just isn’t time to do that, and it is the initially maverick ideas that really change the world…

  154. plasecaha said,

    August 27, 2007 at 11:45 pm

    For one module, and that was a PRACTICAL module involving basic studio skills… I had 160 students to teach in 16 weeks. Each of whom did a report, and there were 40 five minute recordings to mark, in addition to assessing them in the studio.

    One module.

    I agree with you, the same thing has bothered me. The problem isn’t simply the fact that the standards are low when they come in, but that you don’t get TIME to do anything about it. You don’t have the time to do much with the teaching.

    The standard doesn’t matter if you have a bit of time to do something about it. But the conveyor belt denies you that opportunity. That’s the double-whammy: poorer intake, AND three times as many.

    What I used to do, was use that module above – which was an induction module – as an opportunity to let the students get to know what I was about and the way I worked, so the ones who were on the same page would select my modules in years two and three.

    Then in the first lecture, make it hard so those who just wanted a doss would realise it wasn’t a doss, would transfer to a more dossy module, and then others who really wanted to do it but who hadn’t been able to get on in the random lottery would take their place.

  155. plasecaha said,

    August 27, 2007 at 11:47 pm

    BTW, I’ve been checking out the AQA papers you suggested, and the difference appears to be considerable. Rather more the deal…

  156. NickConnolly said,

    August 27, 2007 at 11:58 pm

    “No wonder Nick didn’t want us to see the papers”

    Wow, you really need to get that psychic internet connection checked out again. You ability to read my my wants and desires seems to be on the blink again…

  157. plasecaha said,

    August 28, 2007 at 12:05 am

    Long as it’s not your ability to take a joke that’s on the blink, Nick.

    BTW, the number of maths papers they seem to take these days is… well, quite a lot. I mean, they’re shorter, but there’s a lot of ‘em…

  158. NickConnolly said,

    August 28, 2007 at 12:10 am

    “I have just downloaded an OCR paper from a couple of years ago. ”

    Which paper?
    I’m looking here:
    www.ocr.org.uk/Data/publications/specimen_assessment_materials/AS_A_Level25309.pdf

    The cubic polynomial x3+ax2+bx−6 is denoted by f(x) .
    (i) The remainder when f(x) is divided by (x −2) is equal to the remainder when f(x) is divided by
    (x +2). Show that b = −4. [3]
    (ii) Given also that (x −1) is a factor of f(x) , find the value of a. [2]
    (iii) With these values of a and b, express f(x) as a product of a linear factor and a quadratic factor. [3]
    (iv) Hence determine the number of real roots of the equation f(x) = 0 , explaining your reasoning. [3]

    Perhaps you may wish to post your answer Plasecha…
    Remember you have about 1.25 minutes per mark.

  159. plasecaha said,

    August 28, 2007 at 12:17 am

    The AQA paper still has some questions like

    solve the equation 4c(squared) + 15c -4 = 0

    But rather fewer, and rather more of the more challenging ones.

    The Further Pure paper is more the deal, but still has the odd noddy question on matrices.

    Overall, from what I’ve seen so far, I couldn’t really say the AQA papers were devastatingly below par. I don’t know a University lecturer need be overly disturbed by getting a student who’d done reasonably well on these papers.

    The mechanics papers don’t seem too bad either. There was even a ladder question reminiscent of the ones we used to do.

  160. Robert Carnegie said,

    August 28, 2007 at 12:30 am

    May I humbly suggest that terminating the education of students “who would not have got a place before” was perhaps the -problem-. The same as the 11-Plus. Selectivity, and the money spent per student “in real terms”, and maybe the introduction of modern media teaching (on computers and such), are the metrics that I’d look at first. The rest probably works out.

    As for applicability of old O-level tests to new A-levels, there probably is plenty of specialised material deleted from the O-level equivalent – replaced with other worthy stuff, I expect – that is retained in A-level. Science teaching isn’t immune to including material for reasons other than pure intellectual value; tradition, fashion, chauvinism. I believe that Euclid’s geometry was superseded by Descartes’ technically, but Descartes had to wait a couple of hundred years before getting into schools -alongside- the old Greek. Set theory also arrived far too late. And applied science… why in god’s name in 2007 is half of everything still named in Greek and Latin?!

    Incidentally, has my Bunsen burner shamed critics into silence, or has it been disqualified for doing the “research” on Wikipedia!! ;-)

  161. plasecaha said,

    August 28, 2007 at 12:39 am

    Nick, you dullard, the answers are in the mark scheme, including the working!! And I am not your performing sea-lion.

    BTW, the basic papers on the OCR exams seem dramatically easier to even the easiest AQA papers. The AQA lacks most of the noddy questions.

  162. plasecaha said,

    August 28, 2007 at 12:46 am

    This is the Pure Core 1 paper from AQA.

    www.aqa.org.uk/qual/gceasa/qp-ms/AQA-MPC1-W-QP-JUN06.PDF

    Surely that’s rather more challenging than the OCR Core 1 paper.

  163. richas said,

    August 28, 2007 at 12:46 am

    I would argue that my anecdote does stack up, because the anecdote relates to students at a top university, hence it is comparing the achievements of the top 5%. I weep for the chances of the rest of the nation who did not even achieve this standard.

    Secondly, what is the value of a “seismic shift in terms of educational attainment for the majority” if the majority STILL don’t have the basic skills required either at university or in industry?

    So how come this top university is dishing out a higher proportion of firsts and 2:1s? They must be equipped to do the course if teachers like you have any integrity.

    You are comparing 45% at uni to 5%, 55% with zero qualifications to 3% with zero. Of course the exams give a grade at a lower level but those achieving top grades are as they were before the top students. Please just go back to your work as a 19 year old and review it with the same critical eye.

  164. richas said,

    August 28, 2007 at 12:59 am

    You sound like some students I have come across. Or a bit like the “I can do that!!” character on the Catherine Tate show.

    Don’t watch it I’m afraid. I think you are referring to the “dropping” of calculus from the combined science GCSE. Now it would be lovely if a combined GCSE could cover all of the two combined but it can’t so they dropped the bit in the maths – whatever! Yeah but!

    What do you expect? It is not covering the same territory but is still a valid qualification.

  165. plasecaha said,

    August 28, 2007 at 1:01 am

    Robert,

    “May I humbly suggest that terminating the education of students “who would not have got a place before” was perhaps the -problem-. The same as the 11-Plus. Selectivity, and the money spent per student “in real terms”, and maybe the introduction of modern media teaching (on computers and such), are the metrics that I’d look at first. The rest probably works out.”

    There were definitely academics when I was studying who felt that the idea was to sort the wheat from the chaff, and that the chaff weren’t really up to it so shouldn’t really be indulged educationally.

    Part of the way they determined whether or not someone was up to it was their exam results, obviously, but also the idea that the best didn’t really need help.

    Thus, even if you could raise the standard of the rest by better teaching, it still didn’t matter, because in effect they needed more help and so weren’t in the same league.

    Not everyone who wants to see better standards subscribes to this view, however. I certainly don’t. I think standards can be raised for the majority through better teaching, and should be. However, for this to happen, quite a few things can be changed and we can’t currently blame the teachers too much given their lot.

    My current fear is that the increasing obsession with play-based learning in the infants is going to feed through into problems later for secondary teachers to pick up…

  166. richas said,

    August 28, 2007 at 1:11 am

    I could be barking up a nonsensical tree here, but it seems to me that the fascinating thing about comparing papers from 20 years ago with today is that it will reflect much more about society and what it expects from educators, than about education per se…

    QFT

  167. plasecaha said,

    August 28, 2007 at 1:17 am

    BTW, Nick, here’s that OCR paper I cited earlier…

    www.ocr.org.uk/Data/publications/past_question_papers_mark_schemes_and_materials/GCE%20Mathematics%20Question%20Paper%204721%20January%202005.pdf

    Although the situation is similar in the Core 1 part of the link you gave me…

  168. NickConnolly said,

    August 28, 2007 at 1:22 am

    “Nick, you dullard, the answers are in the mark scheme, including the working!! ”

    I trust you not to sneak a peak at the answers or get somebody else to do the question for you. I just tried it myself and I couldn’t full marks on that item in 15 minutes. But surely you won’t have any problems showing us all how easy A-Level maths now are by demonstrating how easy the questions are to answer.

    Up for a challenge? After all any fo..erm..person can declare a question to be easy or not challenging. It might be more enlightening for you to demonstrate how easy it is to answer A-Level mathematics questions.

  169. NickConnolly said,

    August 28, 2007 at 1:32 am

    “BTW, Nick, here’s that OCR paper I cited earlier…

    www.ocr.org.uk/Data/publications/past_question_papers_mark_schemes_and_materials/GCE%20Mathematics%20Question%20Paper%204721%20January%202005.pdf

    Seems fairly challenging to me for 1 non-calculator paper. Some neat little questions:

    The quadratic equation 2×2 +(p +1)x +8 = 0 has equal roots. Find the possible values of p.

    And remember that paper is just 1 of *SIX* units they have to complete to be awarded an A-Level in Mathematics.

  170. NickConnolly said,

    August 28, 2007 at 1:35 am

    ..and the Core 1 paper is designed to be taken after ONE year of their A-level course.

    I know I couldn’t have done that paper at the end of my first year of A level back in ’83. I eventually got a grade A.

  171. NickConnolly said,

    August 28, 2007 at 2:59 am

    “I specifically declared the questions on a PARTICULAR paper to be not-challenging. ”

    Well here you go:
    The quadratic equation 2×2 +(p +1)x +8 = 0 has equal roots. Find the possible values of p.

    Seems challenging enough to me. I’d be surprised if you’ve memorised the answer so please demonstrate how easy the question is by showing us how easily you can solve it.

    “Or those simultaneous equations???”
    I think somebody’s ignorance is showing. The simultanous equation you quoted involved factorising a quadratic – not the hardest skill at A-Level but hardly trivial. I certainly didn’t have to solve simultaneous equations where one was quadratic and the other linear at O-Level. I’d think a competent person would probably lose a couple of marks unless they were careful. I can do questions like that very rapidly but I had to correct at least one error as I did it.

    I’m quite convinced from your general response that you would have trouble gaining a decent mark on that paper in the time given. Feel free to demonstrate otherwise.

  172. plasecaha said,

    August 28, 2007 at 3:14 am

    Nick, is this a debate about standards or is it your desperate attempt to have a go at me because I blew apart your argument about grade boundaries, showing that you couldn’t dismiss the whole endeavour of comparing papers on the back of them?

    Or is it because you are embarrassed when I pointed out you were talking nonsense in saying the examiner had focused on the foundation question, when in FACT she wrote more on the next two questions in the paper. Three sentences, versus 4 and then 6.

    You bailed on that and I’m not surprised. Clearly you don’t mind spending your time giving me games to play. I am not here to dance to your tune, Nick.

    But just so we can put this to bed, I know about things like the Polynomial Remainder Theorem, Nick, OK? They didn’t even mention it in the marking scheme, and it’s the secret to answering the question.

    The theorem says that if you divide a polynomial by (x – r), then the cemainder is given by f(r).

    Since in this case, we are dividing by (x – 2), then we have to work out f(2), in other words substitute 2 for x, which gives us 8 + 4a + 2b – 6.

    Then we do the same for (x + 2), and then, because they say the remainders are equal, we equate them.

    Which gives us…

    8 + 4a + 2b – 6 = -8 + 4a – 2b – 6

    which is the point at which the marking guide kicks in. It doesn’t give any of what I described above.

    4a cancels, and then you solve for b.

  173. plasecaha said,

    August 28, 2007 at 3:26 am

    and surely factoring a quadratic was routine at O level?

  174. plasecaha said,

    August 28, 2007 at 3:27 am

    and as for 100 to the power of 3/2…

  175. NickConnolly said,

    August 28, 2007 at 3:31 am

    “But just so we can put this to bed, I know about things like the Polynomial Remainder Theorem, Nick, OK? ”

    Well now we know how long it takes you to look something up on Wikipedia…
    You had noted the challenge by at least 2.02 am and you posted an answer over 1 hour later. The test paper is 1 hour and 30 minutes long… Somehow I don’t think you’d score very well.

    I think its pretty clear that these supposedly trivial A-Level questions are more than challenging for you, no?

  176. plasecaha said,

    August 28, 2007 at 3:36 am

    See, this is the futility of your challenge.

    Nick, I would have needed to know…
    a) That one could solve the problem with a theorem, and that such an arcane theorem existed
    c) And the name of it

    in order to look it up, and if I knew all that already, I wouldn’t need to look it up. It’s not exactly elaborate. PROVING it is trickier than using it.

    Jeez man, you redefine desperation.

  177. NickConnolly said,

    August 28, 2007 at 3:43 am

    “and surely factoring a quadratic was routine at O level?”

    Factorising a quadratic was a skill tested at O-Level and quadratics are still in GCSE higher teir maths syllabi. Factorising a quadratic as a part step to solving a pair of simultaneous equations is less than routine.

    “and as for 100 to the power of 3/2…”
    What about it? Was that the ONLY question on the paper? Why bother with such silliness. You’ve already acknowledged how stupid it is to pick on the easiest question in a paper and use that as the benchmark for judging how easy it is. Yet here you are again repeating the same moronic fallacy. Just like comparing a foundation tier GCSE question in French with an O-Level question. Talk about falling standards…

  178. NickConnolly said,

    August 28, 2007 at 3:45 am

    “Nick, I would have needed to know…
    a) That one could solve the problem with a theorem, and that such an arcane theorem existed
    c) And the name of it

    in order to look it up,”

    Really? Gosh apparently you’d fail GCSE research skills as well. I think most educated people could find relevant information without knowing in advance what exactly they were looking for. Odd that you claim you can’t.

  179. plasecaha said,

    August 28, 2007 at 3:52 am

    “Really? Gosh apparently you’d fail GCSE research skills as well. I think most educated people could find relevant information without knowing in advance what exactly they were looking for. Odd that you claim you can’t.”

    OK, in your desperation you have become a troll…

  180. NickConnolly said,

    August 28, 2007 at 4:03 am

    “My point here is that… that paper is way too much floor. ”

    Which you back with vague anecdotes and questions taken out of context. Your way too much floor for A-Level maths added up to one question, on one AS paper, a question with fractional powers.

    “I cited SEVERAL trivial questions.”

    You declared several questions as being trivial. You obviously didn’t grasp the simultaneous question for example – revealing in itself. You couldn’t actually demonstrate that you could do any of the questions in the time given.

    “OK, in your desperation you have become a troll…”

    Yet another quality argument…

  181. plasecaha said,

    August 28, 2007 at 4:22 am

    “Which you back with vague anecdotes and questions taken out of context. Your way too much floor for A-Level maths added up to one question, on one AS paper, a question with fractional powers.”

    I cited FOUR. You disputed one, and there are many more in that paper.

    “Yet another quality argument…”

    It wasn’t an argument. It was an observation. You asked me to show I could do it, and I went out of my way to pick something that wasn’t in the marking guide and couldn’t easily be looked up unless you already knew what you were looking for.

    There isn’t any more I could have done. If you were always going to say I looked it up, what’s the point of asking me to do it other than to be a troll? But hey, if believing I can’t do maths makes you feel better Nick…

    Now, can we get back to the debate, instead of all this ad hominem nonsense?

  182. plasecaha said,

    August 28, 2007 at 4:39 am

    As for factoring quadratics in the simultaneous equation, you make it sound like rocket science…

    Here’s the one from the paper you gave me…

    y = x(squared) – 3x + 2 … and…

    y = 3x – 7

    hence, x(squared) – 3x + 2 = 3x – 7

    x(squared) – 6x + 9 = 0

    (x – 3)(x – 3) = 0

    Come on Nick, that’s about as easy as it gets…

  183. NickConnolly said,

    August 28, 2007 at 5:09 am

    “As for factoring quadratics in the simultaneous equation, you make it sound like rocket science…”

    No I make it sound like a non-trivial task suitable for an A-Level paper. Again simply you declaring things trivial doesn’t make it so.

    “Here’s the one from the paper you gave me…
    y = x(squared) – 3x + 2 … and…
    y = 3x – 7
    hence, x(squared) – 3x + 2 = 3x – 7
    x(squared) – 6x + 9 = 0
    (x – 3)(x – 3) = 0″

    Which gives you a mark of 3 out of 5 according to the mark scheme. So you get 60% of a question you declare “as easy as it gets”…

  184. plasecaha said,

    August 28, 2007 at 5:11 am

    But it’s not hard though, is it Nick. But wait, there’s more…

    Meanwhile, the perimeter question… oooh, how hard can it be?

    The width, is x.

    The length… is 10 metres more… ooooh… what could it be?

    *wracks brain, looks it up on the net*

    I know, I know!! x + 10

    Now, how do we work out a perimeter? Could it be… twice the length plus twice the width?

    *draws a diagram*

    Yes, that must be it.

    so thats… 2x plus 2(x + 10)

    Gosh, this is getting tricky.

    Now what? Oh yeah, the perimeter is more than 64 metres, so…

    64 11

    Wow. Nick’s so right. He thinks factoring a quadratic is hard, when it isn’t, so this question must be hard too.

    Remember, this is for A level students, people…

  185. plasecaha said,

    August 28, 2007 at 5:14 am

    argh, it didn’t take the inequality sign.

    64 is less than 4x + 20

    or x is greater than 11

  186. plasecaha said,

    August 28, 2007 at 5:19 am

    Then there’s the 100 to the power 3/2

    And there are plenty more questions like that.

    Sketch the curve y = 1/x

    Wow, we never did anything like that for O level. Really hard.

    Find the gradient of the curve

    y = 2x(squared) at the point where x = 3.

    How easy does it have to be, Nick?

  187. plasecaha said,

    August 28, 2007 at 5:28 am

    Which gives you a mark of 3 out of 5 according to the mark scheme. So you get 60% of a question you declare “as easy as it gets”…

    What? Because I couldn’t be arsed doing the easiest part of all, substituting to get the value for y?

    I was pointing out that the QUADRATIC aspect you kept whinging about was, in fact, a doddle.

    One simple step.

  188. plasecaha said,

    August 28, 2007 at 5:28 am

    you take desperate pedantry to a new level, Nick…

  189. NickConnolly said,

    August 28, 2007 at 6:06 am

    “you take desperate pedantry to a new level, Nick…”

    Yet another stunningly brilliant argument.

    “I was pointing out that the QUADRATIC aspect you kept whinging about was, in fact, a doddle.”

    Took you long enough and only *AFTER* I pointed out it was a quadratic and STILL you didn’t even manage full marks.

    “Wow. Nick’s so right. He thinks factoring a quadratic is hard, when it isn’t, so this question must be hard too.”

    [snore…]

  190. plasecaha said,

    August 28, 2007 at 6:28 am

    “Took you long enough and only *AFTER* I pointed out it was a quadratic and STILL you didn’t even manage full marks.”

    Wait, did you just cede a point there??

    *double-takes*

    No, I KNEW it was a quadratic, why would I mention it? I didn’t imagine you could possibly fool yourself into imagining it added any significant difficulty. Or fool anyone else into believing it, for that matter.

    “[snore…]”

    Good man…

  191. NickConnolly said,

    August 28, 2007 at 7:10 am

    “No, I KNEW it was a quadratic, why would I mention it?”

    Good point. If you mentioned it then your comment that it was “barely O-Level standard” would have looked pretty stupid. So yes, I should concede that you had every incentive NOT to mention it if you were making a half-ars*d attempt to mislead people – but then that would be pretty stupid also. Parsimony means I should assume, rather than you not mentioning it because you were both attempting to mislead people AND doing so incompetently, that you simply hadn’t noticed. Of course if you’d like to argue for the incompetently misleading hypothesis I shan’t dispute it.

  192. plasecaha said,

    August 28, 2007 at 7:15 am

    OK, here’s a funny thing.

    I just checked an O level paper from Nick’s era, the 80s, and guess what? You didn’t have to solve any quadratic equations.

    Now, I knew I had had to do that, so I looked at an older paper, and sure enough, back then, you DID have to solve them for the O level.

    Thus, you can understand why Nick may consider solving quadratics more advanced than O level, while I wouldn’t.

    Interestingly, there’s a perimeter question in the older O level paper, too:

    “Two rectangular rooms each have an area of 240 sq. ft.

    If the length of one of the rooms is x ft. and the other room is 4ft. longer, write down the width of each room in terms of x.

    If the widths of the rooms differ by 3 ft., form an equation in x and show that it reduces to x(squared) + 4x – 320 = 0.

    Solve this equation and hence find the difference between the perimeters of the rooms.”

    …and note that it requires the solving of a quadratic. It’s harder than the perimeter question in the A level paper.

    And never mind dumbing down for the GCSE, the O level was dumbing down already by itself before the GCSE came along.

  193. plasecaha said,

    August 28, 2007 at 7:20 am

    “If you mentioned it then your comment that it was “barely O-Level standard” would have looked pretty stupid.”

    No, because in MY day, we DID do quadratics in the O level.

    Nick, you whinge whenever you think I’m questioning your motives. How about you don’t whinge about mine? It’s an irrelevance. I haven’t gone on about you having a vested interest as a maths teacher, have I?

    Whatever you think of my motives, there are a goodly number of easy questions on that paper, at least compared to the O level in MY day. In your day, I can now see, maybe not so much.

  194. plasecaha said,

    August 28, 2007 at 8:21 am

    Yeah, the more we look, the more it seems the exam boards are a big issue.

    On the older O level paper I just looked at, there was an absence of a few things I could have SWORN we did in our O level. But then, I did the O & C exam, which at the time was reputedly the hardest.

    I don’t know what board the old paper is I’ve been citing above.

    Regarding the modern exams, having seen the AQA papers, they seem pretty reasonable to me. Whereas the OCR paper… yes, the papers do get harder, but that first paper we’ve been talking about… God. There’s nothing as easy in the AQA paper that I could see. It’s quite the difference…

    Did you do any calculus at all for the O level Andy? Anything at all?

  195. richas said,

    August 28, 2007 at 11:30 am

    Plasecha – The O&C was “reputedly” the hardest. You seem to have adopted the modern anecdotal myth of falling standards and accepted the older anecdotal myth of differentiated exam boards.

    There was some differential but not in difficulty. The Oxford and Cambridge board’s market was largely the Grammar and private school sector. They wanted to pretend that their qualifications were of a higher standard valued by certain universities. It was branding

  196. richas said,

    August 28, 2007 at 11:42 am

    Ben – having read this thread do you think it is now time to give up on the idea that just by comparing exam papers from different eras you can assess the validity, reliability and difficulty of the exams compared to each other?

    When looking at an exam system you need to look at how the whole education system is working. In the golden era over half left with nothing now 3% do. In the current system 50% take A levels in the golden era it was about 12%. In the golden era of norm referencing you knew that however well the teachers and students did a large proportion had to fail. Today 3% get nothing for their efforts and students that struggled get a low grade, useful for employers or for considering further study.

    Meanwhile 3% get 3 As. Now this is a pretty small group. We have the differentiation we need and if top employers and universities cannot take this elite group and make a success of it this is a condemnation of them not the exam system.

    The education (and exam) system works much beter today, for many more, it no longer leaves more than half with nothing to show for their education and it produces fewer illiterates than we did in the past. Good. Now we need to look at how the tail of underachievement can be further reduced and the new diplomas seem a worthwhile attempt at this in rejiging the education system to meet those pupils needs.

    If you want to assess how well the exam system works wait for the next set of PISA data comparing us to the rest of the OECD. If we do well (we will) it shows we have a decent education (and exam) system compared to a sensible set of comparators.

  197. ithaca99 said,

    August 28, 2007 at 12:40 pm

    (After the parade…) I think Ben’s idea is a good one. If we were reasoning a priori, there are certain ways we might expect science papers to change, bearing in mind the ways science itself changes.

    A. We have only to think of the Copernican Revolution to see that breakthroughs in understanding can often make a scientific subject much EASIER. Ptolemaic astronomy was INCREDIBLY COMPLICATED because based on a false premise. We might expect advances in scientific research to produce better models of physical phenomena, and to see these reflected in more recent exam papers.

    B. It’s now possible to buy a CD-ROM and drill oneself on (say) the mathematical tools of the trade in a way that no teacher could do without going insane, getting instant, accurate feedback every time. So one can practice trigonometric identities, factoring, the chain rule, till one can do them in one’s sleep. In other words, one might expect questions covering these techniques to be perceived as the easy questions — the quick couple of points.

    My understanding is that many teaching in universities are demoralised because students don’t, in fact, come up equipped with a good mathematical toolbox. If so, we would expect the marking schemes of exam papers to show perceived difficulty of such techniques.

  198. plasecaha said,

    August 28, 2007 at 12:47 pm

    Richas…

    I say “reputedly” precisely because I am admitting I cannot be categorical on the matter. However, a couple of points…

    1) I went to a top-ten, well-known school and a quarter of us got into Oxford or Cambridge every year

    2) So we really didn’t need the branding. Actually, as a result, the school was rather progressive in some ways, adopting Nuffield Physics, and switching from the O & C to the JMB board for Chemistry, because while the latter was easier, they much preferred the syllabus. And I saw a comparison between those papers, and it WAS easier.

    In any event, there’s certainly differentiation now. The difference between the OCR and AQA is stark.

    We’ve also established that the old O level contained questions which are almost identical to questions currently in the Core 1 A level exam of the OCR, and that the OCR contains the odd question that’s too simple even to be on the old O level paper.

    That isn’t something you can just sweep under the carpet. The AQA, however, is much more the deal compared to the OCR. Much closer to the old A level, and quite different to the OCR.

  199. plasecaha said,

    August 28, 2007 at 12:48 pm

    We’ve also established that Nick thinks quadratic equations are scary.

    BTW, I wrote this earlier regarding the QCA thing, and forgot to post it:

    Richas,

    “Whereas I have met some of them, even worked with them. My impression was that they are highly intelligent, professional individuals highly motivated to maintain and enforce standards and improve assessment. In short they were highly professional (if a little anal as many intellectual professional tend to be).”

    You’re being bit trigger-happy… overly keen to leap on some flaw that isn’t there. I didn’t say that the PEOPLE at the QCA were lacking intelligence, did I? No, I said “as an organisation” it’s not very intelligent. You’re supposed to realise at that point that I am criticising the organisation, rather than the people individually, because I made a point of mentioning the organisation, and not the individuals.

    The point is, that it is possible to have very clever people working together in a system that’s dumb, that makes the output not very good, even if the contributors are capable of more.

    Let me give you an example. Asking teachers to come up with all these assessments is an onerous task, and it can be quite difficult to come up with tests and activities over and over again which properly test the mastery of the objectives. Then they have to analyse the data, and many struggled to get their GCSE maths. It does not take a rocket scientist to figure out they are going to have trouble with this, but the QCA did nothing for ages.

    Now, it may have been that there were some intelligent people there who understood this, and even wanted to do something about it, but were constrained by others or the system or whatever. Whatever reason, the RESULT was crap. But I have to say, whoever came up with that exam comparison system could do with some remedial work though…

  200. richas said,

    August 28, 2007 at 1:15 pm

    Let me give you an example. Asking teachers to come up with all these assessments is an onerous task, and it can be quite difficult to come up with tests and activities over and over again which properly test the mastery of the objectives. Then they have to analyse the data, and many struggled to get their GCSE maths. It does not take a rocket scientist to figure out they are going to have trouble with this, but the QCA did nothing for ages.

    You are saying here that the delay was a bad outcome. In fact this time was spent in a great deal of consultation within the education community and a great deal of refinement of the materials because they insisted upon a very high standard.

    There were and are real concerns about teaching to the test and the potential of such materials inadvertently narrowing the curriculum or teaching overall, that they could have restricted the freedom of teachers to address the curriculum in the way they could best achieve results. This meant that a large amount of materials, thoroughly checked and reviewed were needed if this approach were to be used. They did this. Yes it took time but it took time because of their thoroughness, because of the natural conservatism of a regulatory authority and precisely to avoid the possibility that someone convinced of a dumbing down hypothesis would have that hypothesis supported by the QCA teaching and learning cribs.

    They may be a bit slow sometimes but they are thorough and commited to maintaining standards, partly by employing very intelligent and experienced people. A slow, cautious organisation is not a dumb organisation.

  201. plasecaha said,

    August 28, 2007 at 1:36 pm

    Richas

    One gets the feeling that if somehow one doesn’t agree with you on these things, that they are somehow unconcerned about the education of the majority of pupils, or unaware that some of the people working in education are keen to improve things for the benefit of all.

    Just because I may disagree on standards, does not mean I somehow don’t care about the majority, that I want exams to be harder to pick out the best in the way we used to in a norm-referenced system. Nothing could be further from the truth. I want standards raised because I know the majority can be served better.

    Similarly, just because some of the architects of our educational system mean well, does not mean they are doing things right. It’s all very well them taking as much time as they want trying to “get things right” but in the meantime my gf, who is a deputy head along with the other half-million teachers in the system has been labouring for years to make something of the half-baked system they have had foisted upon them.

    A couple of years ago, there had already been 670 educational initiatives since Labour came to power. 670 new things for Heads and staff to accommodate. Some of which involve assessment. To be honest I’m sick and tired of having to explain to friends of mine in education how to handle the data, having to work out a way of dealing with the ludicrous discontinuity between the Foundation Stage and Year One – the levels don’t match, and yet teachers are supposed to track progress.

    An utter idiot designed that, to be sure.

    I could write for hours on the litany of avoidable errors, but to deal with the point of the assessments, It’s all very well trialling the assessment materials to get them right. But the time to do this should have been BEFORE introducing the obligation to assess more frequently. Not years later, after the teachers have had to figure it out for themselves. You can tell your friends that next time you see them. While they take years over it, teachers are having to do it for themselves whilst also teaching.

    It’d better be good, but from what I’ve seen of them so far…

  202. killary45 said,

    August 28, 2007 at 2:29 pm

    I am really amazed that Richas feels able to sustain the argument that the standards at A-level have not been made a great deal easier. Does he really believe that in every school in the country teachers are so much better than they were, or that the pupils are so much more hard working and able than they were? A generation ago 3 grade As was a high achievement at Eton – now it is just about the norm there. Can he tell us whether it is the Eton masters who are so much better than they were, or is it the pupils who are more dedicated to their studies?

    In A-Level Maths the questions now are generally much more predictable than they were, and they are broken down into easy steps to guide the student to the answer.

    It is now possible for a student of barely average ability to gain a grade A in Further Maths. This was not possible in the past because the exam used to contain unpredictable questions requiring genuine mathematical ability or insight. Now, as well as the exam being split into 6 separately examined units, the questions merely require the student to have worked through the examples in the board-specific text book and repeat the methods learned. This means that some students with A grades in Maths and Further Maths go on to start Maths degrees without having the aptitude for degree level work in that subject. Also the more able students are not being sufficiently challenged when in the sixth form, which is a serious detriment to their development as mathematicians.

    My nephew achieved 3 grade As (Maths, Economics and French), but has just failed his first year Law at Southampton University, when he found that he simply could not cope with the standard of work required. It was a very unhappy time for a conscientious student, and it was the result of A-levels being too easy. As a hard working lad with average academic ability he found A-levels completely within his grasp. He had not been prepared for the totally different experience of degree level study at a top university.

    Of course there was no golden age in education. But to suggest that just because pupils now leave with lots of GCSEs they must be doing much better than those in the past is simply false. That would only be true if the exams had maintained their standards. Plenty of pupils left school without any certificates, but after gaining a better grounding in English and Maths, than some students with lots of GCSEs receive now. It is a cruel trick to give pupils a very poor quality education (which exists in the lower sets of many schools today because of the appalling standards of behaviour) and then to award them a set of GCSE certificates and claim they are proof that the school has done a good job

  203. plasecaha said,

    August 28, 2007 at 3:28 pm

    Richas, in common with a lot of others, doesn’t understand the maths.

    He thinks like this:

    1) We used to have a norm-referenced system of awarding grades

    2) This guaranteed that only ten percent or whatever could ever get the highest grade, no matter what happened in the exams

    3) Therefore, if we get rid of the norm-referencing, we allow more people to score the higher grade.

    4) More people are scoring the higher grade, therefore it must have been the norm-referencing holding them back.

    I don’t think he can see the flaw in this…

  204. plasecaha said,

    August 28, 2007 at 3:45 pm

    Good post, btw Killary. Something else folk like Richas don’t get – and I have tried – is that structuring the tests, breaking them down into bite-size chunks can make them considerably easier.

    I mean, you can TELL him, and you can explain WHY… but somehow he seems unable to see the difference between telling someone:

    a) here’s a problem, solve it, and

    b) here’s a problem, so work on this, and when you’ve done that, you have to figure this bit out, and once that’s done, the next challenge is this, and so on…

    To Richas, there’s no difference between telling someone to go build a bridge across the Thames, and between telling them – OK, you need to do a survey, go work on that, and you need to figure out where to build it, and what design to use… and so on.

  205. killary45 said,

    August 28, 2007 at 4:16 pm

    Incidentally the mean performance now at Eton is 4 grade A A-levels. A generation ago only very exceptional students there achieved that level. Which is the most likely hypothesis:
    1) The masters at Eton (and every other school) are doing a far better job than they did, and all pupils are now as good as the small minority were in the past.
    or:
    2) It is now very much easier to gain a grade A.

    I could see how a supporter of the National Curriculum and other reforms could genuinely believe that the better results in maintained schools were caused by those reforms. But how do they explain the incredible improvement at Eton? Thirty years ago it was highly selective both on academic ability and parental income. The highest quality possible staff were recruited from the best universities. Was everyone there in those days so much weaker in every subject than pupils at the school now?

    The change from a quota system took place when Sir Keith Joseph was education minister. He was a hard-line right-winger who worried that the ever increasing number of entries for A-level, which meant that the number of grade As had increased five fold over the previous twenty years, was likely to mean that standards had been diluted. He asked the examination boards to stop giving As to 10% of the entries, but instead to fix a gold standard and only give the A grade to those who truly deserved it. Of course once boards could award as many As as they wanted it was in everyone’s interest to give more and more every year.

  206. richas said,

    August 28, 2007 at 5:53 pm

    Similarly, just because some of the architects of our educational system mean well, does not mean they are doing things right. It’s all very well them taking as much time as they want trying to “get things right” but in the meantime my gf, who is a deputy head along with the other half-million teachers in the system has been labouring for years to make something of the half-baked system they have had foisted upon them.

    I think you mean that the teaching professionals were asked to teach a detailed curriculum. To devise their lessons, to share best practice, essentially to do their (now much better paid) job.

    Meanwhile the QCA was trialing materials, looking at the materials developed by the TEACHING professionals before disseminating the best (and proven) materials for all to use as they think best.

  207. RS said,

    August 28, 2007 at 6:03 pm

    killary45 – if it is the exams getting easier per se that causes the rise – why is the rise twice as large in the private sector as the state sector? How can you explain that without additional school specific factors?

  208. RS said,

    August 28, 2007 at 6:20 pm

    “More people are scoring the higher grade, therefore it must have been the norm-referencing holding them back.”

    Well to some extent that does have to be true (or, as I mentioned above, v.v. with norm referencing preventing criterion based decreases in grades) because norm-referencing cannot allow improved grades.

    I believe one of the problems seen in norm referencing was that the courses were actually quite poor at discriminating – resulting in grade boundaries that were implausibly narrow so the difference between a D and a B could be only 8 marks (1982).

    Pass rates and A grades have been increasing since criterion referencing was introduced in ?1987 for A Levels – and you would expect some increase as teachers learn to teach to the test and gain experience of the examination. So the real question is what rate we would expect grades to inflate if exams were not getting easier?

  209. Geoff_S said,

    August 28, 2007 at 6:34 pm

    I have no idea how hard current school exams are – it’s over 50 years since I took my ‘O’ levels when I attended a state grammar school.

    However, I took and passed 7 subjects including 2 sciences, maths, English and French – I had to choose between biology and geography and my love of maps won :) Despite that, I was not considered bright enough to take ‘A’ levels and so continued my post 16 education at night school and occasional day release. As a huge percentage of youngsters now go on to take ‘A’ levels and from thence University things must have changed somewhere. Either teachers are better, pupils brighter or standards reduced.

  210. richas said,

    August 28, 2007 at 6:39 pm

    A generation ago 3 grade As was a high achievement at Eton – now it is just about the norm there. Can he tell us whether it is the Eton masters who are so much better than they were, or is it the pupils who are more dedicated to their studies?

    By recent accounts they seem very good at helping prices get their second C (in art) to get into Sandhurst.

    Eton has always excelled. Yes three As were rarer when sixty thousand students took A levels, now it is about 300,000. I have no doubt that Eton performed well with a norm referenced exam and that they excel at ensuring their students meet the standard in the standard based tests where if the pupil demonstrates their achievement the number of A grades is not artificially capped.

    It is now possible for a student of barely average ability to gain a grade A in Further Maths.

    This year about 8,000 pupils took further maths. (Up from six thousand three years ago). Of these 56.8% got a grade A (down 0.1% on last year). So about 4,500 Further maths A grade A levels from an age cohort of over six hundred thousand. ess than one percent. Now I am sure this could be pushed up a bit if more institutions taught further maths but 0.75% getting A grade further maths is not a group that is likely to be of “barely average ability”.

    This means that some students with A grades in Maths and Further Maths go on to start Maths degrees without having the aptitude for degree level work in that subject. Also the more able students are not being sufficiently challenged when in the sixth form, which is a serious detriment to their development as mathematicians.

    Yes only 91.6% of first year students make it to their second year. This is a much lower drop out rate than the US where half drop out. Maths has always seen a slightly higher drop out rate but we should remember that this drop out rate includes students changing courses.

  211. richas said,

    August 28, 2007 at 8:31 pm

    Richas, in common with a lot of others, doesn’t understand the maths.

    He thinks like this:

    1) We used to have a norm-referenced system of awarding grades

    2) This guaranteed that only ten percent or whatever could ever get the highest grade, no matter what happened in the exams

    3) Therefore, if we get rid of the norm-referencing, we allow more people to score the higher grade.

    4) More people are scoring the higher grade, therefore it must have been the norm-referencing holding them back.

    I don’t think he can see the flaw in this…

    Wow that pshychic Internet connection is really playing up now.

    I understand the maths thanks. I also understand that that achieving a standard is not constrained to a given percentage. You either achieve that standard or you do not. You achieve an A, B, C based on your demonstration of ability in the curriculum.

    The maths of norm referencing is easy X% fail, Y% pass, Z% get an A. This system has flaws. To start with it guarantees failure and the waste of pupils and teachers time. If you improve teaching and learning you get the same results, the same proportion failed.

    Your misconception, what you fail to grasp is that norm referencing by falsing limiting the maximum number of As is only making them more scarce, not more valuable. There is no mechanism no driver for improvement. We know the Eton boys with the best teachers and the best background will come out on top even when state education does a better job. Norm referencing is all about ranking, it is not about validity or reliability over time, it expressly fails in this, it is unable to accomodate better (or worse) results, it is the dumb quota of sheeps and goats, frankly an assessment disaster for a society like ours today.

    The current GCSEs and A Levels manage to rank almost every pupil. Amost all of them have a range of qualification showing their personal achievement. The norm reference system would (and did) guarantee failure, it would (and did) completely fail to give any sort of assessment to the majority of people.

    Do you see the flaw in an exam system that assumes that regardless what we do 20% fail? Where if pupils learn more or work harder (as they undoubtedly do today compared to my time at sixth form), where if teachers teach better and teaching materials and equipment improve we still give the same fail rate and the same A rate? A laughable inadequate system.

    Therefore, if we get rid of the norm-referencing, we allow more people to score the higher grade.

    No, where if we get rid of the norm referencing we can count those who achieve the standard for each grade. Good.

  212. richas said,

    August 28, 2007 at 8:42 pm

    I could see how a supporter of the National Curriculum and other reforms could genuinely believe that the better results in maintained schools were caused by those reforms. But how do they explain the incredible improvement at Eton? Thirty years ago it was highly selective both on academic ability and parental income. The highest quality possible staff were recruited from the best universities. Was everyone there in those days so much weaker in every subject than pupils at the school now?

    The task for the excellent teachers (in small classes, teaching selected and priviledged pupils) has been changed somewhat.

    Then their task was to make sure that their kids performed as high in the ranking as possible. They were good at it but the upstats from elsewhere (Harrow or Comp) could still pip them for the limited number of As.

    Now their task is to make sure that their kids at least achieve the A standard as outlined in the curriculum. Frankly given the selection, the economic and social background, the expert teaching and the small class sizes it would be a pretty daming indictment if any of their pupils (Harry?) failed to make it into the top 20% that get As.

  213. RS said,

    August 28, 2007 at 8:51 pm

    Further Maths is a bit of a funny subject – essentially it is redundant if you are taking Maths and Physics – hence it is done by students already good at maths and who are already familiar with a large part of the material.

    I believe there is something of a similar problem with RE and Philosophy overlapping.

  214. richas said,

    August 28, 2007 at 8:51 pm

    Something else folk like Richas don’t get – and I have tried – is that structuring the tests, breaking them down into bite-size chunks can make them considerably easier.

    I mean, you can TELL him, and you can explain WHY… but somehow he seems unable to see the difference between telling someone:

    a) here’s a problem, solve it, and

    b) here’s a problem, so work on this, and when you’ve done that, you have to figure this bit out, and once that’s done, the next challenge is this, and so on…

    Thanks for this additional personal attack.

    Can you grasp that the shorter questions are designed to ensure sufficient coverage of the entire curriculum to be able to validly conclude that they have achieved a fixed standard? The questions are targetted because they have to give sufficient coverage to guarantee that the standard has been met.

    A broad general question is fine if you don’t care about curriculum coverage, if all you want is a ranking. Great give them a big broad question, we’ll slap the marks together n the scheme for elements A,B,C and D and if Johnny doesn’t have his psychic Internet device turned on and doesn’t realise we meant him to answer D – TOUGH, we need some failures it may as well be him even if he understood D thoroughly.

    Can you get that the modern paper is more complex and more complete because it cannot rely on the rank and dump approach.

  215. NickConnolly said,

    August 28, 2007 at 11:55 pm

    “No, because in MY day, we DID do quadratics in the O level.”

    But YOUR comment was that it was “barely O-Level standard”. That is a false statement. Simulatenous equations are comfortably O-Level standard as is factoring a quadratic. Having to do both in a question would have been not unchallenging at O-Level in the semi-mythical good-old days.

    “Nick, you whinge whenever you think I’m questioning your motives.”
    No I don’t. I just mock rather silly attempt to diagnose my motives based on nothing.

    “there are a goodly number of easy questions on that paper, at least compared to the O level in MY day”
    There are some questions that overlap with O-Level standard, as has always been the case with A-Level. Originally the lowest non-fail grade for A-Level was exactly that: you were awarded an O-Level. So if your point is that a singl AS paper to be taken after ONE year of study (not two) has SOME question equivalent to the easier levels of A-Level in the semi-mythical good old days then you would be spot on correct.

  216. NickConnolly said,

    August 28, 2007 at 11:59 pm

    ” Which is the most likely hypothesis:
    1) The masters at Eton (and every other school) are doing a far better job than they did, and all pupils are now as good as the small minority were in the past.
    or:
    2) It is now very much easier to gain a grade A.”

    They aren’t mutally exclusive. It IS now much easier to get a grade A. That is an objective fact, as evidenced that more students are getting them and as supported by other data. The question is WHY they are easier.
    For example consider whether teachers are now more willing and more able to coach students effectively for tests – that would imply that the students haven’t got brighter, the teachers no better at teaching and general and the tests being just as intrinsically difficult.

  217. plasecaha said,

    August 29, 2007 at 12:13 am

    Richas.

    .

    “I think you mean that the teaching professionals were asked to teach a detailed curriculum. To devise their lessons, to share best practice, essentially to do their (now much better paid) job.”

    Do you ever think things through? If the way you think is any reflection of the way your mates at the QCA think, we are all screwed.

    What you need to start doing, is thinking things through. If you did that, you would understand what a ridiculous assertion that is.

    Think through what you are asking of the teachers.

    – OK, so a Primary teacher… let’s say they teach five hours a day.
    – that is five hours of lessons they have to teach
    – which means, five hours of lessons to PLAN
    – per day
    – From TEN different subjects
    – And the planning has to meet the objectives for each lesson laid out in the curriculum
    – and the needs of the children
    – which includes making extra provision for those with learning difficulties at the low end
    – AND extra challenges for those at the top end
    – they have to make sure each lessonn follows on from the one before
    – and integrates with other parts of the curriculum
    – the activities need to be engaging
    – and the learning at no point confusing
    – you must avoid the inadvertent introduction of any language or concepts etc. that the pupils aren’t ready for yet
    – and of course you need to design it in a way that’s readily ASSESSABLE, generates material that can be checked to see if the objectives are met
    –if you want to make the learning more efficient, you want to make sure it meets all the principles of learning – such that there is ACTIVE learning involved, and there is reinforcement, and peer assessment and so on, and has things like multiple modes of learning – visual, auditory, kinaesthetic
    – you need to review and evaluate what materials are available elsewhere, or in the school that might be useful…

    That isn’t even a complete list. How long do you think, then, it takes to properly plan and prepare for an hour’s teaching? Whenever I worked in schools, it would take several hours per hour of actual teaching to do a decent job. Many hours, to do it properly.

    Even an infant teacher will have at least four hours teaching to plan per day. There is no way on God’s earth you are going to have a reasonable stab at all the above in a single hour, per hour’s teaching. Even if you COULD, that’s four hour’s planning per school day, on top of the actual teaching itself. Then you have all the meetings to implement those 670 initiatives, and playground duty and assemblies, and being there for the parents at the beginning and end of the day, and planning events like fetes and arts festivals, and planning displays, and linking with the special needs groups, and data to analyse, and teaching resources to make, and the clubs they run for the kids after school, and the co-ordination of your curriculum areas for others, including resourcing…

    and when you finally get home, there is all the marking.

    So, you already don’t have any time left to plan, and yet the planning, done properly, would take hours per hour taught.

    And don’t forget, you have to make sure each year the planning matches up with the intake you are getting from the year below.

    And then, after a few years, when you finally have cobbled together some planning that meets the requirements, you are moved to teach a different year group, and have to start all over again.

    And THEN they want you to build in more regular assessments, and track the performance of the pupils. Only, no one has provided such things, so you have to do it all yourself, and work out a system to manage the data, and analyse it, and then respond to it on a regular basis by adapting your planning, and you do all that while doing all the other stuff…

    And THEN, they change the system entirely, introduce the literacy and numeracy hours, or the foundation stage, and you start all over again.

    And many teachers scraped GCSE maths as it is, so the requirement to deal with quartiles and mean, medians and modes and all that jazz is ludicrous.

    NOW they want them to move to PERSONALISED assessment, regular weekly monitoring for each child… and constantly adapting the planning to suit…

    “Meanwhile the QCA was trialing materials, looking at the materials developed by the TEACHING professionals before disseminating the best (and proven) materials for all to use as they think best.”

    You say the teachers should be doing THEIR job. While the teachers have been wrestling with the above, what have the QCA been doing all these years? Trialling stuff. Have they done THEIR job – providing useful assessments? No. Thus increasing burdens on the teachers.

    The assessments shouldn’t be bolted on as an afterthought anyway. Done properly they should have been integrated with the curriculum from the start.

    If the QCA people actually did THEIR jobs and produced something useful once in a few years, instead of making things worse, it would be easier for teachers to do THEIR jobs.

  218. plasecaha said,

    August 29, 2007 at 2:29 am

    RS

    “Well to some extent that does have to be true (or, as I mentioned above, v.v. with norm referencing preventing criterion based decreases in grades) because norm-referencing cannot allow improved grades.”

    Explain how they were held back, when those scoring lower grades actually performed less well than the top ten percent or whatever getting an A.

    “I believe one of the problems seen in norm referencing was that the courses were actually quite poor at discriminating – resulting in grade boundaries that were implausibly narrow so the difference between a D and a B could be only 8 marks (1982).”

    That’s a good idea, though. Most of the pupils in a norm-referenced system will normally be packed into the middle. Thus, to discriminate, you need narrow boundaries in the middle.

  219. plasecaha said,

    August 29, 2007 at 2:30 am

    RS

    “Pass rates and A grades have been increasing since criterion referencing was introduced in ?1987 for A Levels – and you would expect some increase as teachers learn to teach to the test and gain experience of the examination. So the real question is what rate we would expect grades to inflate if exams were not getting easier?”

    You might expect grades to inflate. You might also expect them to fall initially as teachers don’t yet know the system. The question is, why, in the move to a criterion-based system, should we expect grades to inflate so dramatically? Just because the system ALLOWS it to happen, doesn’t explain why it DOES.

    Think about what we might have expected to see had we just abandoned norm referencing, but kept the exams and curriculum the same, etc.

  220. plasecaha said,

    August 29, 2007 at 2:31 am

    Richas

    “Eton has always excelled. Yes three As were rarer when sixty thousand students took A levels, now it is about 300,000. I have no doubt that Eton performed well with a norm referenced exam and that they excel at ensuring their students meet the standard in the standard based tests where if the pupil demonstrates their achievement the number of A grades is not artificially capped.”

    This is utterly missing the point. The number taking A levels in the COUNTRY may have increased by that much, but not at Eton. So why the increase in the grades at Eton? You can’t say it’s the artificial capping. There was nothing stopping all the students at Eton getting 3As before under the old system. The norm-referencing applied to the results nationwide, not to the results at Eton.

    So, we are left with the idea that, suddenly we introduce the new system, and suddenly, magically, all the students and teachers get dramatically better.

    That is why Killary chose the Eton example. To highlight the paucity of relying on the removal of norm-referencing as an explanation.

  221. NickConnolly said,

    August 29, 2007 at 3:28 am

    “This is utterly missing the point. The number taking A levels in the COUNTRY may have increased by that much, but not at Eton. ”

    You appear to have used a full stop rather than a colon. Let me repunctuate –
    “This is utterly missing the point: The number taking A levels in the COUNTRY may have increased by that much, but not at Eton. “

  222. RS said,

    August 29, 2007 at 10:05 am

    “You might expect grades to inflate. You might also expect them to fall initially as teachers don’t yet know the system.”

    You wouldn’t because criterion referencing was brought in to an existing exam system – you might have expected grades to go down (or up) in a step-change on introduction of the GCSE however.

  223. killary45 said,

    August 29, 2007 at 11:01 am

    RS asked “if it is the exams getting easier per se that causes the rise – why is the rise twice as large in the private sector as the state sector? How can you explain that without additional school specific factors?”
    If that is the case it could be because with their better resources they are more able to take advantage of the easier exams. Apparently in the independent sector as whole 50% of A-levels papers are given grade A.

    Plescha has kindly tried to explain my point about Eton to Richas. Shall I also have a go. Thirty years ago Eton had intelligent, well motivated and hard working pupils. Its masters worked hard to ensure they would get the best A-level results possible so that as many as possible could go to the could go to the best universities. If they did not do this then the paying parents would have been most unhappy. The vast majority took A-levels and Eton’s grades were amongst the best in the country, but 4 grade As was a pretty exceptional performance. Now 4 grade As is the mean result at that school. The best student achieved 10 grade As at A-level.

    I use the independent sector as an example in that it serves as a sort of “control” against the hypothesis that the improvement in A-level grades has been caused government policies in maintained schools. When the “improved” results are announced each year the government delights in taking credit, and claiming that the number of high grades shows the success of their policies. The truth is that the grades have been inflated to disguise the decline in absolute standards in schools.

    Frankly I rather doubt that Eton pupils are much better than they ever were. What evidence for that suggestion is there apart from A-level results? Do universities say that their students are so dramatically better than they were thirty years ago? Most research that I have seen suggests that in fact standards of new entrants to university are declining. The CEM centre at Durham has done some very strong research about this.

    There are two ways of proving that A-levels are much easier than they were. The first is by following Ben’s idea and simply comparing the papers. However this can miss the way that questions are now so very predictable, and so similar to the questions in previous years and in the specially written text books, which are now prepared specifically for each board and for each one-sixth unit of the A-level.

    The second way is seeing how students of the same type in the same schools performance has dramatically improved, with no other likely explanation than easier exams.

  224. plasecaha said,

    August 29, 2007 at 1:28 pm

    Richas,

    “Wow that pshychic Internet connection is really playing up now.”

    I don’t need to be psychic to see the evidence.

    “I understand the maths thanks. I also understand that that achieving a standard is not constrained to a given percentage. You either achieve that standard or you do not. You achieve an A, B, C based on your demonstration of ability in the curriculum.”

    You used to under the old system. That’s one of the things you don’t understand.

    “The maths of norm referencing is easy X% fail, Y% pass, Z% get an A. This system has flaws. To start with it guarantees failure and the waste of pupils and teachers time. If you improve teaching and learning you get the same results, the same proportion failed.”

    No, it only guarantees failure if you fail students. You could just give them a low grade. And if they haven’t done as well, why should they get a higher grade?

    If a teacher improved their teaching and students improved their work under the old system, they would get better grades.

    “Your misconception, what you fail to grasp is that norm referencing by falsing limiting the maximum number of As is only making them more scarce, not more valuable. There is no mechanism no driver for improvement.”

    Yes there is. Indeed, under the new system it isn’t automatic that people have to work better to get a better result, if standards have slipped. Under the old system, if I worked better, I got a better result. I was bottom of the class in history, I suddenly got what it was all about, thanks to a great new teacher, worked my balls off, and got myself an A. I knew I was going to get an A, because I knew what I needed to do to get an A, and I did it, and got it. I always got the grade I figured I was going to get, given my performance.

    The COOL thing about this, which Killary touched on earlier, was that even when the exam was much harder that year, you could still get an A, because of the norm referencing. If the exam is harder, most people are gonna do worse, so they adjust the grades to keep parity with different years.

    BUT, not all students would do equally worse. If you had a real understanding, you could do proportionately better in such an exam, because the harder papers tended to be harder because they contained more of the questions which were less predictable, more removed from the norm, required the ability to go beyond the prepared stuff.

    Even in easier years, you would see questions like that, however, thus, if you wanted to do REALLY well, you focused on understanding more and more, and rote-learning less and less. You could handle the harder questions, and you could still do OK at the easier ones, because you could derive what you needed on the fly without having to drill yourself.

    That’s the problem with the criterion-based system as currently implemented. In being specified in the way that it is, it doesn’t necessarily properly test the ability at the top end, which is to do things like improvise, go beyond the norm, invent and crucially, really understand what’s going on.

    Worse, and this ought to REALLY bother you Richas, because of your concern for the guys lower down: the system does not ENCOURAGE the development of these things. That’s why, when WE got to University, they could throw us in the room filled with the complicated X-ray microscope and all the lead shielding and toss us a few pages of notes and tell us to get on with it, and not to bother the technician because he’s busy doing his OU.

    Whereas when I get students now, I can’t do anything like that. Some can do it, but for the most part, either I have to direct proceedings tightly, or I have to teach them HOW to figure things out for themselves, the stuff they should have been taught at school. Obviously, the latter is preferable, but there isn’t always time…

    You THINK you are benefiting the poor darlings by endowing them with simple, bite-size competencies, instead of endowing them with the ability to teach themselves and figure stuff out for themselves… the powerful higher-order stuff.

    “We know the Eton boys with the best teachers and the best background will come out on top even when state education does a better job.”

    Well, actually the MAIN reason they do well because of the extreme selection that goes on in the private sector. To get into Public School you have to do the Common Entrance Exam, and the better you do at that, the better your choices in terms of which school you can go to. As with Universities, the best schools can cherry-pick the best performers.

    It’s a bit like Grammar schools and the eleven-plus, too. Though one cannot deny that some of these private schools have formidable resources. I know we were VERY privileged in our day in that regard. However, while mine was a top-ten school, in my day the best performer nationally was usually not a Public school, but Manchester Grammar. But back then, things were a bit complicated with Direct Grant schools and stuff.

    “Norm referencing is all about ranking, it is not about validity or reliability over time, it expressly fails in this, it is unable to accomodate better (or worse) results, it is the dumb quota of sheeps and goats, frankly an assessment disaster for a society like ours today.”

    You say this, but where’s the explanation?

    You assume that because norm-referencing curtails the number of As to a percentage, that this is automatically unfair and unrepresentative.

    But it isn’t necessarily so, Richas, is the point. Indeed, the criterion-based system could be the unfair system. That’s what you don’t get, and is what Killary is trying to get across.

    It depends how you set these systems up, how you use them in practice, is the point.

  225. plasecaha said,

    August 29, 2007 at 1:34 pm

    “The current GCSEs and A Levels manage to rank almost every pupil. Amost all of them have a range of qualification showing their personal achievement. The norm reference system would (and did) guarantee failure, it would (and did) completely fail to give any sort of assessment to the majority of people.”

    The failure had nothing to do with the norm referencing. You can have norm-referencing without failing students. And you can fail students under a criterion-based system. And they do fail…

    It wasn’t norm-referencing that failed to give an assessment to the majority, either. They weren’t put in for the old exams, usually because they were too hard for them. When they WERE put in, they failed. So they were given CSEs instead, and if they managed those well enough, they could still earn the equivalent of an O level pass.

    (Now, round about now, I can see you slavering at the chops, feverishly getting ready to leap on a flaw in my argument, that I am advocating a return to this system. Please be reassured, that I’m not. I am simply using it to demonstrate that the problem of failing students wasn’t down to norm-referencing, per se. And you can stop rushing to whinge about me being psychic, either. I’m just going by the many examples of past experience in terms of your rush to read too much into what I am saying, and am saving you a bit of trouble, and me).

    BTW, in addition, you could easily have a criterion-based system that only awarded a handful of As a year.

    “Do you see the flaw in an exam system that assumes that regardless what we do 20% fail? Where if pupils learn more or work harder (as they undoubtedly do today compared to my time at sixth form), where if teachers teach better and teaching materials and equipment improve we still give the same fail rate and the same A rate? A laughable inadequate system.”

    Do I see a flaw in seeing 20 percent fail? Yeah, I would get rid of the failure thing, although to be honest, sometimes you can’t help fail ‘em, like when they don’t hand in any work. So what you do then, is you give them an extension, and if they still don’t do any work, you give them another extension, while the guy in charge of the marks is screaming at you for the results, and you’re getting more and more in trouble, and then the time runs out, but unless you’re an idiot, you still won’t fail them, because they will then have no recourse but to try and look for someone else to blame to get out of the hole, and that is liable to be the class teacher, who may receive a complaint that somehow, for reasons vague and unspecified and unsubstantiated and unchallenged, the student just couldn’t respond to their teaching, even though the rest of the class did very well.

    And YOU know that the real reason they couldn’t respond is that they were out getting utterly trashed the night before your 9am lecture every week because it just so happened student night was the night before, but that will make no difference because the College wants to make the money and you will upset anyone like Richas who might be in charge and didn’t experience the way things are now, and then the failure will be overturned anyway, and your boss is mad at you, and you get the blot of a “complaint” on your copybook. Not smart.

    It’s far easier if they can cite “mitigating” circumstances – their dog died, nail broke, they were “stressed”, etc. Then they don’t have to blame the teacher. If you want to give yourself that safety net so you don’t have to deal with the complaint rubbish later, you can make sure you monitor how much work they are doing before the assessment comes around, make sure you can prove you’ve made efforts to help, and get them to see the college counsellor or something.

    But that’s an extreme case… if you have sense as a teacher, you will try and arrange the assessment so that it involves groupwork – so the others in the group will at least produce something so you don’t have to fail the one who doesn’t do anything. And you try and arrange the assessment so you are present, so you can help make sure they produce something so you don’t have to fail them.

    In HE, where it’s not really allowed to fail students, they do what they have to do in a criterion-based system, and set the bar for failure so low it’s impossible to fall below it in practice. Then the college is happy, because it gets the money for all those extra students who would have failed in the past, and the student is happy because they get their degree, and you’re happy Richas, because no one has failed.

    I mean, they can’t necessarily DO anything useful, and the employers aren’t very happy, and nor are the students themselves later sometimes when this becomes apparent and they wind up in a call centre, and the students who work their butt off are really not very happy at all, but no one’s failed, and the students had a good time, so it’s all good.

    You think it’s unfair if we fail people. And you know, I’m not a fan of failing anyone. I didn’t used to fail ‘em even when we COULD fail ‘em. But the way it is done, matters. Equally, it’s not necessarily all that unfair for a percentage to always get the top grade, especially in comparison with alternatives. That’s what you don’t get, Richas. Indeed, it can be unfair if we arrange a system so that failure doesn’t happen any more, and many more DO get the top grade.

    “No, where if we get rid of the norm referencing we can count those who achieve the standard for each grade. Good.”

    Sure. I’m not arguing for a return to norm-referencing. The problem Killary is pointing out, which you don’t get, is the problem with the IMPLEMENTATION of the current criterion-based approach. The way it doesn’t sufficiently discriminate at the top end, for example.

    This is what you don’t get. You keep banging on saying the new approach must be better because the old one was intrinsically more unfair, and therefore any rise in standards must be legitimate. Neither automatically follows. You still don’t get it, as the Eton example showed, earlier.

  226. RS said,

    August 29, 2007 at 1:58 pm

    “If that is the case it could be because with their better resources they are more able to take advantage of the easier exams.”

    They would, of course, also be more able to take advantage of more difficult exams – but it does establish that something more than easier exams must be at play – resources is your example for private schools, and, of course, extra resources _could_ be an explanation for state schools as well.

  227. plasecaha said,

    August 29, 2007 at 2:14 pm

    Don’t forget the fact that these kids are heavily selected. They are brighter than most. They are brighter than many of the teachers in the state sector. Also, many of their teachers will be Oxbridge, but the teachers don’t necessarily do all that much, because they don’t have to.

    If the exams are easy enough for the majority to get As and Bs, then you have made the exams trivial for these people. Once they become trivial, they can churn them out.

    One of the big barriers to multiple A grades in the past, was the amount of memorisation involved. You could be ABLE enough to do well on a paper, but you couldn’t cram enough in in time.

    If you remove that, they can churn through the papers effortlessly.

  228. richas said,

    August 29, 2007 at 4:05 pm

    If the QCA people actually did THEIR jobs and produced something useful once in a few years, instead of making things worse, it would be easier for teachers to do THEIR jobs.

    The traditional role of the QCA has been to set the national curriculum and regulate the assessment of that curriculum.

    The teachers role has always been the demanding one you summarised. In the past few years for the frst time the QCA has also supervised the production and distribution of teaching materials. This was at the time opposed by some teaching professionals. It was also used as a stick to beat the QCA with in terms of dumbing down by providing teachers crib sheets.

    The QCA materials you deride as too late were innovative. The QCA developed the curriculum and assessments in tandem, now you want to expand their role to providing teaching materials at the same time too? Do you think an organisation you regard as dumb could or should do this?

    Yes it is teachers job to teach the curriculum, to teach it in such a way that their pupils learn as much as possible and achieve the highest grade possible. Not easy but that is their job.

  229. richas said,

    August 29, 2007 at 4:14 pm

    That’s a good idea, though. Most of the pupils in a norm-referenced system will normally be packed into the middle. Thus, to discriminate, you need narrow boundaries in the middle.

    It is not good in terms of reliability and validity to have a narrow marking band between D and B. You want as few candidates as possible at the boundary not every pupil given a B C or D (the majority) to be within 1 or 2 marks of a boundary.

  230. richas said,

    August 29, 2007 at 4:28 pm

    This is utterly missing the point. The number taking A levels in the COUNTRY may have increased by that much, but not at Eton. So why the increase in the grades at Eton? You can’t say it’s the artificial capping. There was nothing stopping all the students at Eton getting 3As before under the old system. The norm-referencing applied to the results nationwide, not to the results at Eton.

    So, we are left with the idea that, suddenly we introduce the new system, and suddenly, magically, all the students and teachers get dramatically better.

    Where do you get the idea this was a sudden change upon introduction. The allegation is of a gradual and slow inflation of grades over the decades.

    In mythical golden days the Eton boys were part of the 12% that took A levels. Their competition was the other public and private schools plus some state Grammar boys and girls who were selected at 11 and again at 15 (leaving) and 16 (O level results and leaving). Many bright but poorer Grammar kids left for economic reasons. Anyway they are in a tough cohort of the top 15% or so with a big bias toward a privileged background. Private education has remained fairly stable throughout the period at 7% or so of the cohort.

    If we kept norm referencing and expanded the A Level population to 50% of the cohort (as we have done) how do you think those at Eton with their selection and privilege would have done? For a privileged fixed number (about 100 a year isn’t it?)?

    I’d predict pretty well being in the top 5-10% vs the top 50% not 15% of the population (where the new entrants have a wider economic, social and probably educaional gap) is probably easier. Indeed it is possible that the grade inflation for Eton would be just as great under either system.

    Eton knows how to get their kids to the A standard. Good. It should be a fairly easy task for them given the resources and the selection but I am glad they do it for most (but not all) their pupils.

  231. richas said,

    August 29, 2007 at 4:41 pm

    Frankly I rather doubt that Eton pupils are much better than they ever were. What evidence for that suggestion is there apart from A-level results? Do universities say that their students are so dramatically better than they were thirty years ago? Most research that I have seen suggests that in fact standards of new entrants to university are declining.

    Well they do award more firsts and uper seconds now. Are they dumbing down too (despite doing so well in international comparisons) or are the universities taking this “declining” input and producing a superior output.

    By the way by “research” I think you mean “anecdote”.

  232. richas said,

    August 29, 2007 at 4:48 pm

    If a teacher improved their teaching and students improved their work under the old system, they would get better grades.

    Yes if an individual pupil improved their reslt they got a better grade – but at the same time another pupil who performed exactly the same would be bumped down. That’s norm referencing.

    If you have a general improvement for all (or just for a lot) all you get is the same output in terms of grade. How does this measure achievement accurately? It doesn’t – it just measures rank, it just sorts sheep from goats.

    A standards based system allocates the grad based on that individuals performance against the curriculum. Simple in terms of output, it allows improved performance to be rewarded but it is a more complex assessment.

  233. plasecaha said,

    August 29, 2007 at 6:02 pm

    Richas,

    “Thanks for this additional personal attack.”

    Thanks for misrepresenting me again. It wasn’t a personal attack. I was characterising your position in this debate.

    For the record, I think it’s great you are flying the flag for the standards thing, for student opportunity and all that, and playing a big part in the debate, and I respect all that, and wouldn’t want to attack you personally in a hurtful way.
    (I do find it hard to avoid putting things like being misrepresented in its place though).

    “Can you grasp that the shorter questions are designed to ensure sufficient coverage of the entire curriculum to be able to validly conclude that they have achieved a fixed standard?”

    I can grasp that this may be the intention, yes. You have a lot of faith in the architects of the system to do everything perfectly well.

    In practice, in many cases, the new system, despite this intent, has been worse than the old system, because it has been so darn easy to cheat. So in too many cases, all you have assessed is the ability of the teacher, or parent, or a mate, or someone else in the group, or even someone they paid, to achieve a fixed standard.

    And it’s all very well having things fixed, but really… why SHOULD they be? What about all the people who can EXCEED the standard? What about being fair to them?

    But hey, you got your increase in grades, and nobody failed, so everything’s perfect, right?

    “The questions are targeted because they have to give sufficient coverage to guarantee that the standard has been met.”

    If by targeted, you mean broken down into chunks, then no, this isn’t automatically necessary.

    With a lot of problems as they used to be set in the old system, you didn’t need to break it down into chunks to ensure all the bits were covered. The problems set were such that there may have been certain steps required to solve the problem, but the student had to figure these out without prompting.

    They would have to DO the steps to do the problem, and there would be credit in the marking scheme for this. Indeed, the one constant refrain in my era was “show your working”.

    Usually, there was only one way to solve the problem, but sometimes, there were several ways, and you could get credit for either. Unless they wanted a PARTICULAR method, in which case they would specify. Or, maybe, you’d come up with a different method of your own, in which case: Bingo. That was special.

    In other words, you can choose problems that require a certain method, and give criterion-style marks for each step, without specifying the steps.

    Sometimes, for some problems, they WOULD specify steps, but this would be for especially hard or unfamiliar things. One Physics question, I remember, involved analysing wind effects in shopping precincts. That was broken down a bit.

    To summarise, IF the students can figure out the steps for themselves, they don’t need the structure provided. The only important thing is that the structure is credited in the marking scheme.

    “A broad general question is fine if you don’t care about curriculum coverage, if all you want is a ranking. Great give them a big broad question, we’ll slap the marks together n the scheme for elements A,B,C and D and if Johnny doesn’t have his psychic Internet device turned on and doesn’t realise we meant him to answer D – TOUGH, we need some failures it may as well be him even if he understood D thoroughly.”

    You can design a broad question to give curriculum coverage, as I have explained above. Let me give you an example from my day: Show how Nelson won the Battle of Trafalgar.

    To answer that question, I needed to be able to specify all the key moments in the battle, the status of the fleets, the tactics involved, the unexpected events, the outcome, all without prompting or structure, AND pull them all together to show how they affected the outcome. It took about 25 minutes. You HAD to do those things to answer the question, and credit for them would be given in the marking. Indeed, once I saw that question turn up, I knew I’d get an A, even though I’d gone into that exam with only a B so far.

    In the process of doing all that, I was demonstrating the key competencies of historical analysis. I just didn’t have to told what those competencies were in the exam, or be told how to apply them in that specific case.

    But yes, if Johnny hasn’t been taught very well or hasn’t learnt his stuff, and doesn’t know how to identify the key issues for himself and address them, then yeah, you’re gonna have to hold his hand for it.

    You have to understand that just because the GRADES in the old system were norm-referenced, the MARKING wasn’t so much. The MARKING was much more criterion-based. So, I knew I’d get points for certain things that had to be there. Show your working.

    Indeed, if you were running out of time, as I invariably did, what you did was answer the rest in skeleton form. You would note down the key points and the process you WOULD have gone through, had you had more time to flesh it out. This would at least give you a few extra marks.

    In other words, under the old system, you would get CREDIT for coming up with the structure. Now, not only can you not get credit for this advanced skill, you don’t even have to do it.

    In addition to “show your working”, another refrain was “do an essay plan”. Commonly you would have five essays per 3 hour paper, and I would allocate half-an-hour to each, with a half-hour left to quickly plan each of them. Not everyone did this, thinking that it took time away from the essay itself, but I always did it, even taking longer than the half-hour, because it allowed you to be more efficient in the essays themselves, and if you ran out of time, you could include the plan to show the examiner what you had planned to do, and get some marks.

  234. plasecaha said,

    August 29, 2007 at 6:16 pm

    “You appear to have used a full stop rather than a colon. Let me repunctuate –
    ‘This is utterly missing the point: The number taking A levels in the COUNTRY may have increased by that much, but not at Eton.’“

    I know I didn’t rush to reply to your last post Nick, but post-stalking me with nonsense like this is even more desperate than usual. And a bit creepy. Really, you can disabuse yourself of the notion that I need lessons in English from you, or that any of us do, frankly. Especially since one only has to refer to your previous post – in which even though it was a brief post you still managed to make errors, writing “evidenced that”, and “no better at teaching and general” – to see that you can’t even use words like “that” or “and” correctly.

    That was a correct use of a hyphen, by the way. You tend to use them in place of colons. Meanwhile, your suggestion I use a colon was incorrect, because “the number taking A levels in the country” etc., was not a point in itself, but evidence for a point I made later. Even as a pedant and a troll, you fail, and let’s hope you don’t teach English as well as maths…

    Anyway, being as you’ve left me more pedantry to wade through…

    “But YOUR comment was that it was “barely O-Level standard”. That is a false statement. Simulatenous equations are comfortably O-Level standard as is factoring a quadratic. Having to do both in a question would have been not unchallenging at O-Level in the semi-mythical good-old days.”

    Well, it depends on your standard. In the O Level paper from your era, I can see that it would actually have EXCEEDED the standard. The ability to solve quadratics was not tested. I don’t know whether or not you had to do them, your argument suggests possibly that you were, but you can clarify that. If you were, then clearly there may have been quite some variation between boards.

    By the standards I had to do for my O level, that question was poor. By the standards of the paper I cited, too… in the Peripheral question, the O level students had to formulate the quadratic before solving it without being given any equations to begin with, unlike the case with simultaneous equations.

    But really, you are just quibbling. You are trying to trap me in an error as an excuse to dismiss other things I point out. But even if I do make an error Nick, it won’t make everything I say subsequently wrong. You cocked up when you said the examiner had said the most about that foundation question, but I haven’t used that to dismiss everything you said subsequently. I pointed out most of what you said that was wrong independently of what you said before.

    So that angle’s dead in the water Nick, let it go.

    “No I don’t. I just mock rather silly attempt to diagnose my motives based on nothing.”

    How can you mock something someone hasn’t done? You claim people are psychic, inventing the internal, and you trump that by inventing the observable, illusions. That’s worse than being psychic, that’s delusional. I’m relieved I’m not psychic, because I really wouldn’t want to get into your head Nick, believe me. And you are still being a hypocrite.

    So that’s dead too.

    “There are some questions that overlap with O-Level standard, as has always been the case with A-Level. Originally the lowest non-fail grade for A-Level was exactly that: you were awarded an O-Level. So if your point is that a singl AS paper to be taken after ONE year of study (not two) has SOME question equivalent to the easier levels of A-Level in the semi-mythical good old days then you would be spot on correct.”

    Yes, one could be awarded an O level for doing an A level. This does not automatically mean that the lowest grade was only O level standard, and I do not recall doing O level questions on the A level paper.

    My point, actually, you keep missing. I am not comparing the AS level paper with the O level for its own sake. My main concern is to compare the OCR paper with the AQA. Having checked for myself, I have already come to the conclusion that on the evidence thus far, one can fairly say that for the AQA board at least, it is sufficiently comparable to the old A level to not give much cause for concern.

    The interesting thing is the comparison with the OCR paper, which has questions much easier than any I could find on the AQA. Part of my demonstration of this lay in comparing with the old O level.

    I’m not debating whether the OCR paper is harder than the old O level. Overall, clearly it IS harder. I am just wondering why it seems so much easier than the AQA, and of course, the implications of this.

  235. benko said,

    August 29, 2007 at 7:03 pm

    Obviously kids are getting stupider every year. Equally obviously easier questions can’t explain the trend in exam results, as IQ scores have also been steadily increasing, and those questions haven’t changed (as a number of posts have mentioned). I can think of only one conjecture that has not been refuted: morphic resonance.

    As Sherlock Holmes (or was it Karl Popper?) said:
    when you have eliminated the impossible, whatever remains, however improbable, must be the truth.

  236. plasecaha said,

    August 29, 2007 at 8:01 pm

    The questions not changing makes them easier, and as for other things not refuted, these include the dumbing down due to breaking things down into little pieces, and the absence of more open-ended questions, the abundance of cheating, the increased teacher assistance, etc. Morphic resonasce is possibly what the other guys are using to account for these things.

    Popper, being the guy who said you couldn’t prove theories only falsify them, suggested that checking their own theories for flaws was the primary endeavour of a scientist. You might find that handy, Benko…

  237. plasecaha said,

    August 29, 2007 at 8:05 pm

    Btw, does anyone have any data on WHEN IQ levels started rising? Has it been over the last 30 years or so, or did it magically start when the new exams came in?

    If it’s diet related, it presumably coincides with whenever people started getting taller…

  238. plasecaha said,

    August 29, 2007 at 8:14 pm

    Lol, it’s the Flynn effect. OK, scratch that last question…

  239. richas said,

    August 29, 2007 at 8:38 pm

    The questions not changing makes them easier, and as for other things not refuted, these include the dumbing down due to breaking things down into little pieces, and the absence of more open-ended questions, the abundance of cheating, the increased teacher assistance, etc. Morphic resonasce is possibly what the other guys are using to account for these things.

    The questions do change.

    The questions are broken down into smaller parts for the foundation parts – usually for the award of C or D and below.

    Open ended questions (and very open ended coursework) can and are used to assess at the higher levels.

    There is little or no evidence that “cheating” has a greater impact upon results or is more common today than the mythic one or two generation back period so often perceived as golden.

    I think “teacher assistance” is covered under “cheating”.

    Has this style of response been clearer, even though the content is similar to previous comments?

  240. richas said,

    August 29, 2007 at 8:52 pm

    once I saw that question turn up, I knew I’d get an A, even though I’d gone into that exam with only a B so far.

    At last an anecdote that helps to explain why the current system is superior in terms of reliability.

    Your experience was a common one when the whole exam rested upon a very small number of open ended questions. The outcomes for individuals differed widely depending on their luck with the paper. Had they guessed the right bit to revise? Was a similar question in their mock? Was it the one bit of the curriculum they enjoyed?

    If you put a cohort through two different tests in the same format they got different results. This lack of consistency is indicative of lack of reliability in the test.

    I’ll give you another example from my past. English Lit O’Level. We had four main texts, Macbeth, Steinbeck, some American Short Stories and by far the longest text Jane Eyre. Being a callow, ignorant and slightly macho youth I hated the idea of reading the girly Jane Eyre so I didn’t. (I have become less of a philistine since and have read and appreciated it since. I loved Steinbck and read this and others for pleasure before, Macbeth is and was great and the short stories, well they were short and a couple were OK). (We also had a seperate poetry thing for 10%).

    Sructure of the exam 4 questions from minimum three areas. I never read the Jane Eyre never answered a question on it ever. About a third of the text at least a quarter of the curriculum dumped. I got an A. Couldn’t happen now, my gap would be found out by the foundation level questions.

    The new style of a mix of foundation and more in depth question is more thorough. It is less dependent upon chance and because it results in consistent results for the same cohort can be shown with certainty to be more reliable because of this.

  241. richas said,

    August 29, 2007 at 9:01 pm

    Btw, does anyone have any data on WHEN IQ levels started rising? Has it been over the last 30 years or so, or did it magically start when the new exams came in?

    Since they were created.

    www.danenet.org/ncs/forumiqscores.htm

    If a representative sample of today’s children took the Stanford-Binet test used in 1932, about a quarter of them would be defined as very superior, a rating usually accorded to fewer than 3 percent of the population, Dr. Neisser said.

    Now our fixed standard based tests (as with other standard based tests around the world show rising results). A fixed test shows better results over time. Why do we need an hypothesis that tests have got “easier”?

  242. plasecaha said,

    August 30, 2007 at 5:20 am

    Richas,

    “Can you get that the modern paper is more complex and more complete because it cannot rely on the rank and dump approach.”

    You can’t get that it’s entirely possible that it’s less complex and less complete so that it can avoid ranking and dumping them. If you can’t get the difference from the examples above, Jeez…

    But let me give you another example from the past, this time on the difference in terms of coursework: the Physics Investigation. We were given two weeks to research a topic of our own choosing. We had access to the labs, and all the gear, but we were ENTIRELY on our own with the project. We chose our own experiments, and how we would do them, and there was no help whatsoever. We weren’t told what sorts of experiments we should do, or how, or how to analyse the results… nothing.

    We hadn’t even done anything like it before, either, so we had no reference. We had to use our experience of doing lab-work in studying the curriculum, and our knowledge of physics, and anything we could find in the library (no internet), and that was it. The only thing the teachers were allowed to do was to give us a “hint” if things were going catastrophically wrong. And even then, it was just a hint. They wouldn’t actually tell you what was wrong, and even then, you had to figure out what to do about it.

    Two weeks of experiments we had to design ourselves, and we all did fine. But it’s in a completely different league to coursework today. Nuffield being a practical curriculum, we DID have a lot more practice doing experiments. However, we had never designed any ourselves before. We were on our own. Especially since, being a boarding school, you didn’t even have your parents on hand to help out…

  243. plasecaha said,

    August 30, 2007 at 7:09 am

    “The traditional role of the QCA has been to set the national curriculum and regulate the assessment of that curriculum.

    The teachers role has always been the demanding one you summarised. In the past few years for the frst time the QCA has also supervised the production and distribution of teaching materials.”

    Alright, let’s use an analogy, which highlights the absurdity, but preserves the implications of your logic.

    Suppose, you had some debilitating illness, and you went to the Doctor, and he told you you needed a certain drug. And you said, fine, I’ll have that then, and he said, you can’t, because it’s still in trials.

    You might be miffed, but you would probably accept they needed to trial these things.

    What if he THEN told you that they KNEW they would need to develop this drug years ago, but despite that, years after the drug is needed, it is still in trials?

    And then, you go ahead and develop your OWN drug, because you haven’t got any choice, and the drug people still haven’t got it together, and are still saying it’s not ready yet. Meanwhile, you keep taking your own drug and surviving.

    And then those clueless numpties create a new disease so you have to discard your drug and develop a new one for yourself while the numpties start trialling a new drug of their own for a few more years.

    But to you, Richas, that’s all OK. Because everyone is doing what they are supposed to be doing.

    Replace drugs with lesson plans or assessments, and disease with whatever new hairbrained scheme they have inflicted on the public, and you have your argument about the QCA in a nutshell.

    “This was at the time opposed by some teaching professionals. It was also used as a stick to beat the QCA with in terms of dumbing down by providing teachers crib sheets.”

    That is a criticism I wasn’t aware of. I was aware that the QCA materials had been considered dull and barebones, and that Ofsted frowned on them. Personally, I consider the QCA materials a step forward compared to before, regardless of whether they’re no all that good.

    “The QCA materials you deride as too late were innovative. The QCA developed the curriculum and assessments in tandem, now you want to expand their role to providing teaching materials at the same time too?”

    Oh, so it’s burdensome for a whole organisation to provide teaching materials, but somehow OK for a teacher to do it on their own?

    And as for the assessments, we are moving towards regular personalised assessment. Another dramatic increase. Given these things are best integrated to the curriculum anyway, who is best placed to provide materials for that? The teacher? Or the body in charge of assessment and the curriculum?

    “Do you think an organisation you regard as dumb could or should do this?”

    Can an organisation improve? Sure? I don’t actually think the National Curriculum is all that dumb. The new Foundation Stage has some real dumbness in it…

    A few years ago, I saw some Assess-and-Review materials, along with a presentation pack which seemed very good. I don’t know if it was QCA, but I assume so. But it was only for maths, and only for some of the year groups. Crazy.

    “Yes it is teachers job to teach the curriculum, to teach it in such a way that their pupils learn as much as possible and achieve the highest grade possible. Not easy but that is their job.”

    It is ridiculous to let them do it on their own when it requires so much work and there is an organisation best-placed to assist.

  244. elnorgio said,

    August 30, 2007 at 7:33 am

    I’ve taught science, particularly physics but also some chemistry and biology, as well as maths, to GCSE and A level for nearly 10 years now. Over that time, the standard (until the intro of the new GCSE this year – thank the Flying Spaghetti Monster I’m teaching IGCSE this year) has been fairly consistent. That said, if you look back further than that the difference is quite marked. I remember coming across a 70’s O level text a few years ago, that was to a higher level than A level.

    You could also argue that the introduction of modular A levels has had a big influence on subject “hardness”. My kids have been able to resit any dodgy results (some as many as 4 times) until they get the result they want. When I sat mine (1995), it was one exam at the end of the year and that was it.

    This year, I’m teaching the International Baccalaureate. I’ve got to say that it’s a good deal more rigorous than the A level stuff, and to a much higher standard. And it’s not modular. And the kids have to do 6 subjects (plus TOK) instead of 3 or 4. Now that’s proper (but then, it is for rich, spoilt brats on the other side of the world – in every conceivable way – from a comp on a council estate in London).

  245. plasecaha said,

    August 30, 2007 at 7:59 am

    RS

    “You wouldn’t because criterion referencing was brought in to an existing exam system – you might have expected grades to go down (or up) in a step-change on introduction of the GCSE however.”

    Ah, I didn’t read what you said properly. I assumed you meant the introduction of the new exam, otherwise what you said didn’t make any sense. You claimed grades would improve as teachers learned to teach to the test, but if you haven’t actually changed the exam, that wouldn’t apply either.

    So, since you say grades improved SINCE the introduction of criterion referencing, but WITH the old exam, that’s an interesting situation, since it sheds light potentially on the impact of criterion referencing.

    Why did the introduction of criterion referencing alone raise grades? You may say that since the norm-referencing had been removed, more people could achieve higher grades. But the thing is, under the old system, the marks followed a normal distribution anyway. In other words, there actually WAS a top ten percent in terms of raw performance.

    It isn’t the case that loads of people did really well on the actual papers, and then the nasty norm-referencing man came along and docked them points. The students of yore who did not get an A, were not the educational equivalent of Leeds United. They DID actually perform better than their peers that year.

  246. plasecaha said,

    August 30, 2007 at 8:12 am

    Lol, I meant of course that the ones who got an A performed better…

  247. richas said,

    August 30, 2007 at 8:40 am

    That’s what my examples illustrated. The pressure and the vector. Richas tries to dismiss these as anecdotes, because I couched them in terms of my personal experience, but you can’t do that. Those examples are significant. I could have expressed them in a different way, without referring to my personal experience, and they would still stand as potential explanations of what’s going on, but I assumed he would have realised that for himself.

    I agree, you have enough for a hypothesis. I’m just waiting for the evidence that supports it (presumably before the new system of supervised, timebound, coursework kicks in).

    I also find it odd given your defence of teachers workload and professionalism that you now have a hypothesis that relies upon them being a main vector for “cheating”.

  248. richas said,

    August 30, 2007 at 8:47 am

    he hasn’t even tried to prove it.

    Except to cite a noddy QCA review that ignores differences in questions, lets the exam boards pick their own samples, and doesn’t take the greater help provided by teachers these days into account.

    I have tried to explain the mechanism that means that those setting the exams can compare papers over time by comparing curriculum coverage. The way that the questions are trialled and the way that the exam boards set the boundaries by comparing complete papers for boundary candidates to assess their knowledge of the curriculum from this year and previous years.

    The exam boards and the regulators are professionals whose credibility withn the assessment relies upon producing reliable, valid consistent assessment with stable standards. This is the pressure and the vector.

    The QCA review was not “noddy”. Like all good research they outlined what they did not look at in a clear summary for you to abuse. Wha you did not do was then look at what they did do and the conclusions they were able to draw from that.

  249. richas said,

    August 30, 2007 at 9:01 am

    Why did the introduction of criterion referencing alone raise grades? You may say that since the norm-referencing had been removed, more people could achieve higher grades. But the thing is, under the old system, the marks followed a normal distribution anyway. In other words, there actually WAS a top ten percent in terms of raw performance.

    It isn’t the case that loads of people did really well on the actual papers, and then the nasty norm-referencing man came along and docked them points. The students of yore who did not get an A, were not the educational equivalent of Leeds United. They DID actually perform better than their peers that year.

    Thereis still a top 10%. In the case of GCSEs there is the A* that equates to the top 5%. With A Levles the universities will have access to details of the results beyond the grade so that they can compare stupents more precisely than a mere grade.

    You are of course right that the A in norm referencing by definition performed better than a B that year. If performance improves overall then the pupil with a B in the “better” year could easily have out performed the A student in another year.

    How is that a reliable assessment of the standard achieved? Having fixed standards allows us to compare pupils from different years properly.

  250. richas said,

    August 30, 2007 at 9:11 am

    I’m interested that you think doctors should recommend drugs that have not been trialled, maybe they should recommend homeopathy too?

    The QCA approach is somewhat different they invite people to be involved in the development and trialling of materials. As a governor I have some experience of this with the new Diplomas that our school will be developing next year. I hope that when they go national teaching materials will be made available to all. As your position makes clear the opposition to the provision of such materials has declined. It was however a controversial step seen as encroaching on the professionalism of teachers.

    The equivalent is the GP making a patient aware that a trial is underway that they could take part in if they want.

  251. NickConnolly said,

    August 30, 2007 at 9:17 am

    Plasecha:
    “I know I didn’t rush to reply to your last post Nick, but post-stalking me with nonsense like this is even more desperate than usual. And a bit creepy. Really, you can disabuse yourself of the notion that I need lessons in English from you, or that any of us do, frankly. Especially since one only has to refer to your previous post – in which even though it was a brief post you still managed to make errors, writing “evidenced that”, and “no better at teaching and general” – to see that you can’t even use words like “that” or “and” correctly.
    That was a correct use of a hyphen, by the way. You tend to use them in place of colons. Meanwhile, your suggestion I use a colon was incorrect, because “the number taking A levels in the country” etc., was not a point in itself, but evidence for a point I made later. Even as a pedant and a troll, you fail, and let’s hope you don’t teach English as well as maths…”

    Did you maybe here a sort of whoooshing noise just above you as you were typing? That was my probably my point flying over your head. Looks like somebody’s sense of humour has gone the way of their psychic internet connection…

    HINT: With a colon the “This” refers to what follows in “This is utterly missing the point”. Well it made me chuckle.

  252. NickConnolly said,

    August 30, 2007 at 9:36 am

    “y the standards I had to do for my O level, that question was poor. ”

    Oops making stuff up again.

    “And you are still being a hypocrite”
    And more childish name calling…

    “This does not automatically mean that the lowest grade was only O level standard, and I do not recall doing O level questions on the A level paper. ”
    An O-level was an awarded grade for A-Levels in the good-old-days. That you do or don’t recall something is irrelevant. As far as I’m aware what proper research has been done (i.e. something other than the woo plan of Ben’s or your gut-feeling approach) suggests that it the standard for a grade E is easier than in the past – but not by much.

    But hey, where’s the fun in that when we could have a rant about falling standards based on reading chicken entrails?

  253. RS said,

    August 30, 2007 at 1:34 pm

    “You claimed grades would improve as teachers learned to teach to the test, but if you haven’t actually changed the exam, that wouldn’t apply either.”

    They would. Previously norm-referencing prevented any increases or decreases in quality from being detected.

    Criterion referencing allows the increase in teaching to the test that will naturally proceed from the accumulated familiarity with the test however long it has been running.

    Changes in the syllabus, which inevitably happen periodically, could potentially be picked up as step changes in the pass rate.

    “Why did the introduction of criterion referencing alone raise grades? You may say that since the norm-referencing had been removed, more people could achieve higher grades. But the thing is, under the old system, the marks followed a normal distribution anyway. In other words, there actually WAS a top ten percent in terms of raw performance.”

    I’m afraid I don’t understand your point. I’m saying that it is inevitable that some teaching to the test will happen, and this will continue to happen as experience accumulates over time – a good example is the familiarity and past questions that Public schools had with the Oxford entrance exam. Therefore a criterion referenced system will always show a trend to increasing marks – and obviously this could not happen with norm referencing – the question is whether we think the increase in passes is greater than would be expected by teaching to the test, and genuine improvements in teaching methods.

  254. RS said,

    August 30, 2007 at 1:37 pm

    “so long as the grade for a D was around 70% for the OCR paper and 40% for the AQA paper (with equivalent intervals) then it would be just as hard to get a D on the ‘easier’ paper than on the ‘harder’ paper.”

    Of course there can be questions as to the validity of the test itself. For example, a completely random multiple choice exam will still sort people into percentiles without necessarily revealing anything about students ability.

  255. plasecaha said,

    August 30, 2007 at 2:41 pm

    “HINT: With a colon the “This” refers to what follows in “This is utterly missing the point”. Well it made me chuckle.”

    That doesn’t make any sense, because I wasn’t missing the point. Why would I see something that doesn’t make any sense? Unless you mean HE was missing the point. But since I was going on to make my point, it still doesn’t make any sense. Keep smiling.

    “y the standards I had to do for my O level, that question was poor. ”

    Nope, I gave an example earlier to show why.

    “An O-level was an awarded grade for A-Levels in the good-old-days. That you do or don’t recall something is irrelevant.””

    I agreed with that. I disagreed that the standard automatically had to be O level.

    “As far as I’m aware what proper research has been done (i.e. something other than the woo plan of Ben’s or your gut-feeling approach) suggests that it the standard for a grade E is easier than in the past – but not by much.”

    But hey, where’s the fun in that when we could have a rant about falling standards based on reading chicken entrails?”

    OK, so we have your “awareness” to go on instead. Cool. Personally, I think we’d be better off with the entrails…

    Meanwhile, back at the debate: being as you teach GCSE maths, how does the standard of those questions on the OCR paper compare with GCSE?

    “Now if the above ISN’T the case then that is another story. But you don’t know which way round the issue is. Lets say the 40% paper actually gave at D’s for a score of 30% and the 70% paper gave them for 80% then actually your subjective impression of difficulty would be way off. the ‘easier’ paper would actually be too hard and the ‘harder’ paper too easy.”

    That’s exactly my point. When I said I was wondering why, I wasn’t wondering how grade boundaries work. I made clear earlier I understood, when I gave the example of how you could shove the grade A around to compensate for differences in difficulties in papers. Oddly, you have overlooked that. I was making the point that we don’t know which way round, but that it is POSSIBLE there is a disparity between boards.

    I chose the A grade example, rather than your D grade, because it highlights better the LIMITATIONS in moving the grade boundaries around. You can only move the A grade up so far to make an easier paper harder in practice. You chose an example that, strangely, makes your case seem stronger than it is. Is that a failure to understand grade boundaries on your part, Nick? Or something deliberate to hide the truth?

    Oh yeah, you can drop the requirement for an A on the harder paper. You could drop it to 10 percent if you like, but who’s going to do that? The lower you set the A grade on the harder paper, the more you make the questions redundant. And there is still a hard limit.

    The folly of your approach is highlighted in the example you gave. You’re setting a D grade at 70 percent, thus cramming all of the grades into the top part of the paper. What’s the point of all the lower scores and questions then? There are limits to what can be reasonably expected of grade boundaries, Nick.

    If OCR papers as a collection are significantly easier overall, then the more difficult questions on the AQA are either redundant, OR the paper IS harder.

  256. plasecaha said,

    August 30, 2007 at 2:41 pm

    RS

    “They would. Previously norm-referencing prevented any increases or decreases in quality from being detected.”

    Not necessarily. You can spot and account for longer-term trends in the data, adjusting locally for blips up and down each year. In practice, there doesn’t appear to be evidence there was this gradual improvement. But then again, there wasn’t as much PRESSURE to improve. If pupils didn’t perform well, that was their problem. I think it’s good there is that pressure now. Whether it’s being used in the correct ways is something else.

    “I’m afraid I don’t understand your point.”

    Yeah, to be honest I wasn’t really directing that at you, I was just following on to make a different point.

    “I’m saying that it is inevitable that some teaching to the test will happen, and this will continue to happen as experience accumulates over time – a good example is the familiarity and past questions that Public schools had with the Oxford entrance exam.”

    Well, I take your point, but to be honest, while those schools DID have additional experience over other schools, it came from a different source.

    I said earlier, that all the Oxbridge teachers didn’t have to do all that much at the top Public Schools. In the context of the A levels that was fair, but I thought about it some more and remembered it was very different when it came to Oxbridge.

    Because they were at that level and were very familiar with the system, having been there, they could do quite a lot to help in that regard. In my day, Oxbridge exams were usually seventh term, so the goal was to get past the A level syllabus and onto Oxbridge as quick as possible. Also, they had more to give in terms of which College to go for, and which courses.

    This was all very handy stuff, and it came from having been to Oxford and Cambridge, rather than improving the teaching. Teachers rarely sought to improve their teaching back then. Mostly, they would just work through a textbook. Certainly SOME tried. But there was no global pressure to improve. So it didn’t happen.

  257. RS said,

    August 30, 2007 at 3:21 pm

    “Not necessarily. You can spot and account for longer-term trends in the data, adjusting locally for blips up and down each year. In practice, there doesn’t appear to be evidence there was this gradual improvement”

    How could you tell if only a fixed proportion got each grade? Are you saying there is additional data available?

    Because the evidence that in the 1980s there was concern that grade boundaries were becoming too close together would tend to argue for a change in performance distribution (if we assume similar difficulty between papers) although not in what direction.

  258. plasecaha said,

    August 30, 2007 at 3:28 pm

    Just to be clear, obviously, they were more experienced at Oxbridge exams than other schools, but I didn’t see much evidence of an effort to improve, or pressure to. About a quarter got in every year. Around another quarter were good enough to try, but didn’t get in. A good number of the rest didn’t really have that as their chosen course. They had other plans. The proportion getting into Oxbridge stayed pretty stable, year-on-year. I actually remember checking this at the time, because I was interested to know how good our year was in comparison to others.

    If anything, I imagine the number attending in future years may have fallen, because of the increasing pressure on Oxford and Cambridge to admit more from the state sector, something which was already quite a factor at the time. In my college, there were four of us in my year on the same course. The other three were state-educated.

    There were OTHER ways to skin the cat, anyway, without necessarily improving the teaching.

    The College I went to had no link with my House at school. No one from my house had ever been there in recent times. Once I was in the College though, others from my house also got places in subsequent years.

    Coulda been a coincidence though…

  259. plasecaha said,

    August 30, 2007 at 3:32 pm

    RS

    “How could you tell if only a fixed proportion got each grade? Are you saying there is additional data available?”

    The grades were applied retrospectively, of course. Each year, there would be a different curve from the raw exam performances. You could track that data over time.

    “Because the evidence that in the 1980s there was concern that grade boundaries were becoming too close together would tend to argue for a change in performance distribution (if we assume similar difficulty between papers) although not in what direction.”

    Exactly, there can be evidence of a change in the distribution. You could respond to that if you had to. If it got extreme, you could introduce a new grade, as nowadays…

  260. RS said,

    August 30, 2007 at 4:45 pm

    But do we know whether there is any evidence for or against changes in standards? Or could true increases in performance have been cancelled out by widening participation? I suppose the grades in Public Schools might serve as a proxy since their students presumably always did A Levels and had no widening participation issues.

  261. JQH said,

    August 30, 2007 at 5:14 pm

    Ben, re your request early on, I emailed you some questions from am old CSE physics text book. Closer examination suggests they were taken from past CSE papers

    Nobody’s left the recent GCSE physics paper lying around so I can’t send that for the mo. Mind you it should be available on the net.

    I don’t know about other schools but where I work only the students who are doing Higher level GCSE can get entered for the three seperate sciences ie the ones who would have been capable of O-Level in the old days.

  262. plasecaha said,

    August 30, 2007 at 5:34 pm

    RS,

    Now that’s an interesting idea. I need to think about that, lol. I’m also thinking about whether we should have seen a change in standards going way back, because the Flynn effect suggests we might have…

  263. scentless_apprentice said,

    August 30, 2007 at 6:09 pm

    Hi Ben,

    I can tell you from personal experience that it’s not the difficulty of content that has changed – it’s the structure of the paper that has changed.

    I compared the Mathematics exam papers I took in 1997 to this year’s at the school I teach at in June.

    What I found was that whilst the topics were the same and the expected outcomes were similar – what was different was that the answers were broken down into chunks in a way that would clearly help the student to follow a path to the correct answer.

    Only at the higher end A/A* questions was it expected that the student would do the complete working without direction or in chunks.

    So whilst the subject matter the students have to study isn’t being dumbed down (that said, my exam had matrices and critical path analysis, topics not on the GCSE syllabus these days), the way the questions are asked requires less deduction and planning on the student’s part.

  264. plasecaha said,

    August 30, 2007 at 7:31 pm

    OK, regarding the whole IQ thing, THIS is interesting…

    education.guardian.co.uk/schools/story/0,,1693061,00.html

    The work of the guy behind it was used to benchmark the National Curriculum, so Richas should approve…

  265. plasecaha said,

    August 30, 2007 at 8:01 pm

    Flynn disputes the above, apparently, on account of British pupils showing rises on the WISC test (though declining in arithmetic). But he thinks the effect may have peaked, so it should be interesting.

    Meanwhile, on the matter of grade inflation, most of the rises in the past five or ten years have been in the private/grammar school sector?

    education.guardian.co.uk/schools/story/0,,2150082,00.html

    So it’s been basically static in the state sector? Wow…

  266. NickConnolly said,

    August 30, 2007 at 11:39 pm

    “That doesn’t make any sense, because I wasn’t missing the point.”

    I was saying, in an apparently way, way too subtle way that you were missing the point. I may, of course, be wrong.

  267. NickConnolly said,

    August 30, 2007 at 11:50 pm

    “I agreed with that. I disagreed that the standard automatically had to be O level.”

    Then I haven’t a clue what you are saying. You agree that the in the good-old days A Levels did claim to be able to assess down to O-Level grades and distinguish that from a fail. You also agreed that not all question in the OCR AS paper were at O-Level standard.

    “OK, so we have your “awareness” to go on instead. Cool. ”

    Erm no. You have actual emprical data which I’ve posted which shows that the test have actually got easier for students. That’s the issue – do we go about looking at this question in a silly woo sort of way (your approach) or do we consider empircal data (my approach).

    “There are limits to what can be reasonably expected of grade boundaries, ”

    Undoubtedly but your complaint that some questions on the OCR paper were easier than those on the AQA paper wasn’t of the order of magnitude even by your own claims. Two papers can differ substantially in the difficulty of the items but so long as there is enough overlap grade boundaries can be adjusted to compensate OR even render one test easier than the other despite the apparent difficulty.

  268. richas said,

    August 31, 2007 at 12:28 am

    Oddly, you have overlooked that. I was making the point that we don’t know which way round, but that it is POSSIBLE there is a disparity between boards.

    Which would be why we have a regulator to check this and why the Exam Boards have a curriculum board they are all members of. They provide the mechanism to ensure that the exam boards award grades of similar (as close to identical as possible) standard.

    Do you think you are the only one who has compared questions on papers? They just do it with full information and statistical analysis and apply their professional experience to the task.

  269. richas said,

    August 31, 2007 at 12:43 am

    OK, regarding the whole IQ thing, THIS is interesting…

    education.guardian.co.uk/schools/story/0,,1693061,00.html

    The work of the guy behind it was used to benchmark the National Curriculum, so Richas should approve…

    Not really Piaget has 4 undefined stages of child development with no clear test for them. In fact they are nowhere fully described and nobody knows the boundaries so it would be startling if different researchers 30 years apart assessed them in the same way – you see they have no mark scheme, they have no definition and they have no transition criteria between the stages.

    Entrails I’m afraid.

  270. richas said,

    August 31, 2007 at 12:50 am

    So it’s been basically static in the state sector? Wow…

    No, relatively static in percentage terms. Improved significantly but less than at private schools. In terms of the total A grades awarded more of the extra As have been awarded at state schools. 93% vs 7% is the participation rrate overall but it is closer to 13%/87% at A Level.

    Plus we have a much wider participation rate now compared to a decade ago – in the state sector. The independent sector has been static in size. This wider participation could reasonably be supposed to have mostly included pupils less likely to achieve an A than those who traditionally took the A levels.

  271. NickConnolly said,

    August 31, 2007 at 4:03 am

    “Not really Piaget has 4 undefined stages of child development with no clear test for them. ”

    I wouldn’t let the Piaget aspect count too much as far as the actual test goes. The Piagetian developmental stages were the underlying concept between the items in the Voulme and Heaviness test. The test itself stayed the same and was analysed using a standard psychometric model.

    The interesting issue here is that the test Shayer used was one of practical reasoning about physical objects -essentially the kind of inner working folk model of physics that the children had. Shayer’s result indicate that had declined. That isn’t inconsistent with either Flynn’s results (on an attempted measure of g ) or school test results.
    “On the reasons for the decline reported in this article one can only speculate. Piaget
    believed that it was the whole everyday environmental experience of the child that
    drove cognitive development, with schooling possibly playing only a minor part in the
    process – Vygotsky believed that schooling should change to play a major part (Shayer,
    2003). Passive exposure to many hours of television a week has increased since the
    1960s when 1975 CSMS students entered primary school. Computer games may have
    usurped what might have been, for boys, many hours playing outside with friends with
    things, tools and mechanisms of various kinds rather than virtual reality. However, it is
    possible that a decline in the use of activity methods in the early years of primary
    schools, in favour of an increased proportion of the time dedicated to the 3Rs as
    instanced by the National Numeracy and Literacy projects, may be partly responsible for
    the continuing decline from 2000 to 2003.”
    (Michael Shayer British Journal of Educational Psychology (2007), 77, 25–41)
    In other words the decline in students ability to reason sensibly about physical objects and improvements in their ability to add, spell etc may have the same fundamental cause: Primary schools are doing more formal teaching of the 3Rs and less experiential learning. Twisted on its head we could also say that in the 1970s primary school would have been more likely to have explictly taught Piagetian concepts (eg conservation of volume) because Piagetian ideas were more in vouge: so the past results may have been because of “teaching to the test”.

    I can’t find a free-access version of the paper

  272. plasecaha said,

    August 31, 2007 at 10:50 am

    Richas,

    “No, relatively static in percentage terms. Improved significantly but less than at private schools. In terms of the total A grades awarded more of the extra As have been awarded at state schools. 93% vs 7% is the participation rrate overall but it is closer to 13%/87% at A Level.”

    The interesting thing is how static it has been for the secondary moderns, the ones which have had the brighter ones stripped out and sent to the grammars. At 0.1 percent over five years, that’s pretty static.

    They haven’t made any gains to speak of, the number getting As is frozen at around 10 percent. The teachers are not seeing more people move up from B grades to A the way they are at other schools.

  273. richas said,

    August 31, 2007 at 11:02 am

    The interesting thing is how static it has been for the secondary moderns, the ones which have had the brighter ones stripped out and sent to the grammars. At 0.1 percent over five years, that’s pretty static.

    They haven’t made any gains to speak of, the number getting As is frozen at around 10 percent. The teachers are not seeing more people move up from B grades to A the way they are at other schools.

    It is interesting but hardly surprising the potential A students are going elsewhere. What would be more relevant is their increase in B/C – The O Level equivalents where I am sure they have seen real progress. These are real gains for these pupils.

    One of the things that is often missed in the Secondary school type debate is that the best/worst schools make relatively little difference to the outcomes in terms of grades. Prior achievement and social/economic background are far more important. The value add league tables sow that a “good” school manages to imprve average performance by about two GCSE grades so of say 5 GCSEs Two subjects at B rather than C. Some of the “secondary moderns” are matching this value add – not many but some.

  274. plasecaha said,

    August 31, 2007 at 2:42 pm

    Richas,

    “Not really Piaget has 4 undefined stages of child development with no clear test for them. In fact they are nowhere fully described and nobody knows the boundaries so it would be startling if different researchers 30 years apart assessed them in the same way – you see they have no mark scheme, they have no definition and they have no transition criteria between the stages.”

    To add to what Nick said, there is also this:

    “He helped to develop two-year intervention programmes for those children who had been identified by the Piagetian model as being below average in year 7. Science and maths were the contexts through which the programmes were taught, but the prime focus was on general developmental skills.

    “These programmes [Cognitive Acceleration through Science Education and Cognitive Acceleration through Maths Education] both significantly increased the children’s Piagetian scores and markedly improved their maths and science GCSE grades from those predicted at entry level testing. More important, these children also showed an improvement on predicted grades in other subjects, such as English and history. This showed the programmes had a generic impact, rather than just a specific effect.”

    And this…

    “VH, which concerns the conservation of liquid and solid materials, internal volume and intuitive density, was chosen partly because it has substantial predictive validity for both science and mathematics achievement and is an effective way of alerting teachers to their students’ range of abilities, but also because it is recognised as a test that measures abilities that are not directly teachable. As such, it was an objective research method, free from any process of adaptation to changing circumstance.”

    I’m not the greatest fan of Piaget, but your sweeping condemnation of the approach of someone with good cred. contrasts with the way you blindly urge us to accept the work of the QCA because they are “professionals”, without any assessment of THEIR methods. But yeah, Piaget has a lot to answer for, including a whole bunch of teachers believing that children just learn stuff wen they are “ready” and that they therefore needn’t bother much about the teaching because it is immaterial.

  275. plasecaha said,

    August 31, 2007 at 2:43 pm

    Richas

    “It is interesting but hardly surprising the potential A students are going elsewhere.”

    They don’t all go elsewhere. They already have 10 percent A grades. Why are the B-grade students frozen, but not elsewhere? Why can’t the teaching and the wonderful new criterion-based exams lift them, if it does elsewhere?

    “One of the things that is often missed in the Secondary school type debate is that the best/worst schools make relatively little difference to the outcomes in terms of grades. Prior achievement and social/economic background are far more important.”

    Statistically, that may be true, but we have to be careful. Private schools have had a fair number of people from much less well-to-do backgrounds and much less prior attainment than their peers (because they came from crappier schools) doing as well as or exceeding the performance of their new peers.

  276. richas said,

    August 31, 2007 at 2:50 pm

    I’m not the greatest fan of Piaget, but your sweeping condemnation of the approach of someone with good cred. contrasts with the way you blindly urge us to accept the work of the QCA because they are “professionals”, without any assessment of THEIR methods. But yeah, Piaget has a lot to answer for, including a whole bunch of teachers believing that children just learn stuff wen they are “ready” and that they therefore needn’t bother much about the teaching because it is immaterial.

    I’m sorry if my comments read as a sweeping condemnation of Piaget, they were not intended as such. His focus was on development and stages of development. This is very different to assessment. Twisting his work to create an assessment framework just does not stack up. It would be unclear to Piaget what was being assessed, its relevance and the measures to be used would be unclear even to Piaget.

  277. richas said,

    August 31, 2007 at 2:59 pm

    They don’t all go elsewhere. They already have 10 percent A grades. Why are the B-grade students frozen, but not elsewhere? Why can’t the teaching and the wonderful new criterion-based exams lift them, if it does elsewhere?

    For the last time criterion referencing does not raise grades it is just a mechanism that allows improved performance to be recognised over time.

    I suspect a debate on secondary moderns is a bit much on top of the rest, suffice to say, as you would no doubt expect, I am not a fan. Personally I suspect it has to do with lower ambitions within the school, both teachers and pupils plus lower parental support and ambition.

    Statistically, that may be true, but we have to be careful. Private schools have had a fair number of people from much less well-to-do backgrounds and much less prior attainment than their peers (because they came from crappier schools) doing as well as or exceeding the performance of their new peers.

    Presumably because we know the scholarship exam is all about finding the most deserving not the poor kid with most ability and the highest achievements to date?

    Sorry this very select group, often with very education focussed parents, is not representative. Again – the school can make some difference but it is relatively little – even in the independent sector. Their excellent results are a product of excellent teaching, low class sizes, additional resources and the quality of their intake – both in terms of pupils and parents.

  278. Robert Carnegie said,

    September 1, 2007 at 1:06 am

    Lower down the school, aggregate performance continues to decline in basic reading and writing, and that’s official. So far the cries of “Pah! It is only because the tests were made harder” have not yet reached my ears.

  279. kim said,

    September 3, 2007 at 1:30 pm

    Ben, three years ago Jenni Russell wrote an article for The Guardian in which she demonstrated fairly conclusively that standards had fallen in A-levels (ie not necessarily that the questions were getting easier but that the marks required to pass were lower). Definitely worth reading:
    www.guardian.co.uk/comment/story/0,,1287052,00.html

  280. Duck said,

    September 4, 2007 at 7:56 pm

    In 2000, my secondary school only offered Dual Award Science. We were told at the time that this was not sufficient for taking science A levels – and we were not allowed to do 3 separate sciences. Grrrr. Fortunately I explained this to 6th form, got A*A*, & went on to 3A’s in Science A levels. However, people who might have been able to increase their scientific literacy by doing a Science A-level alongside humanities wouldn’t have been allowed to do so because the GCSE teaching was so dumbed-down. I spent most GCSE Science lessons locked into the chemicals store (mmmmm, fumes & formaldehyde dissected rats), reading 30-year-old copies of Nature by the light of microscope torches, ‘cos that was the only way to avoid getting beaten up by the other kids who were causing a riot. Seriously, I am not exaggerating.
    My old secondary school, Bassingbourn Village College, now has some sort of specialist status as a technology college. Hah. Money-grabbing scum.

    I took GCSE Maths in 1999 (early), & my Dad who is an Electrical Engineer was actually quite impressed by some of the content, particularly the stats. Apparently I was doing stats as a 13-year-old that he’d not touched ’till uni. It missed quite a lot of things he had covered at O level, but did give a useful grounding in understanding things like probability which are important to understand a lot of news stories & generally navigate life. GCSEs are the last time in the UK that you can *make* anyone learn stuff, so maybe it’s as well to gear them to ‘public understanding of science’, and assume that future scientists, who will make up a very small percentage of those sitting GCSEs, can learn what they need at A level and beyond.
    The other bit of GCSE Maths I found particularly good was the coursework – I had to undertake an independent investigation into a mathematical problem – my parents were impressed & hadn’t done anything like that at O level.

  281. richas said,

    September 5, 2007 at 11:06 am

    There are several points in Jenni Russell’s article that are worth focusing on:
    1. The inability of the current A-Level system to discriminate amongst the top 10% of students. This is demonstrably true.

    Except of course that only 3% of the cohort get 3 As and in future the universities will have access to the full marks and so would be able to choose the top A level scores. Having said that it is by no means clear that the raw A level scores are the best way to choose who gets the place – motivation, ability, social background and quality of the school the grades were achieved at could all be more relevant.

    A levels are not only about allocating a few thousand Oxbridge places a year, nor should the whole system be geared towards this niche.

  282. Squander Two said,

    September 5, 2007 at 2:02 pm

    I was in the second year ever to take GCSEs, which means that we used O-Level past papers for practise. The drop in difficulty was obvious to me and every one of my peers about ten minutes after walking into the exam room.

    Sure, that’s anecdotal, but hey.

  283. NickConnolly said,

    September 5, 2007 at 9:16 pm

    “Except of course that only 3% of the cohort get 3 As”

    Good point. The exams in the IB Diploma aren’t neccesarily harder than A-Levels (Higher Maths is probably overall easier by itself). The reputation the IB has for rigour and for good preperation for Uni is the overall demands it makes on students.

  284. meo100 said,

    September 13, 2007 at 6:51 pm

    Can i just clear up the Scottish education system thing, as I think there’s confusion on all sides. jcm07, its not true to say that Scottish kids spend an extra year at primary then do 6 years at secondary. Its actually quite complicated to compare the systems for a number of reasons to do with different timings and nomenclature apart from anything. Anyway here goes.

    The cut off age for going to primary in England/Wales is to be 5 years old by September. In Scotland, you have to be 5 years old before the following March the 1st, so the ages of children in what is called ‘primary 1′ in Scotland is from 4.5 to 5.5 at the start of the school year (in August). In England, the children will range in age from just turned 5 to just about to turn 6. However, even this comparison is not easy because in England some (or maybe all, I’m not sure) schools have a reception year, followed by 6 years (called years 1 to 6). This makes a total of 7 years at primary school. In Scotland, there is no reception year, they just call it ‘primary 1′ to ‘primary 7′. I think this is where some confusion lies, because English people hear the term ‘primary 7′ and assume that Scottish children have spent an extra year at primary. Scottish children will be aged between 11.5 and 12.5 when they go to secodnary. I think this is on average 6 months older than English children going to secondary, not 12 months older.

    Another source of confusion is the number of years spent at secondary. In the Scottish system, there is no equivalent of what is currently called ‘year 9′ in England (that’s 3rd year in old money). Scottish kids will sit their Standard Grades (GCSE equivalent) at the end of 4 years at secondary (called ‘4th year’), whereas English kids sit GCSEs after 5 years (year 11, or 5th year in old money). Now, having skipped this extraneous year the Scottish kids having done their Standard Grades are, on average, 6 months younger than English kids having done GCSE.

    After GCSEs and Standard Grades its time for AS levels (in ‘year 12′ in England) and Highers (in ‘5th year’ in Scotland). Contrary to what’s been said before, the brightest kids in Scotland will get their 5 highers and go off to uni without bothering with 6th year. Most kids in 6th year are either upgrading Highers they didn’t do so well on, or doing more Highers. Its actually quite difficult to muster up enough pupils to even offer 6th year studies (SYS) in most subjects. Teachers are usually really pleased if they have some pupils interested in doing SYS, and they certainly don’t run SYS classes in all subjects every year in most schools. The reason for the low interest in it is because SYS qualifications are no great advantage to getting into uni. Scottish universities want to see your Higher results, SYS isn’t that relevant unless you want to skip first year, which is actually very rarely done. In fact, some teachers advise pupils against doing SYS as its designed to be taught in a lecture and project way (40% of it is project based), a bit more like university teaching than school teaching. Pupils have to be very self motivated, especially as they probably already have the Highers they need to get into uni. If you have great Highers and you bomb out on SYS because you lacked the discipline to push yourself, it looks really bad on your application to uni the next year. Some kids just want to do 6th year because otherwise they feel too young going to uni – at the end of 5th year, Scottish kids are aged between 16.5 and 17.5. The youngest won’t turn 17 till the following February, half way into their first year at uni.

    As for the first year of uni being A-level catch up, that’s quite an oversimplification. A2-level teachers have a varied crop of pupils, and have to teach to the middle to make sure the C and D grade students don’t get left behind. Compare this with a first year cohort in a Scottish university – a self selected, much smaller group who all have As or Bs at Higher, and are, by definition, extremely motivated to do that one subject. University teachers expect far more out of their students than school teachers, and generally get it too. The lab equipment is far superior too, and expectations of lab reports bear very little relation to what’s expected at school. You’re also expected to behave like an adult and be responsible for your own learning right from the off, far more so than if you were still at school.

    I think each of these systems has its relative merits – I think year 9 is a waste of time for most pupils, but going to uni when you’re only 16 is a bit too young.

    There are a couple of regards in which I think science teaching can be better in Scotland though. 1. Science is taught as a practical subject so there is a statutory class size limit of 20 pupils per class. This applies from 1st year at secondary onwards.
    2. Teachers must be qualified to university level to teach a science subject beyond 2nd year at secondary, so e.g. physics teachers must have a degree in physics to teach from Standard Grade upwards. (Or some related degree such as engineering which had enough physics in it to satisfy the stringent demands of the Scottish General Teaching Council)
    3. Teachers in Scotland have a quite generous statutorily protected minimum number of non-contact periods every week. These are invaluable for preparing for practical lessons. In England, although teachers’ timetables included non-contact periods, they are not protected and you can be given cover lessons if other teachers are off and they can’t get in enough supply teachers. In my experience of teaching in England, this happens a lot, and you can be taken off to cover just when you had earmarked a free period for setting up practicals. In the end, you just end up demonstrating the experiment, and bang goes another chance for the kids to do something hands on.

    Having said all that, the decline in kids doing A level science subjects is mirrored in Scotland with a decline in kids opting for Higher sciences, so maybe it doesn’t make a blind bit of difference.

  285. diudiu said,

    December 21, 2009 at 6:06 am

    links of london links of london
    links london links london
    links of london jewellery links of london jewellery
    links of london sale links of london sale
    links london sale links london sale
    links of london bracelet links of london bracelet
    links of london charms links of london charms
    links of london necklace links of london necklace
    links of london bangle links of london bangle
    links of london earrings links of london earrings
    links of london ring links of london ring