Monday, 17 August 2020

Locking disadvantage into the system

Last week’s ‘A’ Level results fiasco reminded me of my own experience of ‘O’ Level results back in 1967. Shortly after we got the results, my French teacher stood in front of the whole class and said that she really didn’t understand how I’d managed to get a grade 3 in French whilst a fellow pupil (whom she also named) only got a grade 6 pass. “I’d have understood it better if the results had been the other way round”, she told us. Being charitable, she was probably trying to boost the confidence of my disappointed peer (and it may well have been fair comment anyway!) but my gratitude for her faith in my ability wasn’t exactly unbounded. Perhaps my fellow student had a bad day and I had a good one; perhaps the questions on the day were simply a better match for what I’d remembered than what he’d remembered; perhaps I was just better at sitting exams – I’ll never know how it happened, only that had my grade been based on teacher assessment rather than examination it would have been lower. (And, had that been repeated in other subjects, life could have turned out very differently.) The point is that teacher assessment is no more perfect a method of assessing ability in a subject than an examination. Both can create anomalies and the two methods will always produce different results for at least some of the pupils. Which result is the fairest is an open question, and our faith in the reliability and accuracy of the system as it impacts individuals is seriously misplaced even in a normal year.
We know for certain that examination performance varies between schools, and if results based on teacher assessment reduce or eliminate those differences, then they are not reflecting accurately what would have happened had the exams been held. (I should note, in passing, the implicit assumption in that statement that the exam results are the accurate ones, itself an assumption open to serious question.) That, in effect, is the justification for making ‘adjustments’ to teacher assessments. They are an attempt to reflect historic differences in performance between schools and, in fairness, they may well have more-or-less achieved that at an overall level. But, by being based on a statistical approach, we can be absolutely certain that the individual pupils whose scores were thus ‘adjusted’ from teacher assessments would not always be the same pupils whose scores would have differed from those same assessments had the exams been held. Reducing it to a mathematical exercise might produce the ‘right’ averages, but it can never produce the ‘right’ results for individuals. It’s an approach which is fundamentally flawed.
We also know the main reason that some schools regularly see lower results on average: they serve poorer communities. We have known for decades that there is a very strong correlation between academic performance as measured by examinations on the one hand and parental income on the other, and none of the actions taken to try and address that have been particularly successful. (I suspect that is primarily because they all set out to address the symptoms rather than the cause in an attempt at some sort of short-term fix, but that’s a subject for another time.) That difference in performance may be well-known and well-established but it isn’t, and never has been, fair. There is something particularly grotesque about a Labour government here in Wales trying to ensure that historical disadvantage based on relative affluence is properly reflected in this year’s results. They are effectively locking in that household income based disadvantage for another whole cohort of young people based on historical results for the schools they attend without the individual members of that cohort having even the limited opportunity to buck the trend which the examination system provides, and from which at least some would have benefited had the exams gone ahead.
There is no perfect solution to this year’s issues (and even less is there an easy fix for the real underlying long-term problem), but applying an algorithm specifically designed to perpetuate the injustices of the past is about the worst solution which anyone could devise. The (eventual and belated) Scottish decision to simply accept the teacher assessments isn’t a perfect one either (the idea that hundreds of teachers in hundreds of schools could ever be grading students precisely and consistently is laughable), but, coupled with a robust appeals process, it’s probably the least worst option in the circumstances. I really don’t understand why ‘Welsh’ Labour prefers to follow an only slightly adapted version of the approach of the English Tories instead.


dafis said...

The entire " non exam results" fiasco reflects the deplorable mindset of the vast majority of our political/ruling elites. 2020 was a year of big crisis, that much is true. the shallowness of the thought given to this problem is evident in that they designed an algorithm, a "solution" which seldom works when exploring hypotheses, so seriously deficient when tackling a set of highly diverse variables in the real world.

Serious abdication of responsibility, no pass just fail, try again. But our governments are so thick skinned and fluent with lines of apology and excuses that this will be forgotten within weeks.

It is arguable that despite cost and inconvenience all students should have been offered a free repeat year so they could punt for their qualifications with a full set of tuition experiences to support them, and 2020 could have been written off the records.

Spirit of BME said...

I think your last paragraph sums this very unhappy place that might impact those effected for their lifetime as in some minds the corvid class of 2020 will be written off as dodgy against other years. However, we are, where we are, and I do not have enough information or experience in these matters to put forward a fix. Sadly, it`s all self-inflicted as you and I went through two pandemics where education did not shut down and the last (1968/9) had an estimated 80,000 deaths based on the current population numbers.
I was interested in your French ‘O’ Level experience, as in my ‘O’ level History exam in the early 60`s, the Head Master advised the Assembly that the Examining Board had advise that my I had come third highest in Wales. Great laughter ensued as everybody knew that although I enjoyed the school, I was total tosh academically and bored out of my brains with the curriculum.