The problem of averages

Tweet

Averages and aggregated data, in general, are a useful way of comparing and presenting data, but they can often hide an important level of detail which makes them dangerous as a basis for setting policy, particularly when they end up being interpreted as though all the individual items are at the ‘average’ level. One example which immediately springs to mind is the regular comparison between average spend per pupil in Welsh schools and average spend per pupil in English schools, a comparison which reflects poorly on Welsh schools, usually indicating a ‘deficit’ of around £600 per pupil per year. Now there are a variety of reasons why there might be a gap, but what such ‘averages’ hide is that there are some Welsh schools which receive more than the English average and some English schools which receive less than the Welsh average. More importantly, whilst the gap between the English and Welsh averages can be ‘solved’ by simply giving £600 per pupil per year extra to all schools in Wales (as some politicians have called for), that won’t address the differences between funding levels when comparing schools within Wales (or within England). It’s addressing the data rather than the problem, but numeracy does not seem to be a particularly strong trait amongst politicians.

We’re seeing another example of this at the moment in relation to the pandemic, with claims by English government ministers that the UK is now ‘past the peak’. It’s one of those statements which is ‘true’ but which isn’t the whole truth. The fact that the UK at an aggregate level is now seeing reducing numbers doesn’t mean that all parts of the UK are seeing reducing numbers, either geographically or in terms of sectors (such as care homes), and taking UK-wide decisions on the easing of the lockdown on the basis of an aggregated data set means that lockdown could be eased at a time which is still too early for any areas or sectors which have yet to pass the peak. It could exacerbate the rate of infection in those areas as a result. Yet that is exactly what the PM’s rhetoric suggests he wants to do. The annoyance with which more cautious statements from the First Ministers of Wales and Scotland are greeted is another indication of the way in which the overall situation is being wrongly interpreted. The First Ministers are right to make it clear that they reserve the right to act differently if the situation in their respective countries is different; I hope that they will also understand that the situation might vary within their countries as well as between them. The same distinction also applies in England – just because London’s figures are falling fairly consistently does not mean that all parts of England are past the peak yet either; they just don’t have First Ministers to stand up for their interests.

When the rate of transmission varies across a country it inevitably causes problems in taking decisions – should we wait to ease the lockdown until all parts are on a clear downward trend, accepting that that will be later than necessary for some; should we ease the lockdown as soon as the overall picture improves sufficiently, accepting that it will be too early for some; or should we vary the decision according to circumstances in different areas? None of these are ideal – all bring real problems, both economically and socially, to say nothing of the impact in terms of sickness and death. The PM’s instinct seems to be to use the aggregate figures as his guide and apply policy across the board; I just wish I could be confident that he and his government were doing that on the basis of an understanding of the compromises involved rather than simply an inability to understand what averages and aggregates are telling us.

Averages don't tell the whole story

Tweet There is an obvious danger that anyone objecting to Cameron’s insistence on continuing intervention in discussion on the Welsh NHS is seen as being a supporter of a service which is failing.  And indeed, some of the responses coming from the Welsh Government do make it appear that they are more concerned with the politics of the matter than with addressing the very real problems that the NHS faces.  The fact that Cameron’s intervention has more to do with politics than with health is no excuse for responding in the same vein.

Part of the problem is that there’s a great deal of heat, but not a lot of light.  Last week, Cameron again drew attention to the difference in waiting times for diagnostic tests, where 30% of people in Wales wait for more than 6 weeks, compared to only 1.5% in England.  In purely numerical terms, that’s a shocking gap between two neighbouring countries, but there’s much more to this than averages can reveal.

From the perspective of each and every individual patient, whether they’re part of the 30% in Wales or the 1.5% in England is irrelevant; the figure for them as individuals is 100%.  And given the small size of Wales compared to England, even on those figures there are almost as many individuals waiting longer than 6 weeks in England as there are in Wales.  If it were really the best interests of patients driving Cameron’s concern, he could halve the total number of people waiting more than 6 weeks in EnglandandWales by eliminating the 1.5% in England, where he does actually have direct responsibility.

There’s another thing about averages as well; whilst they tell the headline story, they hide the detail.  These are averages for all specialities across all locations; there will be a lot of variation in there.  Are there no specialities in particular locations in England where the lists are longer than for the same speciality in some locations in Wales?  I don’t know; but I do know that overall averages will never tell us that.

And how about urgency?  A 6 week target for all tests of all types is nice and easy to shout about, but as a patient, there are some tests that I would want to have immediately, and others that I wouldn’t mind waiting a while for if priority was being given to those in greatest need.  Again, an overall average will never tell us anything about that; yet from a patient perspective, it’s outcomes which matter more than anything.  If Wales could achieve a much lower average (and thus undermine Cameron’s argument) by concentrating on doing the least urgent cases first (if, for instance, they were easier to do), would anyone really think that was the right thing to do?  Fixing the numbers isn't the same as fixing the problems.

Let me stress this: I don’t know the answer to the questions I raise above about the detail underlying the averages.  But I doubt that Cameron does either.  Worse, I’m not sure that he really cares; he’s only after making a political point which has more to do with winning English constituencies in the upcoming election than it does with care for the wellbeing of Welsh patients.

But the underlying problem here isn’t the politics of health, nor the way that politicians use statistics (although it would help if some of them were a little more numerate).  It is that, in Wales as in England, there is a target-meeting approach to managing the health care system, and as long as that continues, the Welsh Government is inviting this sort of criticism. 

Whilst we’d all like to be able to have whatever tests we need ‘on demand’, what really matters to patients is whether we get the tests (and treatment) we need at a time which makes a difference to the outcome for us.  In some cases, that may well be very much less than 6 weeks; in other cases, it could be longer.  Do we want our doctors to be deciding what to do on the basis of meeting a simplistic target to keep the politicians happy, or on the basis of meeting the clinical needs of patients?

Numeracy in the media

Tweet Maths, and numeracy more generally, is not everyone’s strongest suit, to say the least.  It’s an area of education which is – quite rightly – causing a great deal of concern in Wales at present; and one which journalists and reporters often cover.  Sometimes though, they don’t display much of an understanding of the issue themselves, leading to alarmist statements based on numbers, the statistical significance (or lack of) of which is rarely explained.

Take the headline earlier this week about a huge increase in immigration (exclusively referring to immigration from outside the UK of course) into Wales.  Merthyr Tydfil, we were told, had the second largest increase in immigrant numbers of any local authority in the UK.  Between 2001 and 2011 the increase was an enormous 227% conjuring up an image of hordes of foreigners flooding into the town and its surroundings. 

Gareth Hughes has already pointed out some of the failings in this use of the figures.  In 2001 there were 807 foreign-born residents in Merthyr Tydfil, by 2011 that that had increased to 2641.  That amounts to only 9% of the total population, and as Gareth points out, gives a completely different picture of what’s happening.

The question is whether this is deliberate scaremongering by those writing the headlines or whether it displays the very problem which they often refer to when dealing with mathematical and numeracy issues in education.  I’m afraid that I tend to the latter interpretation.  Still, at least it justifies their concern about the nation’s grasp of numbers.