Lotteries instead of point-scores on exams: A great quote from Peter Stone

A story in the Irish Times (25th Feb 2021). This is a paywalled link. The full text of the article appears below.

The Leaving Cert is not fair. Why not just replace it with a lottery?
Joe Humphreys
Unthinkable: We can no longer plead ignorance of the inner workings of our State exams

‘I think recognizing the wider role luck plays in society is very important,’ says TCD political scientist Peter Stone.

The Leaving Cert has had an untouchable status in Irish life. It may be a brutal memory test but it is our brutal memory test – a rite of passage nearly as old as the State itself.

In the past 12 months, however, the bonnet has been lifted on this national treasure and we can no longer plead ignorance of its inner workings. The attempt to manufacture a distribution of grades under pandemic conditions equivalent to those produced by the annual exams has spotlighted long-running questions of fairness.

As a test of ability, the Leaving Cert is fair in the narrow sense that a bobsleigh race between Jamaica and Norway is fair. Contestants do not start with the same advantages, and the format – which lends itself to a parallel grinds industry – gives an extra edge to students from better-off families.

However, there’s a second matter of fairness surrounding the appropriateness of using the Leaving Cert to determine who gets what college places. This must be considered against the backdrop of stark figures showing that, on average, a third-level graduate earns much more over her or his lifetime than a worker who doesn’t have a degree – at least €100,000 more, according to one conservative estimate.

If we are to believe that the process is fair, are we to then accept that middle-class children are smarter than those from working-class areas?

Disagreement among philosophers over what exactly constitutes fairness might encourage us to think of it as a spectrum rather than an end point. In the case of the Leaving Cert, “the challenge in terms of alternatives is that most forms of assessment are subject to the sort of gaming” associated with the annual exams, says Carl Cullinane, head of research and policy at Britain’s Sutton Trust.

“The goal of a ‘tutor-proof’ test is something of an unattainable holy grail. This is a major issue in the UK, particularly in regard to the 11-plus exam, which is subject to high levels of private tuition. But it has yet to be cracked.”

Dr Shane Bergin is critical of the Leaving Cert on pedagogical grounds. The exams do not assess “in any meaningful way” critical thinking or problem-solving, he argues. But the Leaving Cert “does offer a good deal of reliability. Students in Dingle take the same exam as those in Dublin and each script is graded centrally, and anonymously, by expert examiners.

“There are, however, limitations to this approach,” says the assistant professor in science education at UCD. “The examination process shows little flexibility around when students can sit the exam. When grading certain subjects, marking schemes can be quite prescriptive. When we look at the breakdown of exam results, we see various factors such as socioeconomic ones are linked to outcome.

“If we are to believe that the process is fair, are we to then accept that middle-class children are smarter than those from working-class areas?”

As for using the Leaving Cert as a college filter, Cullinane says, “an exam-based system for university entry, in the context of an unequal society, is likely to have adverse impacts on groups with a lower status, even in the absence of bias… Much of this is down to a focus on the relative ranking aspect of examination systems.”

Education, he explains, “can be said to have both an absolute value” – the merit of learning itself – “and also positional value”. The latter is “about how it places an individual in relation to others”.

I think there is a bit of an element of ‘be careful what you wish for’ when it comes to moving away from the Leaving, as it relates to university entry.

“The upshot of this is that in every exam, some fail in order for others to succeed. The meaning and value of a high grade is inseparable from the fact that others have achieved lower. Given that even a ‘good’ exam will replicate existing social inequalities, this means that those achieving high marks each year, largely from socially advantaged groups, are inherently defined as successful in relation to others: those who achieve the lower end of the distribution, largely from socially disadvantaged groups.

“Given the benefits in terms of educational and career progression that accrue to these grades, this is somewhat troubling, and raises the question of how we might better serve those who ‘fail’ these high-stakes exams.”

But what’s the alternative? US universities often use a system of “holistic admissions”, but the weakness of these have been exposed by recent bribery and cheating scandals, Cullinane says. “So I think there is a bit of an element of ‘be careful what you wish for’ when it comes to moving away from the Leaving, as it relates to university entry.

“My personal view is that combining a national exam system with thorough contextualised admissions is likely to be the fairest way of distributing university places.

“Exams are likely the best ways to minimise bias, while the adverse impact on certain groups can, at least to some extent, be alleviated by taking into account the different barriers faced by these groups in achieving those grades. The Dare and Hear schemes in Ireland take this approach, with reduced points requirements for those meeting certain criteria. Scotland also provides a very interesting, albeit recent, example of a more systematic national approach.”

Calling for more fundamental reform, Harvard philosopher Michael Sandel believes the issue can’t be separated from “the tyranny of merit” – which happens to be the title of his latest book. Meritocracy is not a remedy for inequality but a “justification” of it, he argues, and he promotes an idea mooted by several political theorists of setting a threshold of qualification for college and letting chance decide the rest.

Such a lottery system “is not a panacea”, Sandel told The Irish Times last year, but he hoped it would force people “to confront questions about meritocratic hubris and the role of universities as arbiters of opportunity”.

Dr Peter Stone at Trinity College Dublin advocated such a model for years and has written books and papers on the value of lotteries in decision-making.

“I’ve often suggested one could divide university applicants into three categories: well-qualified, qualified and unqualified. You could probably admit everyone in the first group, and of course you’d reject everyone in the third group. But you’d still have a lot of people in the second group, more than you could admit, and there a lottery would make a lot of sense.

“There may be close decisions when it comes to assigning people to groups, but my hunch is that there would be many more black-and-white decisions, easy to make.”

What would he say to a student who lost out in a lottery but who would have gotten their college place if marginal gains were rewarded?

“Precise numbers are irresistible,” he replies. “It looks so clear that if your number is 515 and somebody else’s is 510, or even 514, that you deserve to win. But we all know that these measures are quite rough and ready – the difference between a 510 and a 515 may not be much better than statistical noise.”

For Sandel, part of the attraction of a lottery for college places is its potential for conditioning us to acknowledge the wider role luck plays in personal success. It forces us to ask, for example, whether you deserve the rewards that accrue from your physical or genetic inheritance.

If you succeed in life, you probably earned it, but the person who came in second behind you – who has a much less glamorous life as a result – probably earned it just as well

“I think recognising the wider role luck plays in society is very important, especially – and this may sound a bit paradoxical – in a meritocratic society,” says Stone. “If you have a race for stakes that matter, people are going to train hard for it, and if they do, the difference between first and second place may really come down to luck.

“There’s a saying that people usually deserve what they get, but they often don’t get what they deserve. In other words, if you succeed in life, you probably earned it, but the person who came in second behind you – the one who has a much less glamorous life as a result – probably earned it just as well.”

Bergin believes these issues could be explored through a citizens’ assembly for education, which has been promised in the Programme for Government.

“In recent times I’ve read the line, ‘you cannot propose to change the Leaving Cert unless you have an alternative.’ I think such statements are really unhelpful. Those of us who work in Irish education are creative and caring. It is not beyond us to ask questions of how fair our current system is and for us to imagine how it might be better and fairer.”

8 Responses

  1. Thanks for posting this. I once proposed a similar mechanism for a completely different reason. I was chairing the Australian Government’s “Innovation Australia Board” which was the ultimate decider of which Australian firms won various innovation grants and benefits from government.

    Amid much talk of evidence-based policy (always much talked about, very little pursued) I pointed out that we really hadn’t much idea whether any of our programs achieved anything. The firms receiving the grants grew faster than the average firms, but then they’d been selected for benefits as firms with high potential for growth. They were firms with nifty ideas and technologies they were pursuing – or were so in the opinion of the assessors from the relevant department.

    All the data in the world wouldn’t tell us what we wanted to know. Because it wasn’t causal data. We needed to give some firms the benefit and deny it to others and compare how they each fared. So, in the interests of efficiency, I proposed precisely what is reported here – even though it was proposed in the interests of fairness.

    In fact, the most efficient thing to do from the perspective of finding out the impact of the program would be to randomise all grants, (at least those that met a certain threshold) but that wouldn’t fly politically. So I suggest that there be definite approvals, definite rejections and a class of qualified firms who would receive grants or otherwise at random. Then we could compare the way the two populations of medium performers performed after getting (not getting) the benefit.

    That seems interesting in itself, but it also uncovers an additional benefit of what’s proposed in the article. In addition to not going overboard on ‘meritocracy’ it gives us better data by which we can understand the effect of giving people better access to education.

    Liked by 3 people

  2. The basic issue that causes the supposed “merit-vs-luck dilemma” – that the supply of quality education is limited – is left unaddressed, as if it is an inevitable constraint. Shouldn’t that be the first thing to discuss?


  3. It should certainly be discussed. But there will always be capacity constraints of various kinds, so discussing it and loosening the constraints may well be a good thing to do. But the question of the relation of merit to constrained opportunities can’t be ignored. And it’s toxifying our world right now – so we can do better even if there are no perfect answers.


  4. In some cases (like the grants situation you describe above) the capacity constraints may be inevitable. However, specifically in the case of education the constraints on capacity are much more of a matter of policy than anything else. Arguing against the hidden assumptions behind this policy may be easier (as well as more effective in terms of its implications) than arguing for “luck over merit”.


  5. Yes Yoram, the supply of quality education is limited, but it’s also a ‘positional good’. Peter Stone’s university TCD is more prestigious than my old uni UCD, and that makes the difference.

    Liked by 2 people

  6. Did the board take up your suggestion, Nicholas?


  7. It was a government board. I’ll let you imagine the answer ;)

    Liked by 1 person

  8. I did an engineering school in France. We were 250 in total, one dark-skinned person, and me very ashamed of myself and my country.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: