Conall Boyle on university admittance: (1) why lotteries?

Posted on March 26, 2011 by Yoram Gat

I do not for one moment disagree with the principle that Merit alone should determine university entrance. Rather it is the form of merit used that I would disagree with.

Conall Boyle, Lotteries for Education

In Lotteries for Education Conall Boyle presents a case for using lotteries to supplement standardized test scores as the criterion for admission to universities. He first informs us that it is an empirical fact that such test scores (somewhat inconsistently, I think, covering both IQ tests and subject area exams) are not only the best predictor of university academic performance and graduation rates (explaining about 50% of the variance), but the only predictor of any validity (interviews and extra-curricular activities, for example, having no predictive power at all). Having made this point, Boyle sees it as his main task to convince his readers that having standardized test scores as the only entrance criterion should be avoided.

This task Boyle approaches in various ways throughout the book. In the ultimate chapter three arguments are presented:

A lottery is a “practical and efficient” way to handle borderline cases. That is, it is an easy way to differentiate between applicants whose scores are identical, or are so close that differences in their expected academic performance are negligible.
Accepting the top-scoring quota every year creates “inter-temporal unfairness” in the sense that the cutoff point will fluctuate from year to year. That is, a student with score x would be admitted one year, but another student with an identical score would not be admitted the next year.
“Balancing risk”: Boyle argues the risk of accepting students who fail to graduate should be balanced against the risk of the students who are not accepted but who would have graduated had they been accepted.

Of those three arguments, I find only the first one to be convincing.

The cutoff score should fluctuate only to the degree that the test scores are temporally inconsistent, or to the degree that the demand or supply of slots fluctuates. If the scores are temporally inconsistent, then there is no reason to see the situation Boyle describes as being unfair. A nominal score x in one year is simply not the same score as score x in the next year. And if the supply or demand for slots change, then that could be seen as causing or at least reflecting a change in the value of those slots. Again, in such a situation it is not clear that a fluctuating cutoff score is evidence of inter-temporal unfairness.

Another counter-argument – specifically aimed at supporters of lotteries – would be that the temporal variations can be seen as a form of a natural lottery. The winners are those who happened to apply in a year in which the cutoff score was low, the losers applied when the cutoff score was high. Why would lottery supporters find this objectionable? And why should introducing more randomness to the process by using an artificial randomization device be considered an improvement?

As for balancing risk – Boyle does not offer a rigorous exposition of this argument, and I doubt that it can be made rigorous. Given a fixed number of slots, increasing the admission chance of a certain applicant reduces their risk, but increases the risk of other students. Overall, if we assume with Boyle that high scores indicate high chance of graduation then increasing the admittance chance of a low scorer increases the overall risk (of rejection of a would-be-graduate) of the entire applicant set.

With the only remaining argument being about efficiency in marginal cases, it seems there is little reason not to rely solely (or effectively solely) on test scores. Yet I think there are strong arguments against such a mechanism. It is Boyle’s implicit assumptions that prevent him from utilizing the full force of those arguments, despite hinting at them at various points in the book.

Filed under: Books, Distribution by lot, schools |

5 Responses

Conall, on March 29, 2011 at 5:42 am said:

A very fair summary of the case I made for lotteries in university selection.

Points 1 and 2 – borderlines and inter-temporal fairness, even if accepted would make little difference overall, because by definition few applicants would be affected. But it does at least show that a lottery is a reasonable mechanism to use, not an outrageous novelty to avoid at all costs. This may seem a minor victory, but it is important.

It is the third point, the ‘balance-of-risks’ which introduces some form of universal lottery entrance. Hence the whole population becomes involved, which turns it into a form of sortition. I anticipated that this would be a difficult ‘sell’. A particularly tricky aspect is the different sorts of risk/probabilities which are mixed together: for example probability of success given that a grade X has been achieved; and, risk of rejection in a lottery. You introduce yet another in ‘the natural lottery’ of year-on-year variations in numbers of applicants.

I am more of an engineer than a philosopher, so I looked for a practical explanations and working examples. ‘Balance-of-Risks’ is well understood in Acceptance Sampling which prevails throughout industry. The example of the Netherlands medical school entry which has run for decades without detriment to the effectiveness of the professions involved shows that it works.

This may not be the ‘rigorous exposition’ that Yoram craves, but so what? Public Policy is often formed by the siren voices of silvery-tongued advocates. Rigor requires throwing away much of the messy but vital detail of real existing systems. University entrance by lottery as the Dutch do it happens to work, has stood up to public scrutiny and is popular. Sortition in Action!

LikeLike
peterstone, on March 29, 2011 at 7:18 pm said:

Embarrassed to say I haven’t read the book yet, but I would think that a lot hinges on whether scores are being used (if I may borrow terminology from Willem Hofstee) for purposes of selection (i.e., admit the n best students) or admission (i.e., admit every student with a score over x). Lotteries might play a big role in the latter case, if there were many students over the bar and not enough slots for all of them.

LikeLike
Yoram Gat, on March 30, 2011 at 3:07 am said:

> The example of the Netherlands medical school entry which has run for decades without detriment to the effectiveness of the professions involved shows that it works.

Conall, I don’t doubt that the standard elitist admittance policy is not only not optimal, but is actually quite harmful to society. My critique is of your arguments rather than of your conclusions. If anything, I think you are not going far enough in your proposals. I will pursue these ideas in the to-be-completed parts 2 and 3 of this article.

LikeLike
Yoram Gat, on March 30, 2011 at 3:10 am said:

> a lot hinges on whether scores are being used (if I may borrow terminology from Willem Hofstee) for purposes of selection (i.e., admit the n best students) or admission (i.e., admit every student with a score over x).

Conall argues against the first method and in favor of something along the lines of the second method.

LikeLike
Conall Boyle on university admittance: (2) what merit? « Equality by lot, on April 7, 2011 at 5:33 am said:

[…] Conall Boyle sees admitting students to universities based on standardized test scores as being a meritocratic policy. This is so, according to Boyle, since standardized test scores are a good (indeed, the only) predictor of probability of graduation. There seems to be an obvious gap here: it is far from clear why high probability of graduation can be considered “merit”. Boyle rejects “good works” (such as doing volunteer work for good causes), for example, as being “false merit”, because it is not a predictor of probability of graduation. This seems like an unusual use of the term “merit” – a more suitable term perhaps is “potential” or “promise”. […]

LikeLike