Criteria for the acceptability of an allotment procedure

With the increasing frequency of the application of sortition in society, it has been rightly pointed out that a lack of strict criteria for the validity of the constitutive procedure of the body would allow sortition to be misapplied and manipulated so that its democratic value is annulled. Of course, there are many aspects to constituting political bodies and they all need careful consideration and standardization if we wish to achieve well-functioning democratic decision making. Here I want to advance criteria for standardization of just one aspect – one that is unique to allotted bodies – the allotment procedure.

For an allotment procedure to be considered well designed it should have the following characteristics:

  1. Public statement of n, the number of people who are going to be allotted.
  2. Public statement of the allotment pool – the set of people from whom the allotted will be selected. This statement should make it easy for every person in society to know whether any other person is included in the pool or not.
  3. Public definition of an equal-probability allotment mapping. An allotment mapping is a mapping of each sequence of digits of length L, for some fixed number L, to a list of n people. The mapping is publicly well-defined if it is possible for anybody putting in reasonable effort to determine with reasonable accuracy, given a sequence of numbers, the people who correspond to that sequence. The mapping is equal-probability if the number of sequences which result in a list containing each person in the pool is between N0 and N1 for some fixed numbers N0 and N1, where N1 / N0 < 1.01.
  4. A public definition of a procedure to generate a practically-random equal-probability sequence of L digits. A procedure for generating sequences of digits is practically-random equal-probability if there is a general acceptance that as the procedure is launched as far as anyone can determine every sequence among the 10L possible sequences is equally likely to be the outcome of the procedure.
  5. The random sequence procedure should be launched after the four public statements above were made. Its launch and applications must be public so that it is applied exactly once and once it is applied, its outcome (i.e., the sequence of digits it generated) is immediately public.

Other than the first criterion, all of these criteria are non-trivial to implement. There is, of course, some similarity to a prize lottery procedure, but there are some complications associated with the fact that each person must have exactly one “winning ticket” and of course with the fact that the prize – political power – could be of much greater value than any other lottery prize ever.

In particular, making sure that the digit sequence selection procedure is practically-random equal-probability is fraught with difficulties since any physical randomization device may potentially be rigged by powerful attackers. If you have ideas for rigging-resistant randomization procedures please post them in the comments.

34 Responses

  1. Yoram,

    You are right to distinguish between the first criterion and the others. If descriptive representation is the criterion for a “well-functioning democratic decision making” body then the optimum number of participants for a pre-specified decision threshold should be specified (or at least audited) by independent statisticians (rather than the commissioning body). There is also a need to specify exactly how it will be ensured that as many as possible of those selected will actually serve and how ongoing descriptive representativity will be maintained. If these criteria are not established at this stage then your other four criteria will be rendered trivial — I don’t recall any instance of the outcome of the National Lottery being disputed on the grounds of a flawed allotment procedure.


  2. Yoram,
    A key is that in addition to actually being a fair draw, that it be readily perceived as impossible to manipulate. The public statement in advance is crucial, but perhaps not enough to also create the perception of fairness. Ideally we would develop some ritualized procedure that even had an element of showmanship. Also, on point 4 … might it be acceptable to have a procedure which did not literally provide an equal chance to each member of the pool independent of who else was selected, so long as nobody could know who had a lesser of greater chance, or predict or manipulate the selection process? I also like the idea of intersecting procedures that draw on either natural phenomenon or diffuse input that assure nobody can steer the draw. (For example, selecting a seed number by multiplying the third decimal point digit of a public digital barometer at exactly noon by the second and fourth digits of some stock market index also at noon… except that lacks any showmanship.)


  3. I’m puzzled by these concerns about the technicalities of the draw procedure, given that this doesn’t seem to be a point raised by the millions of people who throw away their own good money buying lottery tickets every week. But I do agree with Terry that there is some value in showmanship — the UK National Lottery actually shows the ball machine in operation on prime time TV.


  4. Hi Terry,

    > A key is that in addition to actually being a fair draw, that it be readily perceived as impossible to manipulate

    Yes – as I defined the term above, a procedure is practically-random only if people accept it as random.

    > might it be acceptable to have a procedure which did not literally provide an equal chance to each member of the pool independent of who else was selected, so long as nobody could know who had a lesser of greater chance, or predict or manipulate the selection process?

    Yes – as long as people perceive all outcomes to be equally likely before the procedure starts, then the procedure is acceptable.

    > selecting a seed number by multiplying the third decimal point digit of a public digital barometer at exactly noon by the second and fourth digits of some stock market index also at noon

    Both of those procedures seem to me prone to manipulation. A digital barometer will involve some sophisticated hardware that would make it quite difficult to verify its output is unmanipulated and unpredictable. A stock market index is the outcome of a complicated calculation involving a large number of inputs. This would be even more difficult to verify that the output is unmanipulated and unpredictable.

    BTW, depending on how these are combined, it may be enough to control one of them in order to control the output.

    I think we need something better.


  5. Hi everyone, this is a very important discussion topic that I don’t think receives sufficient attention. I don’t have the book with me now, but I recall that one of the final chapters of Conall Boyle’s book, Lotteries for Education, had some very useful practical tips for how to run lotteries effectively.


  6. Delighted you brought up this topic. I have been a ‘winner’ in two lotteries — one for tickets to watch tennis at Wimbledon, the other to attend Buckingham Palace for the Queen’s Jubilee Concert. In neither case was any effort made to assure applicants of the validity of the draw. Trivial entertainment, so did it matter?

    Another was when I was called for Jury Service in the Law Courts. Only a statement of ‘drawn by lot’ appeared on my summons. I asked around when I got to the Court and was told that a clerk in an office has a copy of the electoral register, and picks names “at random” from random pages of the register (this was 1994!). Subsequently I learned that the process was computerised.

    Maybe no evil intent was involved, but not explaining the procedure, nor carrying out the draw in public, or allowing the numbers to be checked for true randomness leaves the process wide open to manipulation.


  7. Hi Simon and Conall,

    Thank you very much for your comments. I completely forgot about Conall’s treatment of this subject (although I have read the book and even reviewed a part of it for EbL 1, 2, 3).

    Yes – chapter 12, “Lottery practicalities: how it should be done”, makes very useful observations on the difficulties associated with randomization and exhibits a healthy dose of suspicion about the possibilities for manipulating the process (like Conall’s comments above). I think it is fair to say, however, that it does not give a standard for what can be considered acceptable, so this important work is still to be done.


  8. Terry’s original comment on the corruptibility of policy-making juries that inspired this thread has little in common with Conall Boyle’s interest in lotteries, so I think we need to make the distinction explicit:

    1. Conall is the founding father of the Blind Break lottery school, which is concerned with the fair (equal chance) distribution of scarce goods (such as university places and Buckingham Palace garden party tickets) and the impartial allocation of the “short straw” (military call up, trial jury service etc) along with the choice of who gets to live or die (decimation, lifeboat allocation etc). Needless to say concrete individuals are very concerned about the impartiality of the lottery algorithm as their personal fate depends on it.

    2. Terry’s original concern was the manipulation of descriptively-representative bodies charged with making political decisions on behalf of the population that they represent. Whilst it’s important that the impartiality of the selection algorithm be trusted, the principal reason that such bodies are unrepresentative is if participation is not quasi-mandatory, length of service is long and speech acts are included as part of the mandate (for reasons that we have discussed ad nauseam).

    Although these two uses of random selection are entirely distinct, overlap occurs when, for example, trial jury service is not mandatory (thereby over-representing retired and unemployed citizens) or when it is possible to opt out or defer military service (thereby over-representing citizens who do not go to college or have friends in high places). But we do need to keep these two ideal types separate and recognise that for type 2 lotteries, an impartial selection algorithm will do little to ensure the impartiality of the resulting decision-making body. As such Conall’s excellent book (and, as his publisher, I’m shooting myself in the foot here) is of little relevance.


  9. Not only the drawing itself is of importance but also the system of sortition.
    Very disturbing : Quote “A contractor is being sought to randomly select 120 members of the public to serve on the Assembly. The individuals will be broadly representative of Scotland’s adult population in terms of age, gender, socio-economic class, ethnic group, geography and political attitudes”. Unquote
    This means probably that selection will be done by an uncontrollable “scientific methode” (stratification of some kind with a final manual selection). For a Simple Random Sampling from a register of residents (or the electoral list) I don’t think that a ‘contractor’ is needed and 120 people is not enough to be ‘representative’ in some kind.
    What can we expect form 120 people in regard of stratified ‘representativeness’?
    For example:
    How many age groups ? take 3 groups (young, middle age and old)
    Gender : let’s be conservative and take 2
    Socio economic class : lets take 3 (poor, middle class, rich)
    Ethnic groups : 3 (English, Gaelic, others)
    Geography : 2 (rural and towns)
    Political attitudes : 2 (left and right)
    This means 3 x 2 x 3 x 3 x 2 x 2 = 216 we have to cut down somewhere I presume.
    And that is called ‘representative’?
    Can somebody explain where I am wrong?
    The principal reasons for using stratified random sampling rather than simple random sampling include:
    1. Stratification may produce a smaller error of estimation than would be produced by a simple random sample of the same size. This result is particularly true if measurements within strata are very homogeneous.
    2. The cost per observation in the survey may be reduced by stratification of the population elements into convenient groupings.
    3. Estimates of population parameters may be desired for subgroups of the population. These subgroups should then be identified.


  10. Paul,

    Agree with all of the above. I’m particularly concerned by the assumption that creating a small stratified sample using six crude population parameters will produce an accurate portrait in miniature of the target population and that voluntarism will not significantly bias the sample. This is what will give sortition a bad name, and concerns regarding the nature of the randomisation algorithm are trivial by comparison.


  11. Hello Keith, I presume that it is logical to accept that not all applications with sortition need the same sortition system and representativeness. That is why, at this moment we only talk about sortition criteria at the legislative level. Even then we can discuss if the Oregon CIR is at the legislative level or not. For me it is. It intervenes in decision making at legislative level (referendum in this case). That is why we propose to use the Arnstein ladder (or the cube from Archon Fung) to first determine what the level of ‘participation’ is and what criteria, and with what level of concessions, can be accepted. Not an easy task.

    What we see now is that commercial companies are entering in the ‘democratic’ field for organising the events. I don’t say that they are incompetent, on the contrary, and that is even more worrying. As an example, in Belgium this is


  12. Paul:> I presume that it is logical to accept that not all applications with sortition need the same sortition system and representativeness.

    Absolutely, Blind Break theorists like Conall Boyle and Oliver Dowlen have no interest in sortition as a system of representation. As far as they are concerned lotteries are an impartial way to select individuals, so they naturally focus on the algorithm used and the need to ensure the lottery is a public, or even ceremonial, event. The principal focus of this blog, however, is representative decision-making groups and I’m concerned that the focus on Blind Break criteria might be seen to be sufficient, whereas it’s really just the (trivial) starting point.


  13. This is my question:

    What we are looking for at the moment is an evaluation methode of the selection proces for panels like the Oregon Citizens Initiative Review.
    Those panels don’t answer to any of the basic ‘democratic principles’ mentioned in (Dimitri Courant) except maybe ‘equality’ depending on the sortiton methode used.

    III- Democratic principles
    I will distinguish four democratic principles, or values, of sortition: equality, impartiality, representativeness and legitimacy, each being subdivided in three elements.

    For example: The 20 Review panelists were randomly selected from registered voters in Massachusetts using a scientific method to ensure it is representative of the overall electorate based on place of residence, party affiliation, age, gender, educational attainment, and race/ethnicity.

    The question is: in how far can we put ‘democracy’ in the hands of specialists who’s work we, as citizens, can’t evaluate and by consequence neither can we evaluate the results. Is a (scientifically) ‘manipulated system’ that aims by the means of stratification and correction by specialists for a better ‘diversity’ acceptable in a democratic proces that needs to be also impartial, representative and legitimate. Is it possible to develop a selection system that is in compliance of the criteria (Dimitri Courant for example) for the use of sortition in the ‘democracy’ (legislative) domain.


    At this moment I even think that a ‘descriptive representation’ that is acceptable in terms of margin of error and confidence level that starts with about 500 people is sufficient to be ‘acceptable’ for a democratic use (decision making at legislative level). Only when in combination with the other criteria (high rotation, short mandate, different Jury’s,.. ) the system can become and stay acceptable. In my view all political systems (even theocracy, monarchy, electoral aristocracy, etc..) has to be ‘acceptable’ for most of the citizens. Legislation is by his nature enforceable, also on those who disagree. Therefore we have to be very critical in accepting ways of developing legislation. Especially when we call it a ‘democratic instrument’.


  14. Paul:> Is a (scientifically) ‘manipulated system’ that aims by the means of stratification and correction by specialists for a better ‘diversity’ acceptable in a democratic process?

    No it isn’t (especially when only 20 citizens are involved). I agree with Dimitri’s criteria and everything you say in the last paragraph, especially “Legislation is by his nature enforceable, also on those who disagree”. That’s why it’s so important to demonstrate that the decision outcome is invariant between different samples of the same population. In an electoral system everyone gets the chance to vote and most people accept the outcome of the election, even if they are in the losing camp. The same criterion needs to be applied to decision making by allotted sample — claiming that the general will emerge by magic from the black box of a deliberative exchange between a tiny group of consensus-seeking volunteers is not good enough. Invariance of outcome (to an agreed confidence level) is the minimum standard for decision-making by allotted body, and I’m puzzled why this claim leads to such indifference (or hostility) on this forum. I imagine Paul will be the only one who will respond (not that there’s any need, as we agree with each other).


  15. I do understand the enthusiasm of sortinistas when sortition is used anywhere and in any form. And I do understand that anything that starts up has to take some risks otherwise there would be no start at all. I can’t imagine the Oregon CIR starting with Simple Random Sampling with 500 citizens and different Jury’s to compose the panel of experts and manage the organisation. But that sayd, we have to correct when possible before known and previous unknown shortcomings become dangerous. Until now we don’t experienced heavy attack from people who know what they are dealing with and how to attack a threat to their interests.
    I presume that the attack will not be about details only known to specialists, error rate, confidence level, stratification etc.. but at the core of the sortiton system as a political and democratic instrument. Maybe we can see the best attack strategie emerge in the Scottish example: Scotland’s opposition parties attacking the government’s citizen assembly proposal as untrustworthy
    And what is our defence?

    Liked by 1 person

  16. Maybe we have to develop something comparable with the ‘code of good practice for referenda’, with motivation:


    (for those interested it exists also in French : COMMISSION EUROPEENNE POUR LA DEMOCRATIE PAR LE DROIT

    Code de bonne conduite en matière référendaire – Lignes directrices sur la tenue des référendums et projet de note explicative ) )

    They have also commented in one occasion (as far as I know) on a proposal with sortition: : Prytanées (articles 9-13)

    The ‘Code of good practice for the use of sortition in politics’. Maybe we can start discussing the title ;-)


  17. Paul,

    My principal concern is the recent focus of this blog on the selection algorithm as opposed to the myriad other factors that preserve ongoing representativity. It’s possible that both commissioning bodies and the commercial contractors have been influenced by the naive assumptions of sortition advocates on this forum and elsewhere.

    >I presume that the attack will not be about details only known to specialists, error rate, confidence level, stratification etc. but at the core of the sortiton system as a political and democratic instrument.

    The two things are deeply interconnected as problems with the former (e.g. a stratified sample of 20 volunteers) could well lead to the discrediting of the latter. That’s why we need a different name for this sort of project — the use of sortition in the selection process is of no more relevance than whether they serve tea or coffee in the refreshment breaks.

    Liked by 1 person

  18. anyway, I started with the ‘evaluation grid for the use of sortition at legislative level’ and I hope that with constructive criticism it can be ameliorated.


  19. Paul,

    Had a quick look — guess it needs a lot of fleshing out. Not sure that this thread is the appropriate place, but agree that the multiple criteria you raise are far more relevant than agonising over the selection algorithm. And your criteria would be of little interest to Blind Break theorists like Boyle and Dowlen.


  20. 1. Whether sortition is used for a small body or large one, it is still sortition. Large bodies exhibit representativeness as well as the (blind break) anti-corruption and anti-manipulation sanitizing effect. Small bodies can only reliably exhibit this latter effect… but sometimes that is enough for particular purposes. In the Italian republics of the Renaissance they used sortition while completely unconcerned with the accuracy of the representativeness. The oregon Citizens Review is also primarily concerned with the impartiality of the panel, rather than its representativeness.

    2. Random selection can be combined with stratification in a legitimate way to improve representativeness when mandatory service is not an option (at least if the number of demographic traits is not unmanageable). It does nop harm to keep randomly drawing until an equal number of each agree to serve. Voluntary service is always less ideal, but until laws are passed making service mandatory we will need to rely ona) removing as many obstacles to service as possible, b) providing substantial inducements, and c) some stratification.

    3. Keith’s standar of identical outcomes for duplicate bodies is unreasonable. Two duplicate elected legislatures would rarely act identically. Even the exact SAME body of representatives could often make different decisions on different days. And… a referendum with mandatory voting could deliver different outcomes from the same voters on different days.


  21. Sorry for several typos above…
    Most can be figured out from context… but the sentence in the middle of point 2 should read:
    “It does no harm to keep randomly drawing until an equal number of men and women agree to serve.”


  22. From the site
    How are the citizen panelists selected?
    A random sample of registered voters (typically 10,000) receives a letter in the mail explaining the Citizens Initiative Review and inviting them to apply to participate and verify that they can review the selected measure objectively. Of the applications returned, project organizers conduct an open and transparent process to select a panel that reflects voter demographics according to official census and elections data. The resulting panel reflects the population of the state, county, or city in terms of age, geography, political perspective, gender, race, ethnicity, and educational attainment.
    This means that only the initial selection is done by sortition as far as I understand. Then we have an ‘open and transparent process to select a panel’ that reflects the population in 7 sub categories … and that with 24 citizens? And we don’t talk about any proportionality of those categories of course because it hasn’t to be ‘representative’, only impartial, according to Terry.

    This is not what they are saying:

    Citizens Juries convene panels of everyday people, who are randomly selected and representative, to help elected officials tackle tough policy questions.

    But even then..

    ‘Information you can trust’ is their main claim. Well, for me they have to do better.


  23. I do remember something about an experiment regarding the ‘identical outcome’ done by James Fishkin but I can’t find it for the moment. Somebody?


  24. some reading at Participedia is enlightening but not very reassuring Citizens’ Jury


  25. Terry,

    Good to see that Paul and I are not just talking to each other. Regarding your points:

    >The Oregon Citizens Review is also primarily concerned with the impartiality of the panel.

    Using an impartial selection mechanism does nothing to ensure the ongoing impartiality of the persons selected, even though they “verify that they review the selected measure objectively” (the modern equivalent of the Heliastic oath). The fact that there are only 24 of them and that they have to apply to participate guarantees that they will not be representative in any respect other than the 7 sub categories. So, as Paul points out, the role of sortition in the process is secondary.

    >when mandatory service is not an option

    As you know we both advocate quasi-mandatory participation (via incentives, rewards, removing obstacles and civic republican propaganda).

    >Keith’s standard of identical outcomes for duplicate bodies is unreasonable.

    My standard is for invariance within a pre-specified confidence interval. As for elections, the Federalist (trustee) perspective was adopted, whereas the binding mandate perspective is more suited for a descriptively representative body (as advocated by the Anti-Federalists). If that is the case then the issue arises as to which sample of the same population should prevail. The fact that successive referenda may deliver different results is of no relevance, as we are interested in the considered preferences of the demos at one particular point in time.

    Paul:> I do remember something about an experiment regarding the ‘identical outcome’ done by James Fishkin

    Yes and the experiment failed (as documented by Bob Goodin). I put it to Jim that the likely cause was the variation generated by the small group deliberation, but he insisted (in deference to Habermasian norms) that this was an essential element of the DP. However if the DP procedure can’t deliver consistent outcomes then it’s of little or no use.


  26. Here’s the full Goodin reference:

    Among deliberative democrats, the main evidence offered on the question of whether different deliberative groups come to similar conclusions derives from a series of Deliberative Polls conducted in three different locations in Texas on issues of energy policy. Fishkin proudly reports that in all three cases opinion shifted in the same direction, however:

    After deliberating on the matter, fully half the respondents at one site thought that ‘investing in conservation’ was the ‘option to pursue first to provide additional electrical power to the service territory’; elsewhere under a third thought so.

    Alter deliberating on the matter, over a third of respondents at one site thought that ‘renewable energy’ should be the top option; elsewhere only a sixth thought so.

    Clearly this degree of variation rules out political decision making by DP-style assemblies as there is no way of knowing which of the three samples represents the considered preferences of the people. Assuming identical information/advocacy between the three groups (and the same random selection/sample size methodology), I put it to Jim that the small-group deliberations might well be the culprit, but he would not countenance dropping this part of the DP methodology (for Habermasian reasons). However if we want government by allotted sample then we need to privilege ongoing representativity over deliberative norms.

    Goodin, R., Innovating Democracy: Democratic Theory and Practice After the Deliberative Turn, (OUP: 2008), pp. 115-116


  27. >Keith: Had a quick look — guess it needs a lot of fleshing out. Not sure that this thread is the appropriate place, but agree that the multiple criteria you raise are far more relevant than agonising over the selection algorithm<

    The evaluation grid has the advantage that it exists and can be used to discuss about any implementation of sortition in the political environment. Furthermore we prove that it is possible to write proposals in compliance with those criteria.
    Nevertheless I am sure that it can be improved. The publication is dated so there can be updates. My intention is to relate the criteria, with motivation, with the Arnstein ladder (in 8 steps) or the democracy cube from Archon Fung (who needs coordinates in this case).


  28. Paul, I think you would need to make that a new thread, as it’s somewhat removed from Yoram’s original post.


  29. Yoram, I think that Dimitri Courant answered your question Du klérotèrion à la cryptologie : l’acte de tirage au sort au xxie siècle, pratiques et instruments (en Français )


  30. Publication in English is expected around end of this year.


  31. I’ve only skimmed the paper, but it seems to me it does not address the difficulties associated with generating publicly verifiable random numbers. Using digits taken from a stock index, for example, seems to me to be highly susceptible to manipulation.

    Other difficulties associated with trusting the allotment authority also seem underappreciated.


  32. […] allotment procedure The criteria for an impartial random-selection algorithm have been covered by Yoram Gat’s recent post. Terry Bouricius has also argued that the need to be seen to be fair might require some sort of […]


  33. […] was some discussion on Equality-by-Lot recently about developing allotment standards for sortition-based decision […]


  34. After a lot of sleepless nights ;-) with the question: what system would I personally accept and consider possible to defend to other people I came to the conclusion, a bit in line with Terry (Ideally we would develop some ritualized procedure that even had an element of showmanship), that the mechanical lottery drum is the best option. In Belgium with 8.000.000 people eligible for voting it would only need 7 drums (and a spare one) with each ten numbered balls (from 0 to 9) to produce a number corresponding with a citizen on the electoral list (we all have an identity card number of 11 digits that can be connected to a number on the electoral list) , let’s say every 10 seconds. It then takes one hour to produce 360 numbers, every two hours we have a descriptive representative Jury if everyone participate (legislative decisive Jury). If it isn’t fast enough we can place more lines of drums. The procedure can be on TV and monitored by a (not representative) Jury of 6 members appointed bij Simple Random Sampling from the electoral list with the same system and renewed every 4 hours with a day paid (senators wage) and expenses (not obligatory).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: