There was some discussion on Equality-by-Lot recently about developing allotment standards for sortition-based decision making bodies. Among other issues, the question of stratification got some attention. It turns out that Sortition Foundation, which is engaged in such activities, has a document (PDF) describing their procedure. It could be interesting and useful to compare the procedure laid out in the document with proposals for standards which were discussed. I invite readers to do so in the comments below or by contributing a post.
Filed under: Applications, Sortition |
Adam Cronkright
Ali
Ahmed R. Teleb
avanderven
avergne
Alex Kovner
azakaras
Brett Hennig
campbellwallace
Simon Threlkeld
Nick Coccoma
codyhipskind
Common Lot Sortitionist
Conall Boyle
constitutionalism
davidschecter
graytmatter
gtridimas
H.J. Hofkirchner
John Burnheim
jgarry20151
johnedwardtaylor
johnhuang865
Jonathan Crock
keithsutherland
Jacob Richter
lukewhito
SimonPek
Nathan El
Nicholas Gruen
Owen Shaffer
Patrick Chalmers
Paul Gölz
paulnollen
peterstone
Pierre-Etienne Vandamme
pronfeld
Emil
rcaze
richwbrown
Roger Knights
roslynfuller
Pierre-Etienne Vandamme
shaharavin
tbouricius
Ted Wachtel
Joshua Laferriere
theodorestathis
Wayne Liebman
Yoram Gat
Oliver Milne
Equality by lot 
This might be acceptable for a 3 or 4 level on the Arnstein ladder. At least in this article (pdf) there is no information about the stratification and the resulting error and confidence level. Therefore I can not even defend this procedure. It is also household based (adres) but I don’t know what might be better if the official electoral or residence list (in this case) is not accessible.
LikeLike
> there is no information about the stratification and the resulting error and confidence level
I am not sure what that means. Even with a perfect procedure, what would this mean?
> household based
Good point. There would be at least two issues associated with that. First, if all households are equally probable to be selected, the selection would under-represent people who live in large households. Second, how are people selected within the household?
(However, I think that there are much more serious problems associated with this procedure.)
LikeLike
Given the very low response rates, the limited stratification criteria and the tiny size of the final assembly, it’s hard to know what is meant by their clam that the sample is ‘representative’. This kind of procedure can only give sortition a bad name.
LikeLike
The flaws in this document indicate the urgency of agreeing on the criteria for a representative citizens’ assemby: https://equalitybylot.com/2019/07/25/criteria-for-a-representative-citizens-assembly/. When someone as knowledgeable on sortition as Brett Hennig comes up with this sort of nonsense we really have a problem.
LikeLike
yoram > there is no information about the stratification and the resulting error and confidence level<
I am not sure what that means. Even with a perfect procedure, what would this mean?
As far as I know (my question about geographic stratification is still not answered or I don't understand the answer which is more likely) there is no objective evaluation possible of a stratified sample. The only thing we can say is that error rate and confidence level MIGHT be better than a Simpel Random Sampling OF THE SAME SIZE . Only the proposed size of the panel is known in the proposal of the sortition foundation. As long as there is no better explanation the error rate for 50 selected citizens may be better than 14% with a confidence level of 95% for a population of 50.000 people. I don't know the influence of a random preselection of 10% (5.000) on the calculations or the influence of the response rate. But I presume that if I have to defend this proposal I need all those details (at least).
LikeLike
Hi Paul,
Yes – you are correct. The effect of stratification in terms of reducing the sampling variance would depend on the inter-strata variance for the question being considered.
The random preselection itself does not matter one way or another. The issue is that it is used as a device (a very problematic one) for compensating for different response rates in the population.
LikeLike
Comparing SF’s procedure to the criteria I proposed shows that there are several points where improvement can be expected.
Public statement of the allotment pool: the determination of which households are in the pool and which are not should be made public. This should include the number of households and an easy way to for the public to determine whether a specific household is in the pool.
Public definition of an equal-probability allotment mapping: how are random numbers mapped to household selection? This should allow the public to reproduce, given the output of the randomization device, the selection of the households in the pool.
The randomization procedure: how are the random numbers being generated? Which random numbers did in fact turn up? Which households did they map to?
Furthermore, since the procedure contains post-selection steps, there needs to be full transparency about those steps: which households responded? If not all of those become part of the allotted body, how was the selection done? If the selection was based on some characteristics of the households, what were those characteristics? The per-household data that was used for this selection should be made public. If further randomization was employed, this procedure needs to meet the same criteria specified above.
In general, the guiding principle should be that the entire procedure should be pre-defined and reproducible by any interested member of the public, so that it is clear that no room for manipulation (“judgement calls”) is left. At that point it would be possible to have a substantive discussion whether the procedure itself is sound and lives up to the ideal of political equality that it supposedly embodies.
LikeLike
Yoram:> If the selection was based on some characteristics of the households, what were those characteristics?
I believe the initial invitation is to households (presumably to reduce postage costs), but the post-selection, based on the 4-7% of the individuals in the initial sample that accepted the invitation, is stratified by four crude population parameters derived from the last census. Given that the final sample is far too small to ‘describe’ the target population in any meaningful sense I think fixing this problem would be little more than polishing a turd.
LikeLike
Can I draw your attention to two books from Keith’s publication firm Imprint Academic which address the stratification issue:
‘The Athenian Option’ by Anthony Barnett, and
‘Lotteries for Education’ by yours truly.
On p128 I look at Huddersfield University which admits students by lottery, but with quotas (stratified) for Male, Female and Mature. I discuss Barnett’s proposal that his reformed House of Lords reserves 50% of seats for either sex.
LikeLike
Hi everyone, I invite anyone on this forum to contact the Sortition Foundation through our website if they want to know more about our processes. We will happily discuss them with you.
We are currently finishing our tenth such selection process in the UK. The process is modelled very closely on that of the newDemocracy Foundation (Australia) and MASS LBP (Canada) processes that have successfully involved randomly selected individuals in citizens’ assemblies and the like on well over 50 occasions.
Regards,
Brett
LikeLike
Sorry that Brett is not prepared to discuss his proposals on EbL — the principal international forum for sortition debate. My criticisms of the Sortition Foundation methodology would certainly extend to newDemocracy — it’s interesting to compare and contrast Lyn Carson’s (newDemocracy) paper in the Legislature by Lot volume to those of Fishkin and Owen/Smith — a real case of chalk and cheese.
LikeLike
It’s worth noting that Brett writes polemical books calling for the overthrow of the electoralist establishment, but then founds an organisation that provides contractual services to organisations seeking a faux-democratic patina for their activities. Leaving aside the Sortition Foundation’s need to generate an income, I’m sure this is entirely well-intentioned but I think the tiny number of people involved in their allotted committees (ditto with newDemocracy) indicates a fundamental divide between deliberative democracy and sortition theory. The former focuses on the procedural factors necessary to enable the free exchange of reasons between equal participants, so the “democracy” is entirely endogenous. The assumption of newDemocracy and Sortition Foundation appears to be that if a small group of volunteers (4-7% acceptance rate from the initial invitation) is re-selected by stratification on the basis of four crude census criteria then the deliberation of the committee will in some sense represent the views of the whole community.
This is clearly false, for all the reasons provided in https://equalitybylot.com/2019/07/25/criteria-for-a-representative-citizens-assembly/ and I’m saddened that Brett is not prepared to debate this on EbL. My main concern is that this will be seen by the world as a good example of sortition in practice.
LikeLike
Not to mention that the whole event is prone to manipulation from the beginning to the end. When I worked as a union representative I met professional negotiators who could single handle a whole union delegation. A always like to see a good craftsman at work but in this case we have to consider that, in a worthwhile case, no money or effort whatsoever will be to much to get the “right” outcome, as we can see in some referenda. At least, the G1000 in Belgium was smart enough to mention that they don’t were looking for “representativeness” but for “maximal diversity”. This avoided all criticism about “representativeness” and permitted them to “scientifically” select who ever they wanted. It is impossible to evaluate such a system. This people at least recognised the weak point in their event and avoided it.
LikeLike
Yes indeed, it would be good if someone from Sortition Foundation or newDemocracy would defend what is, prima facie, a blatantly undemocratic practice. What bugs me is the presumption that an impartial and un-manipulatable selection process will generate a body that shares these virtues. Assuming Brett is unwilling, do we have anyone prepared to play the role of Advocatus Diaboli?
LikeLike
And then you are already generous by assuming that the stratified ‘scientific’ selection process, after a questionable selection by sortition that violates equality at least, is un-manipulatable ;-).
LikeLike
All their initiatives are “a success”. Isn’t that a strong argument?
LikeLike
I’m not often accused of generosity, but there’s a first time for everything!
LikeLike
Paul:> All their initiatives are “a success”. Isn’t that a strong argument?
Yes, I was struck by that — presumably success in the eyes of the corporate body that commissioned the Sortition Foundation to come up with the stratified sample (and paid the bill for professional services rendered).
LikeLike