Equality by lot

Stratification for newbies

8 Responses

1. 

   Anonymous, on October 29, 2023 at 4:54 am said:

   I’d also like to know. I feel fairly confident that someone would have studied it, somewhere. In addition to the size of a randomly-selected group (and being statistically significant, representing the whole), I’d also like to know the statistical theory behind sub-groups. ie. stratification.

   LikeLike
2. 

   paulnollen, on October 29, 2023 at 6:05 am said:

   To Anonymous and Owen,
   this are very important questions and in the past few years I could not find satisfactory answers. I can only give what I think to know so far in order to start the discussion. For our specific use, the citizens’ assembly, we need to stratify. In all cases we allow only a sub group to participate. Be it above a certain age, be it citizens eligible to vote etc. This needs stratification. But nearly all stratification is about inclusion and exclusion. So you have to evaluate that for every stratification. A stratification that has no exclusion is stratification in age groups. But in most cases something else is missing, proportionality. If you decide to work with say 4 age groups, in order to be representative for those groups they have to be proportional. Notice also that “representative” has to be specified. Representative on his own has no meaning. You can also stratify with only objective characteristics, for example age, place of official residence, eligible to vote (or not), gender (official), or you can use other characteristics obtained by questionnaires. Those questionnaires are a profession on his own. People working with this tool know that for example sensible questions (religion, skin colour, etc,) give no reliable answers and that has to be taken into account in the conclusions. In many cases stratification is used to compensate for errors. The volunteer bias can be minimised by using bootstrapping technics, by performing the same selection by sortition twice. You first select a big number of citizens by lot and from that result you select the small number you want to work with. Also another bias can be compensated this way, the “small sample error”. This “small sample error” is explained in other scientific applications, medicine, archaeology, palaeontology etc. In the small sample error it is easy to imagine that in a small sample you can have a very exceptional person selected or it is possible that you missed a big group. In order not to mis a big group you can decide, as an organiser, which groups are important for you or which groups you think are representing the population at hand. Now it is maybe interesting to change from “representative” to “divers”. That leaves for you more flexibility. The group has a maximum diversity. Another problem with stratification is that you want every characteristic in every sub group you choose. For example you have 2 gender, 4 age groups (forget for a moment proportionality), 10 geographic zones, this makes already a group of 2 x 4 x 10 = 80 people. This number can be reduced by ” probability selection” used in algorithm selection (I have no idea how it works). Anyway, the more specialist and manipulations you need to perform the selection (and this is only the selection stage) the further you are away from the “honesty and acceptability (legitimacy)” of selection with sortition. Interesting data can be obtained in the bookkeeping of the organisers, if they publish it. How much is paid to whom to do exactly what. In most cases financial transparency is very low. You can always compare with the “golden standard” from James Fishkins’ Deliberative poll system in order to evaluate what is going on. I don’t want to say that small groups can’t be used for a specific task but for the use in legislation there is far to much manipulation of all kind. Maybe I forgot some items but they will come up in the following reply’s’ I think.

   LikeLike
3. 

   paulnollen, on October 29, 2023 at 12:49 pm said:

   I also want to add that even in the stratification for gender the proportionality in most cases is not respected. In Belgium the correct proportionality is 104 woman for 100 men. In a sample of 100 citizens this is not a priori neglectable I think. There is a lot of explaining to about the choices made but not many are explained or defended. The next chapter is about “statistical representativeness”.

   LikeLike
4. 

   paulnollen, on October 29, 2023 at 12:51 pm said:

   and divers must be diverse from diversity ;-)

   LikeLike
5. 

   paulnollen, on October 30, 2023 at 7:44 am said:

   Part 1
   Understanding and working with statistics is a science on his own. What is lacking for us, people interested in sortition, is a readable explanation for non scientist. The best we can do is read other scientists who are evaluating the work of other scientists in a readable language. In this case the work of Jane Mansbridge “Deliberative Polling as the Gold Standard” was very interesting. Of course, also DP is evolving and tested for the first time “AI” (Artificial inteligence) in his concept (2019). Fishkin claims “statisistical representativeness” and provides the material to evaluate this claim.

   https://deliberation.stanford.edu/what-deliberative-pollingr
   ”The process has the statistical representativeness of a scientific sample but it also has the concreteness and immediacy of a focus group or a discussion group. ”

   It is up to the organisers to provide adequate information in order to evaluate their claims https://www.researchgate.net/publication/239820911_A_Methodology_for_Assessing_Sample_Representativeness
   The term “representative” has many definitions. A variety of definitions can be found elsewhere (Warren, 2004). However, a wide variety of definitions is not helpful since representative is an operational definition. If a definition cannot be agreed on, it is impossible to evaluate representativeness in a meaningful way. If evidence for representativeness is not presented, the data cannot be characterized as effective for project decision-making (Crumbling, 2001)

   LikeLike
6. 

   paulnollen, on October 30, 2023 at 2:16 pm said:

   Part 2

   An interesting paper is also https://boris.unibe.ch/47331/1/13gerber_m-1.pdf

   DP possesses many of the favorable features for successful deliberation (e.g., Fishkin and Luskin, 2005;
   Mansbridge, 2010). Most importantly, these include the gathering of a statistically representative sample, the provision of balanced information material, the possibility to
   question experts and politicians on various policy alternatives, and the presence of trained facilitators who should ensure that group discussions are balanced and civil. Deliberative polls are also meant to “insulate people from social pressure” by depriving participants from reaching a decision, a common statement, or any other kind of mutual agreement that could distort the process of uncoerced argumentation
   ….
   And how can we be sure that post-deliberative opinions are the result of a careful reasoning process instead of the mere provision of additional information or, what would be worse, the consequence of manipulation?

   The paper also compare several citizens assemblies : for an empirical example, the thesis examines a subsample of small group discussions in Europolis, a pan-European deliberative poll on third country migration and climate change that was held in May 2009 in Brussels (Fishkin, 2009). 4 By including
   participants from all 27 EU-member states, Europolis provided an extensive attempt for cross- cultural deliberation in a multilingual setting.

   This event had proportional representation. Not in full but a calculated proportionality is acceptable in most cases (see for example the Penrose system)

   LikeLike
7. 

   paulnollen, on October 30, 2023 at 2:48 pm said:

   Part 3

   We can start now with some basic statistical calculations and notions.
   I propose the Raosoft calculator. http://www.raosoft.com/samplesize.html
   – The margin of error
   – Confidence level
   – population size (above 20.000 calculation results don’t change much)
   – response distribution
   with the recommended input the number of participants is 655.

   This is around the number of participants Fishkin uses in DP.

   Now we can tweak the numbers accordingly the OECD “good practice for citizens assemblies”. We start now with the number of participants in mind we like to have. OECD propose to use 80 % response distribution and we see the number of participants go down to 173. With a little bit more tweaking we get what we want. With a response distribution of 99% we only need 11 participants.
   Demanding statistical representativity is clearly not enough. That is why Fishkin conducts additional tests with control groups.

   Of course there are other calculators and notions but Raosoft are the basic ones I think https://www.gigacalculator.com/calculators/power-sample-size-calculator.php I don’t even understand what they are talking about. Statistical power seems an interesting notion but I can’t tell anything about it.

   LikeLike
8. 

   paulnollen, on October 30, 2023 at 4:46 pm said:

   On the other end of spectrum we find among others the “Oregon CIR” or alike. A small panel (+/- 20) that provides comments on a Citizen’s initiative. https://participedia.net/case/6439 It is up to us to study the provided documentation and make up our mind (selection by questionnaires and algorithms) .

   The demoscan team, based at the University of Geneva and funded by the Swiss National Science Foundation, involved a core research team of 6, an advisory board of 5, and a group of 7 volunteers[9]. Cato Leonard, the co-founder of G1000 in Belgium, was hired as head moderator and assisted by Antoine Andre. The team also received support from Healthy Democracy, a non-profit organization which has overseen numerous CIRs in Oregon[10]. The entity’s process expert, Linn Davis, helped to apply contextual adaptations where needed so that the CIR was appropriate for Sion.
   As the organizers wanted to use the real register of inhabitants for random selection, garnering municipal support was crucial.
   …
   Overall, 205 individuals expressed an interest in participating[15], equating to 10.2% of those who received the invitation letter. As is often the case with recruitment processes involving self-selection[16], skewed demographics were noted within the group of first-stage respondents. Individuals with higher levels of formal education and stronger levels of political interest displayed greater willingness to participate. Right-wing citizens and those over the age of 65 were also under-represented[17]. However, the demoscan team used stratified sortition in the second stage of recruitment to reduce such imbalances. On the 17th September, all 205 citizens were invited to a public session which would involve the final selection of the CIR panel. During this session, Sortition Foundation was used to draw two panels of 20 members from the sample of 205. These panels were then assigned a colour, either red or yellow, and the youngest individual in the room was asked to pick one of two coloured bars out of a non-transparent bag. The colour of the bar selected determined the composition of the final panel[18].

   Participedia is a good source for the study of all kind of events in the political area.
   I hope I have provided some additional information to start the study with a bit more knowledge.

   LikeLike

This site uses Akismet to reduce spam. Learn how your comment data is processed.