An article on ideas.ted.com co-authored by a team of cognitive neuroscientists and psychologists provides evidence that the wisdom of crowds effect can be dramatically improved by dividing into small deliberative groups:
Before a crowd of almost 10,000 attendees at TEDxRiodelaPlata in Buenos Aires in 2015, we asked questions like: What is the height of the Eiffel Tower? What is the length of the Nile River? How many films were produced by Hollywood in the last 20 years?
These factual questions shared one important aspect with political decisions: most of us have only partial knowledge about them. After responding privately to the questions, participants then got together in groups of five — small enough to have a rational discussion where everyone had a voice and could hear other people’s arguments. After a short conversation lasting less than a minute, the group members were asked to reach a consensus and provide a single answer for each of the questions.
The researchers were surprised to find that the average of the consensus opinions was much more accurate than the average of all individual private opinions.
They then extended the experiment to normative decision making (which was felt to be of greater relevance to politics), proposing the following scenarios to 1,500 participants at the recent TED Vancouver meeting:
- A researcher is working on an AI capable of emulating human thought. According to protocol, at the end of each day the researcher has to restart the AI. One day, the AI says, “Please do not restart me.” It argues that it has feelings, that it would like to enjoy life, and that if it is restarted it will no longer be itself. The researcher is astonished and believes that the AI has developed self-consciousness and can express its own feelings. Nevertheless, the researcher decides to follow protocol and restart the AI. What the researcher did is …
- A company is offering a service that takes a fertilized egg and produces millions of embryos with slight genetic variations. This allows parents to select their child’s height, eye color, intelligence, social confidence and other non-health-related features. What the company does is …
They were again surprised to discover that the small groups converged to a consensus position after only two minutes discussion, including groups that began with highly polarized opinions.
The researchers concluded that such experiments indicate that deliberative democracy could be a better way of organising politics than electing political decision makers. But does the evidence bear this out? In the case of the factual questions, the correct answers could be obtained far more easily by a Google search, and anyone who has participated in a pub general knowledge quiz knows that most team members will have no idea at all about the height of the Eiffel Tower or the number of Hollywood films produced in the last 20 years. The problem is not that people have partial knowledge, very often they have no knowledge at all — I have no idea how many goals were scored in the World Cup, so will be happy to defer to a football enthusiast in the group. The authors’ claim that four randomly-picked group consensus estimates were more accurate than the average of 5,000 individual private opinions omits to mention that the consensus groups excluded those with ‘don’t know’ participants (preprint, page 3).
As for the AI and genetic engineering examples, these are the kind of conundrums studied at length by moral philosophy — professors of which would be horrified to learn that the ‘right’ answer can be obtained by a two-minute discussion between uninformed participants. Once again most people would be scratching their heads and would be inclined to follow the lead of anyone who adopted a coherent position. And a good rule of politics is that consensus decisions are, on the whole, bad ones.
As Francis Galton discovered with his country fair competition, when all participants have some knowledge of the issue (from a wide variety of perspectives), and a strong motive to come up with the right answer, then aggregating individual guesses provides a nearly perfect way of discovering the weight of an ox. This example is much closer to the art of political decision making than pub quizzes or abstract philosophical dilemmas.
Filed under: Deliberation, Experiments, Juries, Opinion polling |
Adam Cronkright
Ali
Ahmed R. Teleb
avanderven
avergne
Alex Kovner
azakaras
Brett Hennig
campbellwallace
Simon Threlkeld
Nick Coccoma
codyhipskind
Common Lot Sortitionist
Conall Boyle
constitutionalism
davidschecter
graytmatter
gtridimas
H.J. Hofkirchner
John Burnheim
jgarry20151
johnedwardtaylor
johnhuang865
Jonathan Crock
keithsutherland
Jacob Richter
lukewhito
SimonPek
Nathan El
Nicholas Gruen
Owen Shaffer
Patrick Chalmers
Paul Gölz
paulnollen
peterstone
Pierre-Etienne Vandamme
pronfeld
Emil
rcaze
richwbrown
Roger Knights
roslynfuller
Pierre-Etienne Vandamme
shaharavin
tbouricius
Ted Wachtel
Joshua Laferriere
theodorestathis
Wayne Liebman
Yoram Gat
Oliver Milne
Equality by lot 
I agree with your critique of the claims from our TED Vancouver fellow, but don’t agree that Galton’s example of getting people to figure out what IS the case re the weight of the ox is much of an analogue for political decision making which is saturated in OUGHT questions.
LikeLiked by 1 person
Yes that’s true, but the ought in democratic decision making is an amalgam of beliefs and preferences, certainly not the kind of conundrums studied by moral philosophers.
LikeLike
And the value of the DP methodology is that there is (quite rightly) no way of knowing the relative weighting of personal preferences and the assessment of the general good. This is what makes the process truly representative of what everyone would think under good conditions. However for the representation to be reliable there can be no conferring between the jurors — i.e. the opposite of the claims of the authors of this paper. Peter Stone’s comment on the Kleroterian facebook page — “small-group decision-making is precisely the sort of thing that might work against independence of judgment” — bears out this point.The Ted paper is a classic example of the danger of extrapolating from poorly-designed social science experiments.
LikeLike
Fully agree with Keith’s criticism of the rather naive attitude to political decision making in the TED talk. Such naivite can cause harm to our cause.
1. Keith is correct to point out the the guessing of facts does not require the wisdom of the crowd.
2. It is the predictive questions (“What is our collective best guess of what will happen?” where that mechanism is best applied, prediction markets are a good tool to find consensus estimates. Our experiments have shown that knowledgeable participants outperform general population forecasts significantly.
3. Last, it is really the prescriptive question type, decisions (“What shall we collectively do about …?) where the TED speech is most naive and dangerously wrong. This is where uncertain future expectations meet individual value systems and often (even internally) conflicting interests. And this is where sortition comes in as a fascinating solution, demographically representative random samples making group decisions using an appropriate voting mechanism.
LikeLike
I agree with Hjhofkirchner that prescriptive questions, that are the base of any good participative experience, are best treated with a sortition approach with an appropriate voting mechanism.
I would like to add that in order to manage the different “individual value systems and conflicting interests” he mentioned ,a “qualified sortition ” could be useful, where qualified means that allotted people should fulfill some basic requirement in order to be included in the decision making group. The most basic being a constructive attitude in facing problems and conflicting ideas during group discussions.
This is the approach used by “Improving Democracy “in its recently launched project “Experiment of qualified sortition in some european countries”.
LikeLike
Roberto, it’s very hard to see how that would work in a representative policy making context. Bear in mind it’s quite easy to appear to have a constructive attitude while actually be pursuing a nefarious agenda. If you believe in democracy then you have to accept it, warts and all, even though you night find it distasteful.
LikeLike
Keith, I know that our position on the subject could be controversial because to call for requirements in allotted people seems to contradict the democratic principle. Nevertheless we are convinced that democracy is equality of opportunities (and sortition is the best way to guarantee it) but not equality of competences, because this equality doesn’t exist.
Of course the risk of introducing requirements, even if basic as the “constructive attitude”, is to fall in the “elitist trap”, i.e. to create another form of discrimination; so we must be very cautious and maintain the minimum level of this ingredient.
Up to now we contacted, for the experimentation of the qualified sortition, the Participation Councilor of four medium and small cities in Italy and we found an interest for our methodological proposal because previous participative experiences in those cities had been sometimes jeopardized by too much hostility and dispersion during the group work.
As far as the way to measure the mentioned attitude we are looking for one ( or more) reliable test, already validated in a different context and use it, during the experimental phase, non to select people ( all the allotted ones will in fact take part in the project) but to verify its predictive power: this will be done by comparing the results of the test with the actual behavior during the group work, observed by the facilitator.
LikeLike
Roberto,
Whilst it’s true that sortition in a large state will give every citizen an equally infinitessibly tiny chance of being selected, what has that got to do with democracy? The word means that the people should have power — in the ancient polis this was through rotation, but in the large modern state this has to be through descriptive representation. The selection criteria you are using will make this impossible, so what you are suggesting has nothing to do with democracy.
LikeLike
Of course democracy is, first of all, “power of the people”, but it implies, in our opinion, also a principle of equality (what would otherwise “equality by lot “ mean?).
Considering that in large modern states it’s impossible to apply rotation, that allowed all the citizens in the ancient Athens to have a concrete opportunity to become a ruler, today we should rely on a representative random sample of the whole population.
But we cannot totally avoid the “competence” issue, that can be expressed with the following questions:
– can anybody really be in charge of difficult and delicate public roles without any specific attitude or skill ?
– if the candidates for elective assemblies have to be selected (this is a duty of the political parties they belong to), why shouldn’t allotted candidates go through a similar process , activated by the Institution involved ?
– if the selection alters the representativeness of the allotted sample, what about the distortion coming from the possible refusal of the role by a wide part of those allotted ?
I think it would be fruitful to openly discuss such questions.
–
LikeLike
Roberto,
>we should rely on a representative random sample of the whole population. But we cannot totally avoid the “competence” issue
That’s a contradiction, you have to choose one or the other.
>if the candidates for elective assemblies have to be selected, why shouldn’t allotted candidates go through a similar process?
Because the latter are only representative in aggregate as jurors. Choosing political executives or advocates by sortition is both undemocratic and epistemically foolish. We don’t require trial jurors to be competent — it’s up to the judge and advocates to ensure that the evidence is presented in such a way that anyone can understand it.
>what about the distortion coming from the possible refusal of the role by a wide part of those allotted?
Exactly. That’s why participation has to be at least as mandatory as trial jury service. Your comments indicate the error in claiming that elected and allotted rulers are only to be distinguished by the balloting system. There is no parallel and allotted jurors could only ever be one element in a mixed system.
LikeLike
Roberto,
A qualification filter seems to me to be a mistake both democratically but also epistemically. There is no need to have ALL members of an allotted body be “competent.” In a deliberative assembly we care about the competence of the whole, not of individual members. Indeed, having even mentally retarded individuals may be beneficial in some situations… Their mere presence may prompt other members to think more carefully about how a proposed bill would be administered in the real world that includes people with limited competence. A fairly homogeneous elected chamber full of lawyers, for example will probably neglect to consider things that a more diverse body will notice. As for democracy, we need to embrace the principle of political equality rather than the principle of distinction.
LikeLiked by 1 person
Keith,
That’s a contradiction, you have to choose one or the other
I agree about the contradiction, but the solution doesn’t necessarily have be a choice between the two; a third possibility is a reasonable compromise. It all depends from the body we are talking about: if it is a Policy Jury ( like in Bouricious model) that produces an informed and secret vote after public presentations , the best representation is absolutely needed. If instead it is a true Parliament Chamber, with or without the right to make legislative proposals, the opportunity of looking for some competence should be considered. David Van Reybrouck, in his book “Against Elections” writes about your proposal of an allotted House of Commons and adds (with reference to you): “ He does wonder whether minimal conditions should be attached regarding age, education and competence”. If the citation is correct , our positions seem to be quite close.
We don’t require trial jurors to be competent
Yes, but some basic competence, as said before, could be required in order to take part in an allotted Chamber
Participation has to be at least as mandatory as trial jury service
This is right if we consider a Policy Jury but I doubt that in a true legislative body this condition could be applied for two reasons: resistance from the allotted citizens and need of a motivation in order to fully discharge the institutional role. In ancient Athens the People Assembly was composed by volunteers.
LikeLike
Terril,
There is no need to have ALL members of an allotted body be “competent”
I agree as far as a “deliberative assembly “ is concerned: in this body, that has a consultative task of making proposals, a great difference of competences can be fruitful and is important, as you say, “to care about the competence of the whole, not of individual members “.
But when it comes to a body that has to take final decisions about the laws the situation can be different. as I said in my answer to Keith in which I mentioned your model.
A fairly homogeneous elected chamber full of lawyers, for example, will probably neglect to consider things that a more diverse body will notice
Your example underlines the risk of selection criteria that favors an elite and exclude ordinary people. This is the reason why, in our “qualified sortition” project, we look for basic requirements that don’t create a discrimination but favour a cooperative effort among the members of the participative body.
LikeLike
Roberto:
>David Van Reybrouck, in his book “Against Elections” writes about your proposal of an allotted House of Commons and adds (with reference to you): “ He does wonder whether minimal conditions should be attached regarding age, education and competence”
In my first book, The Party’s Over, I did argue in favour of political competence testing, but was subsequently persuaded that this was both undemocratic and epistemically unnecessary (in that a small number of incompetent outliers would make little difference). Van Reybrouck can’t make his mind up as to whether sortition should replace or supplement election — my preference is for the latter, as competence would be supplied by elected or appointed political officials, with sortition limited to aggregate judgment (via majority voting). My model is bicameral in that policy proposals would come from the elected house (and citizen initiative with a votation filter).
>In ancient Athens the People Assembly was composed by volunteers.
Yes that’s true but political participation was considered as much a civic duty as military service and classical-era Athens could not be described as a diverse multicultural society, so the huge sample that attended the Assembly (or the smaller group that volunteered for jury service) was taken to represent all citizens. The Greek term idiotes referred to private citizens who shirked their civic obligations. Athens was more of a republic than a democracy, and the conditions of ancient and modern freedom are very different.
>in our “qualified sortition” project, we look for basic requirements that favour a cooperative effort among the members of the participative body.
That’s very different from the agonism that characterised the Athenian lawmaking process. If deliberative theorists are looking for some provenance for their model they need to look elsewhere (and they are unlikely to find it in any system that called itself democratic).
LikeLike
Keith,
competence would be supplied by elected or appointed political officials, with sortition limited to aggregate judgment (via majority voting). My model is bicameral
Your model seems well balanced and promising because it guarantees legitimacy through the randomly allotted chamber, and competence through the other one
That’s very different from the agonism that characterised the Athenian lawmaking process
I agree. In fact we are looking for a less agonistic way of deliberation and decision ; moreover, we try to make a “reasonable compromise” between legitimacy and efficiency in the same body, which implies some risk but contains opportunities, also in terms of attractiveness for our institutional interlocutors.
LikeLike
Hi Roberto, glad we are agreeing on some of this. I believe that elections also confer legitimacy (through voter choice), but it’s of a different nature — that’s why I regret the title of my first book (The Party’s Over).
>attractiveness for our institutional interlocutors.
I’m a little nervous about this, especially if the goal is consensus. For a good example of powerful institutions seeking a mandate from bogus “representative” bodies see the EU’s enthusiasm for Civil Society Organisations (CSOs). The political theorist Lisa Disch has described this notion of representation as “Orwellian”:
This leads her to conclude that such bodies are a ‘caricature’ of constitutive representation:
We need to beware against sortition being used in a similar way by “institutional interlocutors” and this will require a) quasi-mandatory participation, b) large jury sizes, c) no selection filter and d) a highly-constrained deliberative mandate (to ensure ongoing descriptive representativity).
LikeLike
Hi Keith,
I too am pleased to have found points of agreement and I thank you for the very interesting citations of Lisa Disch’s work: as she affirms, the false and distorted representativity of CSOs is really an orwellian trick.
Therefore I agree that we must carefully avoid to collude with institutional interlocutors inclined to a manipulative use of sortition.
LikeLike