Repeated judgment elicitation: tapping the wisdom of crowds in individuals

Abstract

<jats:title>Abstract</jats:title> <jats:p>It is commonly recognised that the Wisdom of Crowds [1] enables a group of people with limited knowledge to make, on average, very accurate estimates. This process underlies the success of prediction markets [2], which are used to accurately predict outcomes as disparate as elections and box office takings. In the O&amp;G industry, however, there are clear limits on the number of people who can be canvassed for their opinions. Not only are there a limited number of people with sufficient knowledge to make any estimate at all for a particular problem but confidentiality restricts this further. Recent psychological research, however, has demonstrated benefits of repeated individual judgments – asking the same person to repeatedly estimate a parameter. We critically review individual, repeated-elicitation techniques currently in use and describe a method that avoids many of the problems with these – More-Or-Less Elicitation (MOLE) [3].</jats:p> <jats:p>The MOLE is compared with alternate, single- and repeated-judgments elicitation methods, yielding superior accuracy and calibration to any of the alternate techniques. Its estimates explain an additional 20% of the variance in the parameter of interest over its nearest rival and less than 9% of its elicited ranges did not contain the true value when expected to. We argue that, for the O&amp;G industry, repeated individual judgments have the potential to greatly improve the accuracy and calibration of estimates and, further, that the MOLE harnesses the benefits of repeated judgments while avoiding common problems such as repetition of answers and confirmation bias.</jats:p> <jats:p>This paper reviews the latest research on repeated judgments in elicitation and demonstrates the benefits of repeated, individual judgments for elicited values. The MOLE, which takes advantage of these benefits, is simple and easily transferable to most elicitation domains, enabling its benefits to apply throughout the industry.</jats:p>