Please use this identifier to cite or link to this item:
Scopus Web of Science® Altmetric
Type: Conference paper
Title: Repeated judgment elicitation: tapping the wisdom of crowds in individuals
Author: Welsh, M.
Begg, S.
Citation: SPE Annual Technical Conference and Exhibition, 4-7 October 2009, New Orleans, Louisiana: pp.1-10
Publisher: SPE International
Publisher Place: USA
Issue Date: 2009
ISBN: 9781555632632
Conference Name: SPE Annual Technical Conference and Exhibition (2009 : New Orleans, Louisiana)
Statement of
M. B. Welsh and S. H. Begg
Abstract: <jats:sec> <jats:title>Abstract</jats:title> <jats:p>It is commonly recognised that the Wisdom of Crowds [1] enables a group of people with limited knowledge to make, on average, very accurate estimates. This process underlies the success of prediction markets [2], which are used to accurately predict outcomes as disparate as elections and box office takings. In the O&amp;G industry, however, there are clear limits on the number of people who can be canvassed for their opinions. Not only are there a limited number of people with sufficient knowledge to make any estimate at all for a particular problem but confidentiality restricts this further. Recent psychological research, however, has demonstrated benefits of repeated individual judgments - asking the same person to repeatedly estimate a parameter. We critically review individual, repeated-elicitation techniques currently in use and describe a method that avoids many of the problems with these - More-Or-Less Elicitation (MOLE) [3].</jats:p> <jats:p>The MOLE is compared with alternate, single- and repeated-judgments elicitation methods, yielding superior accuracy and calibration to any of the alternate techniques. Its estimates explain an additional 20% of the variance in the parameter of interest over its nearest rival and less than 9% of its elicited ranges did not contain the true value when expected to. We argue that, for the O&amp;G industry, repeated individual judgments have the potential to greatly improve the accuracy and calibration of estimates and, further, that the MOLE harnesses the benefits of repeated judgments while avoiding common problems such as repetition of answers and confirmation bias.</jats:p> <jats:p>This paper reviews the latest research on repeated judgments in elicitation and demonstrates the benefits of repeated, individual judgments for elicited values. The MOLE, which takes advantage of these benefits, is simple and easily transferable to most elicitation domains, enabling its benefits to apply throughout the industry.</jats:p> <jats:sec> <jats:title>Introduction</jats:title> <jats:p>Recently, the concept of the Wisdom of Crowds [1] (the observation that average or median predictions from a large group tend to be more accurate than individual estimates) has gained significant traction within a variety of disciplines. This work is, however, of limited use in areas where expert knowledge is required simply to understand what is meant by a question and where the supply of experts is limited - either in an absolute sense or due to restraints such as confidentiality or cost of consulting them, as is commonly the case in the oil industry. However, the underlying idea of repeated judgements is appealing and current psychological work [3–6] is focussed on using the insights from the Wisdom of Crowds literature to assist in elicitation tasks where only a single expert is available.</jats:p> <jats:p>This paper gives a general introduction to the problems commonly associated with obtaining estimates from experts (elicitation) and discusses the benefits known to result from the Wisdom of Crowds and why these are unlikely to help oil and gas decisions. The reasons why similar benefits could be expected from appropriately structured individual repeated judgment tasks are then discussed before we review the main findings from a number of papers discussing such tasks. Finally, we discuss their potential for application to decision making in the petroleum industry.</jats:p> </jats:sec> <jats:sec> <jats:title>Elicitation of Uncertainty</jats:title> <jats:p>Elicitation is the conversion of individual or group's beliefs into numerical form - whether a point estimate or a probability distribution. Generally, this is done not for its own sake but to, for example, predict future outcome ranges, or provide inputs for forecasting models [7]. That is, we generally approach experts for their opinions in an attempt to reduce our uncertainty regarding the likelihood of different possible outcomes of some event of interest. Within the oil and gas industry, for example, values elicited from experts in a variety of fields are used to inform exploration, appraisal, project development and on-going operational decisions.</jats:p> </jats:sec> </jats:sec>
Rights: Copyright 2009, Society of Petroleum Engineers
RMID: 0020097395
DOI: 10.2118/125124-MS
Appears in Collections:Australian School of Petroleum publications

Files in This Item:
There are no files associated with this item.

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.