Quantifying the development of agreement among experts in Delphi studies

J.V. Meijering, J.K. Kampen and H. Tobi

Technological Forecasting and Social Change, 2013, vol. 80, issue 8, 1607-1614

Abstract: Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.

Keywords: Delphi; Consensus; Agreement; Association; Methodology; Simulation (search for similar items in EconPapers)
Date: 2013
References: Add references at CitEc
Citations: View citations in EconPapers (19)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0040162513000048
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:tefoso:v:80:y:2013:i:8:p:1607-1614

DOI: 10.1016/j.techfore.2013.01.003

Access Statistics for this article

Technological Forecasting and Social Change is currently edited by Fred Phillips

More articles in Technological Forecasting and Social Change from Elsevier
Bibliographic data for series maintained by Catherine Liu ().