Large-scale group decision-making to facilitate inter-rater reliability of human-factors analysis for the railway system

Jian-Lan Zhou, Ren-Fang Tu and Hai Xiao

Reliability Engineering and System Safety, 2022, vol. 228, issue C

Abstract: The interdependency analysis of human factors usually needs to identify and label factors from accident/incident reports using a binary framework. The labeling procedure involves enormous knowledge and requires a large group of raters, but the raters’ opinions usually highly contradict each other. Thus, inter-rater reliability should be implemented to improve the labeling consistency. However, the inter-rater reliability coefficient is commonly computed on a category-by-category basis, which is inefficient especially when there are many raters. To overcome this deficiency, a feedback-autonomy-based consensus model is proposed to determine the inter-rater reliability of human-factors labeling results involving a large group of raters. The proposed model can manage various non-cooperative behaviors and provide feedback adjustment suggestions to raters for the labeling procedure. Notably, the autonomy mechanism allows raters to freely choose and adjust labels referring to the reference opinions without causing over-adjustment issue. The inter-rater reliability and the interdependency results are performed on 279 accident/incident reports. We apply a commercial tool to obtain the reference results, which are consistent with the unanimous group opinions. Besides, other comparative studies clarify the advantages of the proposed consensus model. The proposed model is convenient for reducing conflicts among raters that usually exist in human-factors analysis.

Keywords: Inter-rater reliability; Human-factors analysis; HFACS; Large-scale group decision-making; Railway system (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0951832022004252
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:reensy:v:228:y:2022:i:c:s0951832022004252

DOI: 10.1016/j.ress.2022.108806

Access Statistics for this article

Reliability Engineering and System Safety is currently edited by Carlos Guedes Soares

More articles in Reliability Engineering and System Safety from Elsevier
Bibliographic data for series maintained by Catherine Liu ().