Measuring and Visualizing Coders’ Reliability: New Approaches and Guidelines From Experimental Data

Iasonas Lamprianou

Sociological Methods & Research, 2023, vol. 52, issue 1, 525-553

Abstract: This study investigates inter- and intracoder reliability, proposing a new approach based on social network analysis (SNA) and exponential random graph models (ERGM). During a recent exit poll, the responses of voters to two open-ended questions were recorded. A coding experiment was conducted where a group of coders coded a sample of text segments. Analyzing the data, we show that the proposed SNA/ERGM method extends significantly our analytical leverage, beyond what popular tools such as Krippendorff’s α and Fleiss’s κ have to offer. The reliability of coding for individual coders differed significantly for the two questions although they were very similar and the same codebook was used. We conclude that the main advantages of the proposed SNA/ERGM method are the intuitive visualizations and the nuanced measurements. Detailed guidelines are provided for practitioners who would like to use the proposed method in operational settings.

Keywords: coding; reliability; SNA; ERGM; qualitative (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0049124120926198 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:somere:v:52:y:2023:i:1:p:525-553

DOI: 10.1177/0049124120926198

Access Statistics for this article

More articles in Sociological Methods & Research
Bibliographic data for series maintained by SAGE Publications ().