Evaluating and tracking qualitative content coder performance using item response theory

Michael Hennessy (), Amy Bleakley and Morgan E. Ellithorpe
Additional contact information
Michael Hennessy: University of Delaware
Amy Bleakley: University of Delaware
Morgan E. Ellithorpe: University of Delaware

Quality & Quantity: International Journal of Methodology, 2023, vol. 57, issue 2, No 11, 1245 pages

Abstract: Abstract Content analysis of traditional and social media has a central role in investigating features of media content, measuring media exposure, and calculation of media effects. The reliability of content coding is usually evaluated using “kappa-centric” agreement measures, but these measures produce results that aggregate individual coder decisions which obscure the performance of individual coders. Using a data set of 105 advertisements for sports and energy drinks media content coded by five coders, we demonstrate that Item Response Theory can track coder performance over time and give coder-specific information on the consistency of decisions over qualitatively coded objects. We conclude that IRT should be added to content analysts’ tool kit of useful methodologies to track and evaluate content coders’ performance.

Keywords: Content analysis; coder agreement; Item response theory (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11135-022-01397-7 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:qualqt:v:57:y:2023:i:2:d:10.1007_s11135-022-01397-7

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11135

DOI: 10.1007/s11135-022-01397-7

Access Statistics for this article

Quality & Quantity: International Journal of Methodology is currently edited by Vittorio Capecchi

More articles in Quality & Quantity: International Journal of Methodology from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().