Classifying Open-Ended Reports: Factors Affecting the Reliability of Occupation Codes

Conrad Frederick G. (), Couper Mick P. () and Sakshaug Joseph W. ()
Additional contact information
Conrad Frederick G.: Survey Research Center, Institute for Social Research, University of Michigan, PO Box 1248, Ann Arbor, MI 48106, U.S.A., and Joint Program in Survey Methodology, 1218 LeFrak Hall, University of Maryland, College Park, MD 20742, U.S.A.
Couper Mick P.: Survey Research Center, Institute for Social Research, University of Michigan, PO Box 1248, Ann Arbor, MI 48106, U.S.A., and Joint Program in Survey Methodology, 1218 LeFrak Hall, University of Maryland, College Park, MD 20742, U.S.A.
Sakshaug Joseph W.: Cathie Marsh Institute for Social Research, University of Manchester, Humanities Bridgeford Street Building, Manchester M13 9PL, U.K., and Institute for Employment Research, Regensburger Str. 104, Nuremberg 90478, Germany.

Journal of Official Statistics, 2016, vol. 32, issue 1, 75-92

Abstract: A source of survey processing error that has received insufficient study to date is the misclassification of open-ended responses. We report on efforts to understand the misclassification of open occupation descriptions in the Current Population Survey (CPS). We analyzed double-coded CPS descriptions to identify which features vary with intercoder reliability. One factor strongly related to reliability was the length of the occupation description: longer descriptions were less reliably coded than shorter ones. This effect was stronger for particular occupation terms. We then carried out an experiment to examine the joint effects of description length and classification “difficulty” of particular occupation terms. For easy occupation terms longer descriptions were less reliably coded, but for difficult occupation terms longer descriptions were slightly more reliably coded than short descriptions. Finally, we observed as coders provided verbal reports on their decision making. One practice, evident in coders’ verbal reports, is their use of informal coding rules based on superficial features of the description. Such rules are likely to promote reliability, though not necessarily validity, of coding. To the extent that coders use informal rules for long descriptions involving difficult terms, this could help explain the observed relationship between description length and difficulty of coding particular terms.

Keywords: Survey processing error; coding error; occupational classification (search for similar items in EconPapers)
Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://doi.org/10.1515/jos-2016-0003 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:vrs:offsta:v:32:y:2016:i:1:p:75-92:n:3

DOI: 10.1515/jos-2016-0003

Access Statistics for this article

Journal of Official Statistics is currently edited by Annica Isaksson and Ingegerd Jansson

More articles in Journal of Official Statistics from Sciendo
Bibliographic data for series maintained by Peter Golla ().