How much data? Measuring the complexity of thematic coding tasks and the amount of training samples needed for supervised modeling

Madison Lore

No rcwhf_v1, SocArXiv from Center for Open Science

Abstract: Traditional methods of thematic analysis are critical for deep, nuanced understanding of complex planning challenges, but the manual intensity of the process limits its applications to increasingly large datasets generated by planning processes. Hybrid, supervised methods of natural language processing offer a solution by allowing researchers to scale theory-driven topics by using traditional methods to first label a sample of the data. However, because the amount of labeled data needed varies across thematic analysis tasks, researchers and practitioners face uncertainty in the time, cost, and feasibility of applying scalable text analysis to planning research and practice. How much data needs to be labeled to achieve acceptable accuracy and how does this depend on the complexity of the task itself? This study proposes theory-informed models for measuring task complexity and defining the relationship between complexity and the number of labeled samples needed. Through the analysis of six thematic analysis tasks in 103 climate action plans (CAPs), we find that by only defining complexity from a codebook of phrases for a topic, akin to the process of manual thematic coding, it is possible to accurately predict the maximum attainable accuracy scores for any given training size and labeling a very small sample of data improves the accuracy of the prediction. This framework provides a way for urban researchers to determine the manual cost of extending a thematic coding task to larger scales before embarking on the process.

Date: 2026-10-28
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6ae1dee196c87493ea020857/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:rcwhf_v1

DOI: 10.31235/osf.io/rcwhf_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().