Agreement Without Accuracy: Six Systematic Defects an LLM Annotation Study Could Not See in Its Own Reliability Statistics

Hon-Ren Lin
Additional contact information
Hon-Ren Lin: National Taipei University of Business

No n3zhf_v1, SocArXiv from Center for Open Science

Abstract: Researchers who use large language models as annotators are commonly asked to demonstrate reliability, and test–retest agreement under a fixed seed is the cheapest evidence. We report a study in which it was systematically misleading. Annotating 1,902 student UML class specifications under a fixed snapshot, seed and zero temperature, we compared three prompt configurations, each run twice. A three-stage pipeline appeared to raise answer-level agreement from .647 to .845, with chance-corrected agreement rising for every code and internal consistency satisfied throughout. Comparing per-code prevalence led us to inspect outputs; five systematic defects emerged: an instruction promising an input never supplied (wrong on 64.7% of its flags at κ = .971); a stage receiving element names stripped of the types it was to judge; residual error after repair; enumerated conditions firing outside their scope; and alignment tables stored without half of what they summarised. A sixth lay in our checking procedure: the audit rule we wrote agreed with a blind reading on only 67.5% of cases, and the twenty cases read to confirm it were drawn from its own output. A blind reading of forty random submissions, judged before any output was seen, is the study's only anchored estimate; under it our one reported repair cannot be shown to help. None was surfaced by a reliability statistic; those that altered the model's inputs or rules raised agreement by easing its task. Reliability from repeated runs should be accompanied by prevalence comparisons and a sample read blind, drawn independently of the detector.

Date: 2026-09-25
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6ab6010afd0ff1b8f7f191b9/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:n3zhf_v1

DOI: 10.31235/osf.io/n3zhf_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().