Same Text, Different Filename: Are LLM Judges Biased?

Sergey Popov

No E2026/13, Cardiff Economics Working Papers from Cardiff University, Cardiff Business School, Economics Section

Abstract: Yes. In 2026, leading economic journals started using Refine.ink in the technical quality assessment of submissions before publication. I study whether renaming a file changes the perception of the document’s quality by LLM judges. I find that three frontier LLMs score the same document differently if you change the document’s filename. A person-related label scores higher than no label; there is no consistent preference of own- or rival-based label, nor a penalty for an Asian name, but there is a context-specific bias in assessments whose direction and size vary with both document and label. With LLM-written documents, some self-premium can be detected, but a significant context-dependent label bias remains. Whether to combat it is a question of the costs of evaluation errors.

Keywords: Large Language Models; LLM-as-a-Judge; Peer Review; Algorithmic Bias (search for similar items in EconPapers)
JEL-codes: C91 J71 (search for similar items in EconPapers)
Pages: 16 pages
Date: 2026-11
References: Add references at CitEc
Citations:

Downloads: (external link)
http://carbsecon.com/wp/E2026_13.pdf (application/pdf)

Related works:
Working Paper: Same Text, Different Filename: Are LLM Judges Biased? (2026)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cdf:wpaper:2026/13

Access Statistics for this paper

More papers in Cardiff Economics Working Papers from Cardiff University, Cardiff Business School, Economics Section Contact information at EDIRC.
Bibliographic data for series maintained by Yongdeng Xu ().