Same Text, Different Filename: Are LLM Judges Biased?

Sergey Popov

Papers from arXiv.org

Abstract: Yes. In 2026, leading economic journals started using Refine.ink in the technical quality assessment of submissions before publication. I study whether renaming a file changes the perception of the document's quality by LLM judges. I find that three frontier LLMs score the same document differently if you change the document's filename. A person-related label scores higher than no label; there is no consistent preference of own- or rival-based label, nor a penalty for an Asian name, but there is a context-specific bias in assessments whose direction and size vary with both document and label. With LLM-written documents, some self-premium can be detected, but a significant context-dependent label bias remains. Whether to combat it is a question of the costs of evaluation errors.

Date: 2026-11
References: Add references at CitEc
Citations:

Downloads: (external link)
https://arxiv.org/pdf/2611.01829 Latest version (application/pdf)

Related works:
Working Paper: Same Text, Different Filename: Are LLM Judges Biased? (2026)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2611.01829

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().