LLM-as-a-Judge: Mastering Systematic Biases for Reliable Information Systems Adoption

LLM-juges: contrôle des biais systématiques pour une adoption fiable des Systèmes d'Information basés sur l'IA

Jonathan Piaget (), Ulysse Rosselet () and Cédric Gaspoz ()
Additional contact information
Jonathan Piaget: HES-SO - Haute École spécialisée de Suisse occidentale = HES-SO University of Applied Sciences and Arts Western Switzerland = Fachhochschule Westschweiz [Schweiz]
Ulysse Rosselet: HES-SO - Haute École spécialisée de Suisse occidentale = HES-SO University of Applied Sciences and Arts Western Switzerland = Fachhochschule Westschweiz [Schweiz]
Cédric Gaspoz: HES-SO - Haute École spécialisée de Suisse occidentale = HES-SO University of Applied Sciences and Arts Western Switzerland = Fachhochschule Westschweiz [Schweiz]

Post-Print from HAL

Abstract: This article explores the LLM-as-a-Judge paradigm as a critical Information System (IS) whose reliability and neutrality are crucial factors for its organizational adoption. The study confirms the robustness of self-preference and highlights a massive position bias, indicating that the judgment is more sensitive to form and provenance than to intrinsic quality. This preference of LLM-judges for synthetic content reveals a risk of stylistic misalignment with human standards. The presence of these structural biases constitutes a major impediment to trust and adoption. Explicit and continuous bias measurement, along with the integration of diversified, blind, and human-calibrated evaluation pipelines, are essential to ensure reliable deployment.

Date: 2026-05-20
New Economics Papers: this item is included in nep-inv
Note: View the original document on HAL open archive server: https://hal.science/hal-05789338v1
References: Add references at CitEc
Citations:

Published in AIM 2026 : 31e conférence de l'Association Information et Management, Haute École spécialisée de Suisse occidentale = HES-SO University of Applied Sciences and Arts Western Switzerland = Fachhochschule Westschweiz [Schweiz] [HES-SO], May 2026, Neuchâtel, Suisse

Downloads: (external link)
https://hal.science/hal-05789338v1/document (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:journl:hal-05789338

Access Statistics for this paper

More papers in Post-Print from HAL
Bibliographic data for series maintained by CCSD ().