EconPapers    
Economics at your fingertips  
 

Fairness Trade-offs in Expert Evaluation: Information, Judgment, and Procedural Design

Karol Borowiecki and Marc Law

No AWP-08-2026, ACEI Working Paper Series from Association for Cultural Economics International

Abstract: Evaluation systems often seek to reduce bias, prevent manipulation, preserve relevant information, maintain comparability, and treat candidates equally, but these fairness goals can conflict. This paper develops a framework for understanding such trade-offs in expert evaluation—including peer review, grant panels, academic hiring, and artistic competitions—where quality is multidimensional and legitimate disagreement among evaluators is possible. We treat the choice among evaluation procedures as a problem of constitutional design: rules must be selected before the specific candidates, evaluators, and circumstances that will test them are known. Classical music competitions are our primary analytical setting because their rules, scores, and controversies make these trade-offs unusually visible, but the framework applies to other settings in which expert judgments are aggregated under incomplete information. The framework highlights two core tensions: information control and outlier rules. First, when information is both quality-relevant and identity-revealing, procedures that suppress bias-relevant cues may also remove information with genuine evaluative content. Second, because sincere minority judgment and strategic manipulation can generate similar observed score patterns, score-based exclusion rules may reduce manipulation while limiting legitimate dissent. Two further trade-offs concerning transparency and standardization are developed as illustrations. The analysis shows that no procedure can eliminate all vulnerabilities at once. Beyond this design implication, the framework also helps explain why evaluation rules differ across stages and settings, and why well-intentioned reforms may create new vulnerabilities.

Keywords: constitutional design; information aggregation; fairness; committee decisionmaking; competitions; bias; manipulation (search for similar items in EconPapers)
JEL-codes: D02 D63 D71 D72 D82 Z11 (search for similar items in EconPapers)
Pages: 30 pages
Date: 2026-11
New Economics Papers: this item is included in nep-des
References: Add references at CitEc
Citations:

Downloads: (external link)
http://files.culturaleconomics.org/papers/AWP-08-2026.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cue:wpaper:awp-08-2026

Access Statistics for this paper

More papers in ACEI Working Paper Series from Association for Cultural Economics International Contact information at EDIRC.
Bibliographic data for series maintained by Paul Crosby ().

 
Page updated 2026-11-17
Handle: RePEc:cue:wpaper:awp-08-2026