Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program

Liu Zhang and Mark Esposito

Papers from arXiv.org

Abstract: Firms increasingly evaluate deployed AI agents using repeated human ratings, yet observed score changes may reflect the measurement process as much as changes in the agent itself. Firms also rarely know how long an evaluation remains informative as deployed systems evolve. We study reliability as a latent state using 2,611 rubric-scored production interviews from a deployed voice-and-video interviewing agent, overlapping human reviewers, the agent's change log, and support tickets. Reviewer severity varied substantially: two reviewers assessing the same batches differed by 0.79 standard deviations on the composite index, and a shift in reviewer composition flattened the raw trend. After adjusting for reviewer effects and smoothing batch-level estimates, evaluated reliability increased by 0.53 standard deviations from March to August 2026. Reliability also shifted materially across logged deploys. At the estimated rate of unaccounted movement, forecast uncertainty reaches the magnitude of the largest observed deploy contrast after about five weeks, although this horizon is imprecisely estimated and based on a small number of step-like changes. An independent operational outcome moved in the same direction: interview-related support tickets fell by about 10% per month relative to technical-issue tickets. These data do not identify a causal effect of evaluation on performance. Instead, they show that longitudinal AI evaluation requires three checks: comparability across raters, evidence currency under drift, and corroboration with consequential operational outcomes.

Date: 2026-10
References: Add references at CitEc
Citations:

Downloads: (external link)
https://arxiv.org/pdf/2610.07003 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2610.07003

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().