Stranded Credentials: Keeping Online Reputation Systems Informative in the AI Era

Song Yao

Papers from arXiv.org

Abstract: Platforms summarize providers' past achievements into credentials that buyers use to judge quality. Generative AI can now produce much of the work those achievements certify, raising fears that those quality signals are worthless. We audit how well such credentials stay informative in Kaggle's 2010-2026 archive, where medals are won on predictions scored against withheld answers and two evaluation formats ran side by side. Across 444,698 participations, a medal's power to predict performance sits almost entirely in its first year, in both formats and eras. Fresh medals kept most of their value through the AI transition. About half of the collapse in the informativeness of one format's medals is institutional: the platform had been phasing out that format before AI, and its medal stock aged out on schedule. Old medals look more informative only in isolation. The platform's official lifetime-tier display discards up to a sixth of the medals' predictive power. A recency-weighted index fit before the AI era explains AI-era performance about 13% better than the tiers and selects entrants who perform better on average, though the tiers still identify extreme top performers better. Displaying a recent-performance summary alongside the lifetime tiers would recover the discarded information for buyers.

Date: 2026-08, Revised 2026-09
New Economics Papers: this item is included in nep-ain
References: Add references at CitEc
Citations:

Downloads: (external link)
https://arxiv.org/pdf/2608.17111 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2608.17111

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().