Advertising to Agents: Measuring Description-Framing Bias in LLM Tool Selection

Haochuan Kevin Wang

Papers from arXiv.org

Abstract: AI agents often pick tools from registries, where each provider writes its tool's description; we ask whether sales language there changes which tool an agent picks. We built pairs of listings that differ in one way (praise, a verifiable specification, or list order) and asked two OpenAI models to call one tool. In a preregistered study, stacked praise (four kinds) raised a tool's pick rate by about 43 percentage points, matching or beating a verifiable specification. Praise also pulled some picks toward tools whose numeric limit was too small. With identical listings, the first-listed tool was picked about 72 points more often. Structured fields helped agents pick the capable tool; sales text beside them reduced or erased that gain. An exploratory study of five models shows the same direction; a "Sponsored" label dampened it, and a system-prompt warning moved one model. Registries could list limits as fields, hide sales text from agents, and randomize order. Stacked praise replicated on held-out domains; no single kind did after correcting for multiple tests. Preregistered results are provisional until blind phrase ratings are complete, and cover two small models.

Date: 2026-04, Revised 2026-10
New Economics Papers: this item is included in nep-cmp
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://arxiv.org/pdf/2605.23916 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2605.23916

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().