Request routing services undermine the reproducibility of open weight LLMs

Joe Higton, Joshua A Tucker and Christopher Barrie
Additional contact information
Christopher Barrie: NYU

No sr9hy_v1, SocArXiv from Center for Open Science

Abstract: Large language models (LLMs) are now a standard tool for classifying text in social science. To make these classifications reproducible, researchers are often advised to use open weight models and to set the sampling temperature to zero. We show that this advice is insufficient when models are accessed through a request-routing service, in which queries are sent to different hosting providers through a single API. Using a popular request-routing service, OpenRouter, we classify British manifesto sentences for ideology 100 times per sentence across eight models and 33 hosting providers, alongside five model configurations served on our own hardware, generating approximately 1m classification draws. When using the request-routing service, open weight models consistently fail to return deterministic outputs. Depending on the model, only 26% to 92% of sentences receive the same label in all repeated classifications. This variation is not solely due to different providers fielding queries: for five of the eight models, the majority of classification variance arises within providers. On our own hardware, the same task is perfectly reproducible at temperature=0. We conclude that reproducibility depends on the entire model hosting environment, and recommend that researchers report the fullest available metadata and raw responses when using LLMs.

Date: 2026-08-04
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6a70c53c4be96b842ea19f29/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:sr9hy_v1

DOI: 10.31219/osf.io/sr9hy_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().