TECP: Token-Entropy Conformal Prediction for LLMs

Beining Xu and Yongming Lu ()
Additional contact information
Beining Xu: School of Engineering, Shenzhen MSU-BIT University, Shenzhen 518000, China
Yongming Lu: MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University, Shenzhen 518000, China

Mathematics, 2025, vol. 13, issue 20, 1-14

Abstract: Uncertainty quantification (UQ) for open-ended language generation remains a critical yet underexplored challenge, particularly in settings where token-level log-probabilities are available during decoding. We present Token-Entropy Conformal Prediction (TECP) , which treats a log-probability-based token-entropy statistic as a nonconformity score and integrates it with split conformal prediction to construct prediction sets with finite-sample coverage guarantees. We work in a white-box regime in which per-token log-probabilities are accessible during decoding. TECP estimates episodic uncertainty from the token-entropy structure of sampled generations and calibrates thresholds via conformal quantiles to ensure provable error control. Empirical evaluations across six large language models and two QA benchmarks (CoQA and TriviaQA) show that TECP consistently achieves reliable coverage and compact prediction sets, outperforming prior self-UQ methods. These results provide a principled and efficient solution for trustworthy generation in white-box, log-probability-accessible LLM settings.

Keywords: token-entropy; conformal prediction; predictive uncertainty; coverage (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/20/3351/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/20/3351/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:20:p:3351-:d:1776184

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().