Information criterion for approximation of unnormalized densities

John Y Choe, Yen-Chi Chen and Nick Terry

PLOS ONE, 2025, vol. 20, issue 3, 1-15

Abstract: This paper considers the problem of approximating an unknown density when it can be evaluated up to a normalizing constant at a finite number of points. This density approximation problem is ubiquitous in statistics, such as approximating a posterior density for Bayesian inference and estimating an optimal density for importance sampling. We consider a parametric approximation approach and cast it as a model selection problem to find the best model in pre-specified distribution families (e.g., select the best number of Gaussian mixture components and their parameters). This problem cannot be addressed with traditional approaches that maximize the (marginal) likelihood of a model, for example, using the Akaike information criterion (AIC) or Bayesian information criterion (BIC). We instead aim to minimize the cross-entropy that gauges the deviation of a parametric model from the target density. We propose a novel information criterion called the cross-entropy information criterion (CIC) and prove that the CIC is an asymptotically unbiased estimator of the cross-entropy (up to a multiplicative constant) under some regularity conditions. We propose an iterative method to approximate the target density by minimizing the CIC. We demonstrate how the proposed method selects a parametric model that well approximates the target density through multiple numerical studies in the Supporting Information.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0317430 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 17430&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0317430

DOI: 10.1371/journal.pone.0317430

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().