Statistical Estimation of the Kullback–Leibler Divergence

Alexander Bulinski and Denis Dimitrov
Additional contact information
Alexander Bulinski: Steklov Mathematical Institute of Russian Academy of Sciences, 119991 Moscow, Russia
Denis Dimitrov: Faculty of Mechanics and Mathematics, Lomonosov Moscow State University, 119234 Moscow, Russia

Mathematics, 2021, vol. 9, issue 5, 1-36

Abstract: Asymptotic unbiasedness and L 2 -consistency are established, under mild conditions, for the estimates of the Kullback–Leibler divergence between two probability measures in R d , absolutely continuous with respect to (w.r.t.) the Lebesgue measure. These estimates are based on certain k -nearest neighbor statistics for pair of independent identically distributed (i.i.d.) due vector samples. The novelty of results is also in treating mixture models. In particular, they cover mixtures of nondegenerate Gaussian measures. The mentioned asymptotic properties of related estimators for the Shannon entropy and cross-entropy are strengthened. Some applications are indicated.

Keywords: Kullback–Leibler divergence; Shannon differential entropy; statistical estimates; k-nearest neighbor statistics; asymptotic behavior; Gaussian model; mixtures (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://www.mdpi.com/2227-7390/9/5/544/pdf (application/pdf)
https://www.mdpi.com/2227-7390/9/5/544/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:9:y:2021:i:5:p:544-:d:510564

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().