Statistical Problem Classes and Their Links to Information Theory

Bertrand Clarke, Jennifer Clarke and Chi Wai Yu

Econometric Reviews, 2014, vol. 33, issue 1-4, 337-371

Abstract: We begin by recalling the tripartite division of statistical problems into three classes, M-closed, M-complete, and M-open and then reviewing the key ideas of introductory Shannon theory. Focusing on the related but distinct goals of model selection and prediction, we argue that different techniques for these two goals are appropriate for the three different problem classes. For M-closed problems we give relative entropy justification that the Bayes information criterion (BIC) is appropriate for model selection and that the Bayes model average is information optimal for prediction. For M-complete problems, we discuss the principle of maximum entropy and a way to use the rate distortion function to bypass the inaccessibility of the true distribution. For prediction in the M-complete class, there is little work done on information based model averaging so we discuss the Akaike information criterion (AIC) and its properties and variants. For the M-open class, we argue that essentially only predictive criteria are suitable. Thus, as an analog to model selection, we present the key ideas of prediction along a string under a codelength criterion and propose a general form of this criterion. Since little work appears to have been done on information methods for general prediction in the M-open class of problems, we mention the field of information theoretic learning in certain general function spaces.

Date: 2014
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://hdl.handle.net/10.1080/07474938.2013.807190 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:emetrv:v:33:y:2014:i:1-4:p:337-371

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/LECR20

DOI: 10.1080/07474938.2013.807190

Access Statistics for this article

Econometric Reviews is currently edited by Dr. Essie Maasoumi

More articles in Econometric Reviews from Taylor & Francis Journals
Bibliographic data for series maintained by ().