An Information Theoretic Comparison of Model Selection Criteria

Dean Foster () and Robert A. Stine

No 1180, Discussion Papers from Northwestern University, Center for Mathematical Studies in Economics and Management Science

Abstract: Information theory offers a coherent perspective on model selection. As in Rissanen's original application of information theory to model selection, our perspective arises from viewing a model as a component of a compressed representation of data in a two-part code. The first part of such a code is an explicit representation of the model used to compress the data. Simpler models have shorter representations. The second part is the encoded data itself. Models which fit better compress the data into shorter sequences. The objective is to choose the model which produces the shortest total message length, requiring an explicit trade-off of model complexity (length of the first part) versus goodness-of-fit (length of second part). In adidtion to Rissanen's MDL criterion, this perspective illuminates the properties of numerous model selection criteria, including AIC, Cp, BIC, RIC, and EBIC. We show that each corresponds to a specific way of coding the model parameters. By selecting the model that minimizes the total message length, our representations of these criteria reproduce their more familiar definitions. Examples from wavelets illustrate the use of these methods.

Date: 1997-02
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.kellogg.northwestern.edu/research/math/papers/1180.pdf main text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nwu:cmsems:1180

Ordering information: This working paper can be ordered from

Access Statistics for this paper

More papers in Discussion Papers from Northwestern University, Center for Mathematical Studies in Economics and Management Science Center for Mathematical Studies in Economics and Management Science, Northwestern University, 580 Jacobs Center, 2001 Sheridan Road, Evanston, IL 60208-2014. Contact information at EDIRC.
Bibliographic data for series maintained by Fran Walker ( this e-mail address is bad, please contact ).