Data Dependent Priors in PAC-Bayes Bounds

John Shawe-Taylor (), Emilio Parrado-Hernández () and Amiran Ambroladze
Additional contact information
John Shawe-Taylor: University College London, Dept. of Computer Science & CSML
Emilio Parrado-Hernández: University Carlos III of Madrid, Dept. of Signal Processing and Communications

A chapter in Proceedings of COMPSTAT'2010, 2010, pp 231-240 from Springer

Abstract: Abstract One of the central aims of Statistical Learning Theory is the bounding of the test set performance of classifiers trained with i.i.d. data. For Support Vector Machines the tightest technique for assessing this so-called generalisation error is known as the PAC-Bayes theorem. The bound holds independently of the choice of prior, but better priors lead to sharper bounds. The priors leading to the tightest bounds to date are spherical Gaussian distributions whose means are determined from a separate subset of data. This paper gives another turn of the screw by introducing a further data dependence on the shape of the prior: the separate data set determines a direction along which the covariance matrix of the prior is stretched in order to sharpen the bound. In addition, we present a classification algorithm that aims at minimizing the bound as a design criterion and whose generalisation can be easily analysed in terms of the new bound. The experimental work includes a set of classification tasks preceded by a bound-driven model selection. These experiments illustrate how the new bound acting on the new classifier can be much tighter than the original PAC-Bayes Bound applied to an SVM, and lead to more accurate classifiers.

Keywords: PAC Bayes Bound; Support Vector Machines; generalization prediction; model selection (search for similar items in EconPapers)
Date: 2010
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-7908-2604-3_21

Ordering information: This item can be ordered from
http://www.springer.com/9783790826043

DOI: 10.1007/978-3-7908-2604-3_21

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().