Regularizing axis-aligned ensembles via data rotations that favor simpler learners

Rico Blaser and Piotr Fryzlewicz

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: To overcome the inherent limitations of axis-aligned base learners in ensemble learning, several methods of rotating the feature space have been discussed in the literature. In particular, smoother decision boundaries can often be obtained from axis-aligned ensembles by rotating the feature space. In the present paper, we introduce a low-cost regularization technique that favors rotations which produce compact base learners. The restated problem adds a shrinkage term to the loss function that explicitly accounts for the complexity of the base learners. For example, for tree-based ensembles, we apply a penalty based on the median number of nodes and the median depth of the trees in the forest. Rather than jointly minimizing prediction error and model complexity, which is computationally infeasible, we first generate a prioritized weighting of the available feature rotations that promotes lower model complexity and subsequently minimize prediction errors on each of the selected rotations. We show that the resulting ensembles tend to be significantly more dense, faster to evaluate, and competitive at generalizing in out-of-sample predictions.

Keywords: random rotation; regularization; ensemble learning; minimal complexity (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 12 pages
Date: 2021-02-01
New Economics Papers: this item is included in nep-ore
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Published in Statistics and Computing, 1, February, 2021, 31(2). ISSN: 0960-3174

Downloads: (external link)
http://eprints.lse.ac.uk/107935/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:107935

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().