Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks

Tengyuan Liang () and Hai Tran-Bach ()
Additional contact information
Tengyuan Liang: University of Chicago - Booth School of Business
Hai Tran-Bach: University of Chicago - Department of Statistics

No 2020-151, Working Papers from Becker Friedman Institute for Research In Economics

Abstract: We utilize a connection between compositional kernels and branching processes via Mehler’s formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of activation functions in compositional neural networks. We study the unscaled and rescaled limits of the compositional kernels and explore the different phases of the limiting behavior, as the compositional depth increases. We investigate the memorization capacity of the compositional kernels and neural networks by characterizing the interplay among compositional depth, sample size, dimensionality, and non-linearity of the activation. Explicit formulas on the eigenvalues of the compositional kernel are provided, which quantify the complexity of the corresponding reproducing kernel Hilbert space. On the methodological front, we propose a new random features algorithm, which compresses the compositional layers by devising a new activation function.

Pages: 41 pages
Date: 2020
New Economics Papers: this item is included in nep-cmp
References: View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://repec.bfi.uchicago.edu/RePEc/pdfs/BFI_WP_2020151.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bfi:wpaper:2020-151

Access Statistics for this paper

More papers in Working Papers from Becker Friedman Institute for Research In Economics Contact information at EDIRC.
Bibliographic data for series maintained by Toni Shears ( this e-mail address is bad, please contact ).