Mean Field Analysis of Deep Neural Networks

Justin Sirignano () and Konstantinos Spiliopoulos ()
Additional contact information
Justin Sirignano: Mathematical Institute, University of Oxford, Oxford OX2 6GG, United Kingdom
Konstantinos Spiliopoulos: Department of Mathematics and Statistics, Boston University, Boston, Massachusetts 02215

Mathematics of Operations Research, 2022, vol. 47, issue 1, 120-152

Abstract: We analyze multilayer neural networks in the asymptotic regime of simultaneously (a) large network sizes and (b) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multilayer neural network output. The limit procedure is valid for any number of hidden layers, and it naturally also describes the limiting behavior of the training loss. The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization. The limit satisfies a system of deterministic integro-differential equations. The proof uses methods from weak convergence and stochastic analysis. We show that, under suitable assumptions on the activation functions and the behavior for large times, the limit neural network recovers a global minimum (with zero loss for the objective function).

Keywords: Primary: 60F99; 68T07; deep learning; deep neural network; mean field limit; weak convergence (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/moor.2020.1118 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormoor:v:47:y:2022:i:1:p:120-152

Access Statistics for this article

More articles in Mathematics of Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().