Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

Jérôme Bolte and Edouard Pauwels

No 19-1044, TSE Working Papers from Toulouse School of Economics (TSE)

Abstract: Modern problems in AI or in numerical analysis require nonsmooth approaches with a exible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic diffrentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.

Keywords: Deep Learning, Automatic differentiation, Backpropagation algorithm,; Nonsmooth stochastic optimization, Defiable sets, o-minimal structures, Stochastic gradient, Clarke subdifferential, First order methods (search for similar items in EconPapers)
Date: 2019-10
New Economics Papers: this item is included in nep-big, nep-cmp and nep-ore
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://www.tse-fr.eu/sites/default/files/TSE/docu ... 2019/wp_tse_1044.pdf Full Text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:tse:wpaper:123631

Access Statistics for this paper

More papers in TSE Working Papers from Toulouse School of Economics (TSE) Contact information at EDIRC.
Bibliographic data for series maintained by ().