Research on three-step accelerated gradient algorithm in deep learning

Yongqiang Lian, Yincai Tang and Shirong Zhou

Statistical Theory and Related Fields, 2022, vol. 6, issue 1, 40-57

Abstract: Gradient descent (GD) algorithm is the widely used optimisation method in training machine learning and deep learning models. In this paper, based on GD, Polyak's momentum (PM), and Nesterov accelerated gradient (NAG), we give the convergence of the algorithms from an initial value to the optimal value of an objective function in simple quadratic form. Based on the convergence property of the quadratic function, two sister sequences of NAG's iteration and parallel tangent methods in neural networks, the three-step accelerated gradient (TAG) algorithm is proposed, which has three sequences other than two sister sequences. To illustrate the performance of this algorithm, we compare the proposed algorithm with the three other algorithms in quadratic function, high-dimensional quadratic functions, and nonquadratic function. Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning. For conveniently facilitate the proposed algorithms, we rewite the R package ‘neuralnet’ and extend it to ‘supneuralnet’. All kinds of deep learning algorithms in this paper are included in ‘supneuralnet’ package. Finally, we show our algorithms are superior to other algorithms in four case studies.

Date: 2022
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/24754269.2020.1846414 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:tstfxx:v:6:y:2022:i:1:p:40-57

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/tstf20

DOI: 10.1080/24754269.2020.1846414

Access Statistics for this article

Statistical Theory and Related Fields is currently edited by Zhao Wei

More articles in Statistical Theory and Related Fields from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().