Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization

Juan Gao (), Xin-Wei Liu (), Yu-Hong Dai (), Yakui Huang () and Junhua Gu ()
Additional contact information
Juan Gao: Hebei University of Technology
Xin-Wei Liu: Hebei University of Technology
Yu-Hong Dai: Chinese Academy of Sciences
Yakui Huang: Hebei University of Technology
Junhua Gu: Hebei University of Technology

Computational Optimization and Applications, 2023, vol. 84, issue 2, No 9, 572 pages

Abstract: Abstract We consider a distributed non-convex optimization problem of minimizing the sum of all local cost functions over a network of agents. This problem often appears in large-scale distributed machine learning, known as non-convex empirical risk minimization. In this paper, we propose two accelerated algorithms, named DSGT-HB and DSGT-NAG, which combine the distributed stochastic gradient tracking (DSGT) method with momentum accelerated techniques. Under appropriate assumptions, we prove that both algorithms sublinearly converge to a neighborhood of a first-order stationary point of the distributed non-convex optimization. Moreover, we derive the conditions under which DSGT-HB and DSGT-NAG achieve a network-independent linear speedup. Numerical experiments for a distributed non-convex logistic regression problem on real data sets and a deep neural network on the MNIST database show the superiorities of DSGT-HB and DSGT-NAG compared with DSGT.

Keywords: Distributed non-convex optimization; Machine learning; Momentum methods; Optimization algorithms; Convergence rate (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10589-022-00432-5 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:coopap:v:84:y:2023:i:2:d:10.1007_s10589-022-00432-5

Ordering information: This journal article can be ordered from
http://www.springer.com/math/journal/10589

DOI: 10.1007/s10589-022-00432-5

Access Statistics for this article

Computational Optimization and Applications is currently edited by William W. Hager

More articles in Computational Optimization and Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().