Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization
Juan Gao (),
Xin-Wei Liu (),
Yu-Hong Dai (),
Yakui Huang () and
Junhua Gu ()
Additional contact information
Juan Gao: Hebei University of Technology
Xin-Wei Liu: Hebei University of Technology
Yu-Hong Dai: Chinese Academy of Sciences
Yakui Huang: Hebei University of Technology
Junhua Gu: Hebei University of Technology
Computational Optimization and Applications, 2023, vol. 84, issue 2, No 9, 572 pages
Abstract We consider a distributed non-convex optimization problem of minimizing the sum of all local cost functions over a network of agents. This problem often appears in large-scale distributed machine learning, known as non-convex empirical risk minimization. In this paper, we propose two accelerated algorithms, named DSGT-HB and DSGT-NAG, which combine the distributed stochastic gradient tracking (DSGT) method with momentum accelerated techniques. Under appropriate assumptions, we prove that both algorithms sublinearly converge to a neighborhood of a first-order stationary point of the distributed non-convex optimization. Moreover, we derive the conditions under which DSGT-HB and DSGT-NAG achieve a network-independent linear speedup. Numerical experiments for a distributed non-convex logistic regression problem on real data sets and a deep neural network on the MNIST database show the superiorities of DSGT-HB and DSGT-NAG compared with DSGT.
Keywords: Distributed non-convex optimization; Machine learning; Momentum methods; Optimization algorithms; Convergence rate (search for similar items in EconPapers)
References: View references in EconPapers View complete reference list from CitEc
Citations: Track citations by RSS feed
Downloads: (external link)
http://link.springer.com/10.1007/s10589-022-00432-5 Abstract (text/html)
Access to the full text of the articles in this series is restricted.
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
Persistent link: https://EconPapers.repec.org/RePEc:spr:coopap:v:84:y:2023:i:2:d:10.1007_s10589-022-00432-5
Ordering information: This journal article can be ordered from
Access Statistics for this article
Computational Optimization and Applications is currently edited by William W. Hager
More articles in Computational Optimization and Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().