Accelerating incremental gradient optimization with curvature information

Hoi-To Wai (), Wei Shi, César A. Uribe (), Angelia Nedić () and Anna Scaglione ()
Additional contact information
Hoi-To Wai: Chinese University of Hong Kong
Wei Shi: Princeton University
César A. Uribe: Massachusetts Institute of Technology
Angelia Nedić: Arizona State University
Anna Scaglione: Arizona State University

Computational Optimization and Applications, 2020, vol. 76, issue 2, No 3, 347-380

Abstract: Abstract This paper studies an acceleration technique for incremental aggregated gradient (IAG) method through the use of curvature information for solving strongly convex finite sum optimization problems. These optimization problems of interest arise in large-scale learning applications. Our technique utilizes a curvature-aided gradient tracking step to produce accurate gradient estimates incrementally using Hessian information. We propose and analyze two methods utilizing the new technique, the curvature-aided IAG (CIAG) method and the accelerated CIAG (A-CIAG) method, which are analogous to gradient method and Nesterov’s accelerated gradient method, respectively. Setting $$\kappa$$κ to be the condition number of the objective function, we prove the R linear convergence rates of $$1 - \frac{4c_0 \kappa }{(\kappa +1)^2}$$1-4c0κ(κ+1)2 for the CIAG method, and $$1 - \sqrt{\frac{c_1}{2\kappa }}$$1-c12κ for the A-CIAG method, where $$c_0,c_1 \le 1$$c0,c1≤1 are constants inversely proportional to the distance between the initial point and the optimal solution. When the initial iterate is close to the optimal solution, the R linear convergence rates match with the gradient and accelerated gradient method, albeit CIAG and A-CIAG operate in an incremental setting with strictly lower computation complexity. Numerical experiments confirm our findings. The source codes used for this paper can be found on http://github.com/hoitowai/ciag/.

Keywords: Incremental aggregated gradient method; Curvature information; Nesterov accelerated gradient method (search for similar items in EconPapers)
Date: 2020
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10589-020-00183-1 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:coopap:v:76:y:2020:i:2:d:10.1007_s10589-020-00183-1

Ordering information: This journal article can be ordered from
http://www.springer.com/math/journal/10589

DOI: 10.1007/s10589-020-00183-1

Access Statistics for this article

Computational Optimization and Applications is currently edited by William W. Hager

More articles in Computational Optimization and Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().