Speeding up L-BFGS by direct approximation of the inverse Hessian matrix

Ashkan Sadeghi-Lotfabadi () and Kamaledin Ghiasi-Shirazi ()
Additional contact information
Ashkan Sadeghi-Lotfabadi: Ferdowsi University of Mashhad
Kamaledin Ghiasi-Shirazi: Ferdowsi University of Mashhad

Computational Optimization and Applications, 2025, vol. 91, issue 1, No 9, 283-310

Abstract: Abstract L-BFGS is one of the widely used quasi-Newton methods. Instead of explicitly storing an approximation H of the inverse Hessian, L-BFGS keeps a limited number of vectors that can be used for computing the product of H by the gradient. However, this computation is sequential, each step depending on the outcome of the previous step. To solve this problem, we propose the Direct L-BFGS (DirL-BFGS) method that, seeing H as a linear operator, directly stores a low-rank plus diagonal (LRPD) representation of H. Employing the LRPD representation enables us to leverage the benefits of vector processing, leading to accelerating and parallelizing the calculations in the form of single instruction, multiple data. We evaluate our proposed method on different quadratic optimization problems and several regression and classification tasks with neural networks. Numerical results show that DirL-BFGS is faster overall than L-BFGS.

Keywords: Limited-memory BFGS; Low-rank plus diagonal approximation; Vectorization; Single instruction multiple data (SIMD) (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10589-025-00665-0 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:coopap:v:91:y:2025:i:1:d:10.1007_s10589-025-00665-0

Ordering information: This journal article can be ordered from
http://www.springer.com/math/journal/10589

DOI: 10.1007/s10589-025-00665-0

Access Statistics for this article

Computational Optimization and Applications is currently edited by William W. Hager

More articles in Computational Optimization and Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().