Preconditioning meets biased compression for efficient distributed optimization

Vitali Pirau (), Aleksandr Beznosikov, Martin Takáč, Vladislav Matyukhin and Alexander Gasnikov
Additional contact information
Vitali Pirau: Moscow Institute of Physics and Technology
Aleksandr Beznosikov: Moscow Institute of Physics and Technology
Martin Takáč: Mohamed bin Zayed University of Artificial Intelligence
Vladislav Matyukhin: Moscow Institute of Physics and Technology
Alexander Gasnikov: Moscow Institute of Physics and Technology

Computational Management Science, 2024, vol. 21, issue 1, No 14, 22 pages

Abstract: Abstract Methods with preconditioned updates show up well in badly scaled and/or ill-conditioned convex optimization problems. However, theoretical analysis of these methods in distributed setting is not yet provided. We close this issue by studying preconditioned version of the Error Feedback (EF) method, a popular convergence stabilization mechanism for distributed learning with biased compression. We combine EF and EF21 algorithms with preconditioner based on Hutchinson’s approximation to the diagonal of the Hessian. An experimental comparison of the algorithms with the ResNet computer vision model is provided.

Keywords: Non-convex minimization; First-order methods; Accelerated methods (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10287-023-00496-6 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:comgts:v:21:y:2024:i:1:d:10.1007_s10287-023-00496-6

Ordering information: This journal article can be ordered from
http://www.springer. ... ch/journal/10287/PS2

DOI: 10.1007/s10287-023-00496-6

Access Statistics for this article

Computational Management Science is currently edited by Ruediger Schultz

More articles in Computational Management Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().