CaAdam: Improving Adam optimizer using connection aware methods

Rémi Genet and Hugo Inzirillo
Additional contact information
Rémi Genet: DRM - Dauphine Recherches en Management - Université Paris Dauphine-PSL - PSL - Université Paris Sciences et Lettres - CNRS - Centre National de la Recherche Scientifique
Hugo Inzirillo: CREST - Centre de Recherche en Économie et Statistique - ENSAI - Ecole Nationale de la Statistique et de l'Analyse de l'Information [Bruz] - X - École polytechnique - IP Paris - Institut Polytechnique de Paris - ENSAE Paris - École Nationale de la Statistique et de l'Administration Économique - IP Paris - Institut Polytechnique de Paris - CNRS - Centre National de la Recherche Scientifique

Working Papers from HAL

Abstract: We introduce a new method inspired by Adam that enhances convergence speed and achieves better loss function minima. Traditional optimizers, includingAdam, apply uniform or globally adjusted learning rates across neural networks without considering their architectural specifics. This architecture-agnosticapproach is deeply embedded in most deep learning frameworks, where optimizers are implemented as standalone modules without direct access to the network's structural information. For instance, in popular frameworks like Keras or PyTorch, optimizersoperate solely on gradients and parameters, without knowledge of layer connectivity or network topology. Our algorithm, CaAdam, explores this overlooked area by introducing connection-aware optimization through carefully designed proxies of architectural information. We propose multiple scaling methodologies that dynamically adjust learning rates based on easily accessible structural properties such as layer depth, connection counts, and gradient distributions. This approach enables more granular optimization while working within the constraints of current deep learning frameworks. Empirical evaluations on standard datasets (e.g., CIFAR-10, Fashion MNIST) show that our method consistently achieves faster convergence and higher accuracy compared to standard Adam optimizer, demonstrating the potential benefits of incorporating architectural awareness in optimization strategies.

Keywords: Machine; Learning (search for similar items in EconPapers)
Date: 2025-02-01
Note: View the original document on HAL open archive server: https://hal.science/hal-04923974v1
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hal.science/hal-04923974v1/document (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-04923974

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().