AMC: Adaptive Learning Rate Adjustment Based on Model Complexity

Weiwei Cheng, Rong Pu and Bin Wang ()
Additional contact information
Weiwei Cheng: School of Computer Science and Engineering, Northeastern University, Shenyang 110167, China
Rong Pu: School of Computer Science and Engineering, Northeastern University, Shenyang 110167, China
Bin Wang: School of Computer Science and Engineering, Northeastern University, Shenyang 110167, China

Mathematics, 2025, vol. 13, issue 4, 1-23

Abstract: An optimizer plays a decisive role in the efficiency and effectiveness of model training in deep learning. Although Adam and its variants are widely used, the impact of model complexity on training is not considered, which leads to instability or slow convergence when a complex model is trained. To address this issue, we propose an AMC (Adam with Model Complexity) optimizer, which dynamically adjusts the learning rate by incorporating model complexity, thereby improving training stability and convergence speed. AMC uses the Frobenius norm of the model to measure its complexity, automatically decreasing the learning rate of complex models and increasing the learning rate of simple models, thus optimizing the training process. We provide a theoretical analysis to demonstrate the relationship between model complexity and learning rate, as well as the convergence and convergence bounds of AMC. Experiments on multiple benchmark datasets show that, compared to several widely used optimizers, AMC exhibits better stability and faster convergence, especially in the training of complex models.

Keywords: optimizer; Adam; model complexity (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/4/650/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/4/650/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:4:p:650-:d:1592524

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().