Recent Advances in Optimization Methods for Machine Learning: A Systematic Review

Xiaodong Liu, Huaizhou Qi, Suisui Jia, Yongjing Guo and Yang Liu ()
Additional contact information
Xiaodong Liu: School of Telecommunications Engineering, Xidian University, Xi’an 710071, China
Huaizhou Qi: School of Telecommunications Engineering, Xidian University, Xi’an 710071, China
Suisui Jia: School of Telecommunications Engineering, Xidian University, Xi’an 710071, China
Yongjing Guo: School of Telecommunications Engineering, Xidian University, Xi’an 710071, China
Yang Liu: School of Telecommunications Engineering, Xidian University, Xi’an 710071, China

Mathematics, 2025, vol. 13, issue 13, 1-29

Abstract: This systematic review explores modern optimization methods for machine learning, distinguishing between gradient-based techniques using derivative information and population-based approaches employing stochastic search. Key innovations focus on enhanced regularization, adaptive control mechanisms, and biologically inspired strategies to address challenges like scaling to large models, navigating complex non-convex landscapes, and adapting to dynamic constraints. These methods underpin core ML tasks including model training, hyperparameter tuning, and feature selection. While significant progress is evident, limitations in scalability and theoretical guarantees persist, directing future work toward more robust and adaptive frameworks to advance AI applications in areas like autonomous systems and scientific discovery.

Keywords: optimization methods; machine learning; gradient-based optimization; swarm intelligence; deep learning (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/13/2210/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/13/2210/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:13:p:2210-:d:1696073

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().