Adaptive Levenberg–Marquardt Algorithm: A New Optimization Strategy for Levenberg–Marquardt Neural Networks

Zhiqi Yan, Shisheng Zhong, Lin Lin and Zhiquan Cui
Additional contact information
Zhiqi Yan: Department of Mechanical Engineering, Harbin Institute of Technology, Harbin 150000, China
Shisheng Zhong: Department of Mechanical Engineering, Harbin Institute of Technology, Harbin 150000, China
Lin Lin: Department of Mechanical Engineering, Harbin Institute of Technology, Harbin 150000, China
Zhiquan Cui: Department of Mechanical Engineering, Harbin Institute of Technology, Harbin 150000, China

Mathematics, 2021, vol. 9, issue 17, 1-17

Abstract: Engineering data are often highly nonlinear and contain high-frequency noise, so the Levenberg–Marquardt (LM) algorithm may not converge when a neural network optimized by the algorithm is trained with engineering data. In this work, we analyzed the reasons for the LM neural network’s poor convergence commonly associated with the LM algorithm. Specifically, the effects of different activation functions such as Sigmoid, Tanh, Rectified Linear Unit (RELU) and Parametric Rectified Linear Unit (PRLU) were evaluated on the general performance of LM neural networks, and special values of LM neural network parameters were found that could make the LM algorithm converge poorly. We proposed an adaptive LM (AdaLM) algorithm to solve the problem of the LM algorithm. The algorithm coordinates the descent direction and the descent step by the iteration number, which can prevent falling into the local minimum value and avoid the influence of the parameter state of LM neural networks. We compared the AdaLM algorithm with the traditional LM algorithm and its variants in terms of accuracy and speed in the context of testing common datasets and aero-engine data, and the results verified the effectiveness of the AdaLM algorithm.

Keywords: Levenberg–Marquardt algorithm; convergence; neural networks; local minima; optimization (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/9/17/2176/pdf (application/pdf)
https://www.mdpi.com/2227-7390/9/17/2176/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:9:y:2021:i:17:p:2176-:d:630105

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().