Local Linear Approximation Algorithm for Neural Network

Mudong Zeng, Yujie Liao, Runze Li and Agus Sudjianto
Additional contact information
Mudong Zeng: Department of Statistics, Pennsylvania State University, University Park, PA 16802, USA
Yujie Liao: Department of Statistics, Pennsylvania State University, University Park, PA 16802, USA
Runze Li: Department of Statistics, Pennsylvania State University, University Park, PA 16802, USA
Agus Sudjianto: Corporate Model Risk, Wells Fargo Bank, Charlotte, NC 28202, USA

Mathematics, 2022, vol. 10, issue 3, 1-22

Abstract: This paper aims to develop a new training strategy to improve efficiency in estimation of weights and biases in a feedforward neural network (FNN). We propose a local linear approximation (LLA) algorithm, which approximates ReLU with a linear function at the neuron level and estimate the weights and biases of one-hidden-layer neural network iteratively. We further propose the layer-wise optimized adaptive neural network (LOAN), in which we use the LLA to estimate the weights and biases in the LOAN layer by layer adaptively. We compare the performance of the LLA with the commonly-used procedures in machine learning based on seven benchmark data sets. The numerical comparison implies that the proposed algorithm may outperform the existing procedures in terms of both training time and prediction accuracy.

Keywords: local linear approximation; fisher scoring; layerwise optimization (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/3/494/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/3/494/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:3:p:494-:d:741782

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().