A Derivative-Incorporated Adaptive Gradient Method for Federated Learning

Huimin Gao, Qingtao Wu, Hongyan Cao (), Xuhui Zhao, Junlong Zhu and Mingchuan Zhang
Additional contact information
Huimin Gao: College of Information Engineering, Henan University of Science and Technology, Luoyang 471023, China
Qingtao Wu: College of Information Engineering, Henan University of Science and Technology, Luoyang 471023, China
Hongyan Cao: China Research Institute of Radiowave Propagation, Qingdao 266107, China
Xuhui Zhao: College of Information Engineering, Henan University of Science and Technology, Luoyang 471023, China
Junlong Zhu: College of Information Engineering, Henan University of Science and Technology, Luoyang 471023, China
Mingchuan Zhang: College of Information Engineering, Henan University of Science and Technology, Luoyang 471023, China

Mathematics, 2023, vol. 11, issue 15, 1-21

Abstract: As a new machine learning technique, federated learning has received more attention in recent years, which enables decentralized model training across data silos or edge intelligent devices in the Internet of Things without exchanging local raw data. All kinds of algorithms are proposed to solve the challenges in federated learning. However, most of these methods are based on stochastic gradient descent, which undergoes slow convergence and unstable performance during the training stage. In this paper, we propose a differential adaptive federated optimization method, which incorporates an adaptive learning rate and the gradient difference into the iteration rule of the global model. We further adopt the first-order moment estimation to compute the approximate value of the differential term so as to avoid amplifying the random noise from the input data sample. The theoretical convergence guarantee is established for our proposed method in a stochastic non-convex setting under full client participation and partial client participation cases. Experiments for the image classification task are performed on two standard datasets by training a neural network model, and experiment results on different baselines demonstrate the effectiveness of our proposed method.

Keywords: federated learning; stochastic gradient descent; learning rate; differential (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/15/3403/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/15/3403/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:15:p:3403-:d:1210489

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().