A comprehensive survey of fractional gradient descent methods and their convergence analysis

Sroor M. Elnady, Mohamed El-Beltagy, Ahmed G. Radwan and Mohammed E. Fouda

Chaos, Solitons & Fractals, 2025, vol. 194, issue C

Abstract: Fractional Gradient Descent (FGD) methods extend classical optimization algorithms by integrating fractional calculus, leading to notable improvements in convergence speed, stability, and accuracy. However, recent studies indicate that engineering challenges—such as tensor-based differentiation in deep neural networks—remain partially unresolved, prompting further investigation into the scalability and computational feasibility of FGD. This paper provides a comprehensive review of recent advancements in FGD techniques, focusing on their approximation methods and convergence properties. These methods are systematically categorized based on their strategies to overcome convergence challenges inherent in fractional-order calculations, such as non-locality and long-memory effects. Key techniques examined include modified fractional-order gradients designed to avoid singularities and ensure convergence to the true extremum. Adaptive step-size strategies and variable fractional-order schemes are analyzed, balancing rapid convergence with precise parameter estimation. Additionally, the application of truncation methods is explored to mitigate oscillatory behavior associated with fractional derivatives. By synthesizing convergence analyses from multiple studies, insights are offered into the theoretical foundations of these methods, including proofs of linear convergence. Ultimately, this paper highlights the effectiveness of various FGD approaches in accelerating convergence and enhancing stability. While also acknowledging significant gaps in practical implementations for large-scale engineering tasks, including deep learning. The presented review serves as a resource for researchers and practitioners in the selection of appropriate FGD techniques for different optimization problems.

Keywords: Fractional calculus; Optimization algorithms; Gradient descent; Variable fractional order (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0960077925001675
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:chsofr:v:194:y:2025:i:c:s0960077925001675

DOI: 10.1016/j.chaos.2025.116154

Access Statistics for this article

Chaos, Solitons & Fractals is currently edited by Stefano Boccaletti and Stelios Bekiros

More articles in Chaos, Solitons & Fractals from Elsevier
Bibliographic data for series maintained by Thayer, Thomas R. ().