Optimization on multifractal loss landscapes explains a diverse range of geometrical and dynamical properties of deep learning

Andrew Ly and Pulin Gong ()
Additional contact information
Andrew Ly: University of Sydney
Pulin Gong: University of Sydney

Nature Communications, 2025, vol. 16, issue 1, 1-14

Abstract: Abstract Gradient descent and its variants are foundational in solving optimization problems across many disciplines. In deep learning, these optimizers demonstrate a remarkable ability to dynamically navigate complex loss landscapes, ultimately converging to solutions that generalize well. To elucidate the mechanism underlying this ability, we introduce a theoretical framework that models the complexities of loss landscapes as multifractal. Our model unifies and explains a broad range of realistic geometrical signatures of loss landscapes, including clustered degenerate minima, multiscale structure, and rich optimization dynamics in deep neural networks, such as the edge of stability, non-stationary anomalous diffusion, and the extended edge of chaos without requiring fine-tuning parameters. We further develop a fractional diffusion theory to illustrate how these optimization dynamics, coupled with multifractal structure, effectively guide optimizers toward smooth solution spaces housing flatter minima, thus enhancing generalization. Our findings suggest that the complexities of loss landscapes do not hinder optimization; rather, they facilitate the process. This perspective not only has important implications for understanding deep learning but also extends potential applicability to other disciplines where optimization unfolds on complex landscapes.

Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41467-025-58532-9 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:16:y:2025:i:1:d:10.1038_s41467-025-58532-9

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-025-58532-9

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().