Exploring Flatter Loss Landscape Surface via Sharpness-Aware Minimization with Linear Mode Connectivity

Hailun Liang, Haowen Zheng, Hao Wang, Liu He, Haoyi Lin and Yanyan Liang ()
Additional contact information
Hailun Liang: School of Computer Science and Engineering, Macau University of Science and Technology, Macau 999078, China
Haowen Zheng: School of Computer Science and Engineering, Macau University of Science and Technology, Macau 999078, China
Hao Wang: School of Computer Science and Engineering, Macau University of Science and Technology, Macau 999078, China
Liu He: School of Computer Science and Engineering, Macau University of Science and Technology, Macau 999078, China
Haoyi Lin: School of Computer Science and Engineering, Macau University of Science and Technology, Macau 999078, China
Yanyan Liang: School of Computer Science and Engineering, Macau University of Science and Technology, Macau 999078, China

Mathematics, 2025, vol. 13, issue 8, 1-26

Abstract: The Sharpness-Aware Minimization (SAM) optimizer connects flatness and generalization, suggesting that loss basins with lower sharpness are correlated with better generalization. However, SAM requires manually tuning the open ball radius, which complicates its practical application. To address this, we propose a method inspired by linear connectivity, using two models initialized differently as endpoints to automatically determine the optimal open ball radius. Specifically, we introduce distance regularization between the two models during training, which encourages them to approach each other, thus dynamically adjusting the open ball radius. We design an optimization algorithm called ’Twin Stars Entwined’ (TSE), where the stopping condition is defined by the models’ linear connectivity, i.e., when they converge to a region of sufficiently low distance. As the models iteratively reduce their distance, they converge to a flatter region of the loss landscape. Our approach complements SAM by dynamically identifying flatter regions and exploring the geometric properties of multiple connected loss basins. Instead of searching for a single large-radius basin, we identify a group of connected basins as potential optimization targets. Experiments conducted across multiple models and in varied noise environments showed that our method achieved a performance on par with state-of-the-art techniques.

Keywords: sharpness-aware minimization (SAM); linear mode connectivity (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/8/1259/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/8/1259/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:8:p:1259-:d:1632862

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().