A Taxonomy and Theoretical Analysis of Collapse Phenomena in Unsupervised Representation Learning

Donghyeon Kim, Chae-Bong Sohn, Do-Yup Kim () and Dae-Yeol Kim ()
Additional contact information
Donghyeon Kim: Department of Defense Acquisition Program, Kwangwoon University, Seoul 01897, Republic of Korea
Chae-Bong Sohn: Department of Electronics and Communications Engineering, Kwangwoon University, Seoul 01897, Republic of Korea
Do-Yup Kim: Department of Information and Telecommunication Engineering, Incheon National University, Incheon 22012, Republic of Korea
Dae-Yeol Kim: Department of Artificial Intelligence, Kyungnam University, Changwon 51767, Republic of Korea

Mathematics, 2025, vol. 13, issue 18, 1-28

Abstract: Unsupervised representation learning has emerged as a promising paradigm in machine learning, owing to its capacity to extract semantically meaningful features from unlabeled data. Despite recent progress, however, such methods remain vulnerable to collapse phenomena, wherein the expressiveness and diversity of learned representations are severely degraded. This phenomenon poses significant challenges to both model performance and generalizability. This paper presents a systematic investigation into two distinct forms of collapse: complete collapse and dimensional collapse. Complete collapse typically arises in non-contrastive frameworks, where all learned representations converge to trivial constants, thereby rendering the learned feature space non-informative. While contrastive learning has been introduced as a principled remedy, recent empirical findings indicate that it falls to prevent collapse entirely. In particular, contrastive methods are still susceptible to dimensional collapse, where representations are confined to a narrow subspace, thus restricting both the information content and effective dimensionality. To address these concerns, we conduct a comprehensive literature analysis encompassing theoretical definitions, underlying causes, and mitigation strategies for each collapse type. We further categorize recent approaches to collapse prevention, including feature decorrelation techniques, eigenvalue distribution regularization, and batch-level statistical constraints, and assess their effectiveness through a comparative framework. This work aims to establish a unified conceptual foundation for understanding collapse in unsupervised learning and to guide the design of more robust representation learning algorithms.

Keywords: unsupervised representation learning; collapse phenomena; complete collapse; dimensional collapse; optimization; contrastive learning; feature decorrelation; representation diversity (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/18/2986/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/18/2986/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:18:p:2986-:d:1750256

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().