Resolution of similar patterns in a solvable model of unsupervised deep learning with structured data

Andrea Baroffio, Pietro Rotondo and Marco Gherardi

Chaos, Solitons & Fractals, 2024, vol. 182, issue C

Abstract: Empirical data, on which deep learning relies, has substantial internal structure, yet prevailing theories often disregard this aspect. Recent research has led to the definition of structured data ensembles, aimed at equipping established theoretical frameworks with interpretable structural elements, a pursuit that aligns with the broader objectives of spin glass theory. We consider a one-parameter structured ensemble where data consists of correlated pairs of patterns, and a simplified model of unsupervised learning, whereby the internal representation of the training set is fixed at each layer. A mean field solution of the model identifies a set of layer-wise recurrence equations for the overlaps between the internal representations of an unseen input and of the training set. The bifurcation diagram of this discrete-time dynamics is topologically inequivalent to the unstructured one, and displays transitions between different phases, selected by varying the load (the number of training pairs divided by the width of the network). The network’s ability to resolve different patterns undergoes a discontinuous transition to a phase where signal processing along the layers dissipates differential information about an input’s proximity to the different patterns in a pair. A critical value of the parameter tuning the correlations separates regimes where data structure improves or hampers the identification of a given pair of patterns.

Keywords: Deep learning; Structured disorder; Bifurcations; Mean field (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0960077924004004
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:chsofr:v:182:y:2024:i:c:s0960077924004004

DOI: 10.1016/j.chaos.2024.114848

Access Statistics for this article

Chaos, Solitons & Fractals is currently edited by Stefano Boccaletti and Stelios Bekiros

More articles in Chaos, Solitons & Fractals from Elsevier
Bibliographic data for series maintained by Thayer, Thomas R. ().