Phase transition analysis for shallow neural networks with arbitrary activation functions

Otavio Citton, Frederieke Richert and Michael Biehl

Physica A: Statistical Mechanics and its Applications, 2025, vol. 660, issue C

Abstract: In this work we extend the statistical physics based analysis of so-called soft committee machines (SCM), two-layered feedforward neural networks with differentiable hidden unit activation, in student–teacher model scenarios. In particular, we study phase transitions with the training set size as observed in off-line learning for SCM with arbitrary activation functions. The analysis is facilitated by expanding the relevant quantities in a basis of Hermite polynomials and truncating the expansion to a relatively small number of terms. We derive a sufficient condition for the presence of a second order phase transition and find an analytical expression for the point in training where an unspecialized phase becomes unstable. In addition we derive an expression for the asymptotic learning behavior for general activation functions. Moreover, we demonstrate that the approach facilitates the treatment of mismatched scenarios, where student and teacher network have different activation functions. This enables the investigation of more realistic learning scenarios within the statistical physics framework.

Keywords: Statistical physics; Phase transition; Neural networks; Activation function; Machine learning (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0378437125000081
Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:phsmap:v:660:y:2025:i:c:s0378437125000081

DOI: 10.1016/j.physa.2025.130356

Access Statistics for this article

Physica A: Statistical Mechanics and its Applications is currently edited by K. A. Dawson, J. O. Indekeu, H.E. Stanley and C. Tsallis

More articles in Physica A: Statistical Mechanics and its Applications from Elsevier
Bibliographic data for series maintained by Catherine Liu ().