Prunability of Multi-Layer Perceptrons Trained with the Forward-Forward Algorithm

Mitko Nikov, Damjan Strnad and David Podgorelec ()
Additional contact information
Mitko Nikov: Faculty of Electrical Engineering and Computer Science, University of Maribor, SI-2000 Maribor, Slovenia
Damjan Strnad: Faculty of Electrical Engineering and Computer Science, University of Maribor, SI-2000 Maribor, Slovenia
David Podgorelec: Faculty of Electrical Engineering and Computer Science, University of Maribor, SI-2000 Maribor, Slovenia

Mathematics, 2025, vol. 13, issue 16, 1-23

Abstract: We explore the sparsity and prunability of multi-layer perceptrons (MLPs) trained using the Forward-Forward (FF) algorithm, an alternative to backpropagation (BP) that replaces the backward pass with local, contrastive updates at each layer. We analyze the sparsity of the weight matrices during training using multiple metrics, and test the prunability of FF networks on the MNIST, FashionMNIST and CIFAR-10 datasets. We also propose FFLib—a novel, modular PyTorch-based library for developing, training and analyzing FF models along with a suite of FF-based architectures, including FFNN, FFNN+C and FFRNN. In addition to structural sparsity, we describe and apply a new method for visualizing the functional sparsity of neural activations across different architectures using the HSV color space. Moreover, we conduct a sensitivity analysis to assess the impact of hyperparameters on model performance and sparsity. Finally, we perform pruning experiments, showing that simple FF-based MLPs exhibit significantly greater robustness to one-shot neuron pruning than traditional BP-trained networks, and a possible 8-fold increase in compression ratios while maintaining comparable accuracy on the MNIST dataset.

Keywords: Forward-Forward; sparsity; pruning; model compression; machine learning; neural network (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/16/2668/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/16/2668/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:16:p:2668-:d:1727864

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().