Multiobjective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networks

Ambrogio Maria Bernardelli (), Stefano Gualandi (), Simone Milanesi (), Hoong Chuin Lau () and Neil Yorke-Smith ()
Additional contact information
Ambrogio Maria Bernardelli: Department of Mathematics, University of Pavia, 27100 Pavia, Italy
Stefano Gualandi: Department of Mathematics, University of Pavia, 27100 Pavia, Italy
Simone Milanesi: Department of Mathematics, University of Pavia, 27100 Pavia, Italy
Hoong Chuin Lau: School of Computing and Information Systems, Singapore Management University, Singapore 178902, Singapore
Neil Yorke-Smith: Socio-Technical Algorithmic Research (STAR) Laboratory, Delft University of Technology, 2600 GA Delft, Netherlands

INFORMS Journal on Computing, 2025, vol. 37, issue 3, 623-643

Abstract: Training neural networks (NNs) using combinatorial optimization solvers has gained attention in recent years. In low-data settings, the use of state-of-the-art mixed integer linear programming solvers, for instance, has the potential to exactly train an NN while avoiding computing-intensive training and hyperparameter tuning and simultaneously training and sparsifying the network. We study the case of few-bit discrete-valued neural networks, both binarized neural networks (BNNs) whose values are restricted to ±1 and integer-valued neural networks (INNs) whose values lie in the range { − P , … , P } . Few-bit NNs receive increasing recognition because of their lightweight architecture and ability to run on low-power devices: for example, being implemented using Boolean operations. This paper proposes new methods to improve the training of BNNs and INNs. Our contribution is a multiobjective ensemble approach based on training a single NN for each possible pair of classes and applying a majority voting scheme to predict the final output. Our approach results in the training of robust sparsified networks whose output is not affected by small perturbations on the input and whose number of active weights is as small as possible. We empirically compare this BeMi approach with the current state of the art in solver-based NN training and with traditional gradient-based training, focusing on BNN learning in few-shot contexts. We compare the benefits and drawbacks of INNs versus BNNs, bringing new light to the distribution of weights over the { − P , … , P } interval. Finally, we compare multiobjective versus single-objective training of INNs, showing that robustness and network simplicity can be acquired simultaneously, thus obtaining better test performances. Although the previous state-of-the-art approaches achieve an average accuracy of 51.1 % on the Modified National Institute of Standards and Technology data set, the BeMi ensemble approach achieves an average accuracy of 68.4% when trained with 10 images per class and 81.8% when trained with 40 images per class while having up to 75.3% NN links removed.

Keywords: binarized neural networks; integer neural networks; mixed-integer linear programming; structured ensemble; few-shot learning; sparsity; multiobjective optimization (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/ijoc.2023.0281 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:orijoc:v:37:y:2025:i:3:p:623-643

Access Statistics for this article

More articles in INFORMS Journal on Computing from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().