Bimodal-Distributed Binarized Neural Networks

Tal Rozen (), Moshe Kimhi, Brian Chmiel, Avi Mendelson and Chaim Baskin
Additional contact information
Tal Rozen: Technion—Israel Institute of Technology, Haifa 3200003, Israel
Moshe Kimhi: Technion—Israel Institute of Technology, Haifa 3200003, Israel
Brian Chmiel: Technion—Israel Institute of Technology, Haifa 3200003, Israel
Avi Mendelson: Technion—Israel Institute of Technology, Haifa 3200003, Israel
Chaim Baskin: Technion—Israel Institute of Technology, Haifa 3200003, Israel

Mathematics, 2022, vol. 10, issue 21, 1-14

Abstract: Binary neural networks (BNNs) are an extremely promising method for reducing deep neural networks’ complexity and power consumption significantly. Binarization techniques, however, suffer from ineligible performance degradation compared to their full-precision counterparts. Prior work mainly focused on strategies for sign function approximation during the forward and backward phases to reduce the quantization error during the binarization process. In this work, we propose a bimodal-distributed binarization method (BD-BNN). The newly proposed technique aims to impose a bimodal distribution of the network weights by kurtosis regularization. The proposed method consists of a teacher–trainer training scheme termed weight distribution mimicking (WDM), which efficiently imitates the full-precision network weight distribution to their binary counterpart. Preserving this distribution during binarization-aware training creates robust and informative binary feature maps and thus it can significantly reduce the generalization error of the BNN. Extensive evaluations on CIFAR-10 and ImageNet demonstrate that our newly proposed BD-BNN outperforms current state-of-the-art schemes.

Keywords: convolutional neural networks; binarization; quantization; efficient inference deployment (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/21/4107/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/21/4107/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:21:p:4107-:d:962829

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().