Probabilistic Automated Model Compression via Representation Mutual Information Optimization

Wenjie Nie, Shengchuan Zhang and Xiawu Zheng ()
Additional contact information
Wenjie Nie: Department of Artificial lntelligence, Xiamen University, Xiamen 361101, China
Shengchuan Zhang: Department of Artificial lntelligence, Xiamen University, Xiamen 361101, China
Xiawu Zheng: Department of Artificial lntelligence, Xiamen University, Xiamen 361101, China

Mathematics, 2024, vol. 13, issue 1, 1-18

Abstract: Deep neural networks, despite their remarkable success in computer vision tasks, often face deployment challenges due to high computational demands and memory usage. Addressing this, we introduce a probabilistic framework for automated model compression (Prob-AMC) that optimizes pruning, quantization, and knowledge distillation simultaneously using information theory. Our approach is grounded in maximizing the mutual information between the original and compressed network representations, ensuring the preservation of essential features under resource constraints. Specifically, we employ layer-wise self-representation mutual information analysis, sampling-based pruning and quantization allocation, and progressive knowledge distillation using the optimal compressed model as a teacher assistant. Through extensive experiments on CIFAR-10 and ImageNet, we demonstrate that Prob-AMC achieves a superior compression ratio of 33.41× on ResNet-18 with only a 1.01% performance degradation, outperforming state-of-the-art methods in terms of both compression efficiency and accuracy. This optimization process is highly practical, requiring merely a few GPU hours, and bridges the gap between theoretical information measures and practical model compression, offering significant insights for efficient deep learning deployment.

Keywords: probabilistic model compression; representation mutual information; neural network compression; automated compression pipeline; information theory (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/1/108/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/1/108/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2024:i:1:p:108-:d:1556419

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().