Model Compression Algorithm via Reinforcement Learning and Knowledge Distillation

Botao Liu, Bing-Bing Hu (), Ming Zhao, Sheng-Lung Peng () and Jou-Ming Chang
Additional contact information
Botao Liu: School of Computer Science, Yangtze University, Jingzhou 434025, China
Bing-Bing Hu: School of Computer Science, Yangtze University, Jingzhou 434025, China
Ming Zhao: School of Computer Science, Yangtze University, Jingzhou 434025, China
Sheng-Lung Peng: Department of Creative Technologies and Product Design, National Taipei University of Business, Taipei 10051, Taiwan
Jou-Ming Chang: Institute of Information and Decision Sciences, National Taipei University of Business, Taipei 10051, Taiwan

Mathematics, 2023, vol. 11, issue 22, 1-12

Abstract: Traditional model compression techniques are dependent on handcrafted features and require domain experts, with a tradeoff between model size, speed, and accuracy. This study proposes a new approach toward resolving model compression problems. Our approach combines reinforcement-learning-based automated pruning and knowledge distillation to improve the pruning of unimportant network layers and the efficiency of the compression process. We introduce a new state quantity that controls the size of the reward and an attention mechanism that reinforces useful features and attenuates useless features to enhance the effects of other features. The experimental results show that the proposed model is superior to other advanced pruning methods in terms of the computation time and accuracy on CIFAR-100 and ImageNet dataset, where the accuracy is approximately 3% higher than that of similar methods with shorter computation times.

Keywords: model compression; reinforcement learning; knowledge distillation; attention mechanism; automatic pruning; network efficiency (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/22/4589/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/22/4589/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:22:p:4589-:d:1276927

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().