DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy

Xin Cheng, Zhiqiang Zhang, Wei Weng, Wenxin Yu and Jinjia Zhou ()
Additional contact information
Xin Cheng: Graduate School of Science and Engineering, Hosei University, Tokyo 184-8584, Japan
Zhiqiang Zhang: School of Science and Technology, Southwest University of Science and Technology, Mianyang 621010, China
Wei Weng: Institute of Liberal Arts and Science, Kanazawa University, Kanazawa City 920-1192, Japan
Wenxin Yu: School of Science and Technology, Southwest University of Science and Technology, Mianyang 621010, China
Jinjia Zhou: Graduate School of Science and Engineering, Hosei University, Tokyo 184-8584, Japan

Mathematics, 2024, vol. 12, issue 11, 1-10

Abstract: The complexity of deep neural network models (DNNs) severely limits their application on devices with limited computing and storage resources. Knowledge distillation (KD) is an attractive model compression technology that can effectively alleviate this problem. Multi-teacher knowledge distillation (MKD) aims to leverage the valuable and diverse knowledge distilled by multiple teacher networks to improve the performance of the student network. Existing approaches typically rely on simple methods such as averaging the prediction logits or using sub-optimal weighting strategies to fuse distilled knowledge from multiple teachers. However, employing these techniques cannot fully reflect the importance of teachers and may even mislead student’s learning. To address this issue, we propose a novel Decoupled Multi-Teacher Knowledge Distillation based on Entropy (DE-MKD). DE-MKD decouples the vanilla knowledge distillation loss and assigns adaptive weights to each teacher to reflect its importance based on the entropy of their predictions. Furthermore, we extend the proposed approach to distill the intermediate features from multiple powerful but cumbersome teachers to improve the performance of the lightweight student network. Extensive experiments on the publicly available CIFAR-100 image classification benchmark dataset with various teacher-student network pairs demonstrated the effectiveness and flexibility of our approach. For instance, the VGG8|ShuffleNetV2 model trained by DE-MKD reached 75.25%|78.86% top-one accuracy when choosing VGG13|WRN40-2 as the teacher, setting new performance records. In addition, surprisingly, the distilled student model outperformed the teacher in both teacher-student network pairs.

Keywords: multi-teacher knowledge distillation; image classification; entropy; deep learning (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/11/1672/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/11/1672/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:11:p:1672-:d:1403167

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().