Accelerating and Compressing Transformer-Based PLMs for Enhanced Comprehension of Computer Terminology

Jian Peng and Kai Zhong ()
Additional contact information
Jian Peng: College of Foreign Languages, Changsha Normal University, Changsha 410000, China
Kai Zhong: College of Computer Science and Electronic Engineering, Hunan University, Changsha 410000, China

Future Internet, 2024, vol. 16, issue 11, 1-22

Abstract: Pretrained language models (PLMs) have significantly advanced natural language processing (NLP), establishing the "pretraining + fine-tuning" paradigm as a cornerstone approach in the field. However, the vast size and computational demands of transformer-based PLMs present challenges, particularly regarding storage efficiency and processing speed. This paper addresses these limitations by proposing a novel lightweight PLM optimized for accurately understanding domain-specific computer terminology. Our method involves a pipeline parallelism algorithm designed to accelerate training. It is paired with an innovative mixed compression strategy that combines pruning and knowledge distillation to effectively reduce the model size while preserving its performance. The model is further fine-tuned using a dataset that mixes source and target languages to enhance its versatility. Comprehensive experimental evaluations demonstrate that the proposed approach successfully achieves a balance between model efficiency and performance, offering a scalable solution for NLP tasks involving specialized terminology.

Keywords: pretraining models; parallelism; mixed compression method; computer-specialized terms (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/16/11/385/pdf (application/pdf)
https://www.mdpi.com/1999-5903/16/11/385/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:16:y:2024:i:11:p:385-:d:1504025

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().