Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models

Navya Veginati

International Journal of Scientific Research in Science and Technology, 2024, vol. 11, issue 5, 864-871

Abstract: Large language models based on transformers have shown outstanding performance in several natural language processing tasks. But, when such models are fine-tuned, catastrophic forgetting is likely to occur causing one to forget the knowledge that has been learned before. This paper suggests a better attention-based model to enhance the knowledge retention in fine-tuned transformer models. The method combines attention mechanisms that are aware of retention and knowledge distillation methods to retain important contextual information. The proposed model successfully balances knowledge preservation and new learning by proposing adaptive attention scaling and feature alignment strategies. Experimental evidence indicates that the models have highly improved accuracy, precision, recall, F1-score, and knowledge retention over existing models, including BERT, DistilBERT, and TinyBERT. The results indicate that the use of attention enhancement and learning through retention awareness is effective, offering an effective solution to knowledge decline in large language models.

Keywords: Transformer Models; Knowledge Retention; Attention Mechanism; Knowledge Distillation; Fine-Tuning (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrst.com/home/article/view/IJSRST52310284 Abstract page (text/html)
https://ijsrst.com/home/article/download/IJSRST52310284/IJSRST52310284 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:etm:ijsrst:v11:y2024:i5:id:1524

DOI: 10.32628/IJSRST52310284

Access Statistics for this article

More articles in International Journal of Scientific Research in Science and Technology from Technoscience Academy
Bibliographic data for series maintained by Pankaj Sharma ().