Reduced Forgetfulness in Continual Learning for Named Entity Recognition Through Confident Soft-Label Imitation

Huan Zhang, Long Zhou () and Miaomiao Gu
Additional contact information
Huan Zhang: School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China
Long Zhou: School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China
Miaomiao Gu: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China

Mathematics, 2024, vol. 12, issue 24, 1-22

Abstract: Continual Learning for Named Entity Recognition (CL-NER) is a crucial task in recognizing emerging concepts when constructing real-world natural language processing applications. It involves sequentially updating an existing NER model with new entity types while retaining previously learned information. However, current CL methods are struggling with a major challenge called catastrophic forgetting. Owing to the semantic shift of the non-entity type, the issue is further intensified in NER. Most existing CL-NER methods rely on knowledge distillation through the output probabilities of previously learned entities, resulting in excessive stability (recognition of old entities) at the expense of plasticity (recognition of new entities). Some recent works further extend these methods by improving the distinction between old entities and non-entity types. Although these methods result in overall performance improvements, the preserved knowledge does not necessarily ensure the retention of task-related information for the oldest entities, which can lead to significant performance drops. To address this issue while maintaining overall performance, we propose a method called Confident Soft-Label Imitation (ConSOLI) for continual learning in NER. Inspired by methods that balance stability and plasticity, ConSOLI incorporates a soft-label distillation process and confident soft-label imitation learning. The former helps to gather the task-related knowledge in the old model and the latter further preserves the knowledge from diluting in the step-wise continual learning process. Moreover, ConSOLI demonstrates significant improvements in recognizing the oldest entity types, achieving Micro-F1 and Macro-F1 scores of up to 8.72 and 9.72, respectively, thus addressing the challenge of catastrophic forgetting in CL-NER.

Keywords: named entity recognition; continual learning; knowledge distillation; soft-label (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/24/3964/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/24/3964/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:24:p:3964-:d:1545736

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().