Mitigating Catastrophic Forgetting in Fine-Tuned Large Language Models: An Experimental Study of LoRA and O-LoRA

Xinlan Zhang

Artificial Intelligence and Digital Technology, 2026, vol. 3, issue 1, 52-61

Abstract: Large language models (LLMs) have become a hot topic in AI, and since the GPT series they have achieved remarkable success across many domains. However, directly using a general-purpose model often fails to meet the needs of specific applications, which motivates fine-tuning with domain-specific data. Nevertheless, parameter-efficient fine-tuning (PEFT) methods such as LoRA may perform poorly on certain algorithmic benchmarks, raising concerns about cat-astrophic forgetting. In this paper, we conduct extensive experiments to confirm this phenomenon and investigate O-LoRA as a mitigation strategy. Results show that O-LoRA can effectively alleviate catastrophic forgetting under continual instruction fine-tuning, but its effectiveness can be sensitive to hyperparameters on some datasets. Overall, O-LoRA provides a practical direction for mitigating catastrophic forgetting during continual fine-tuning of LLMs.

Keywords: large language models; catastrophic forgetting; parameter-efficient fine-tuning; LoRA; O-LoRA (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://soapubs.com/index.php/AIDT/article/view/1380/1258 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:axf:aidtaa:v:3:y:2026:i:1:p:52-61

Access Statistics for this article

More articles in Artificial Intelligence and Digital Technology from Scientific Open Access Publishing
Bibliographic data for series maintained by Yuchi Liu ().