Structure-Aware Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

Yahao Hu, Yifei Xie, Tianfeng Wang, Man Chen and Zhisong Pan ()
Additional contact information
Yahao Hu: Command and Control Engineering College, Army Engineering University of PLA, Nanjing 210007, China
Yifei Xie: Command and Control Engineering College, Army Engineering University of PLA, Nanjing 210007, China
Tianfeng Wang: Command and Control Engineering College, Army Engineering University of PLA, Nanjing 210007, China
Man Chen: Command and Control Engineering College, Army Engineering University of PLA, Nanjing 210007, China
Zhisong Pan: Command and Control Engineering College, Army Engineering University of PLA, Nanjing 210007, China

Mathematics, 2023, vol. 11, issue 20, 1-16

Abstract: With the growing scale of pre-trained language models (PLMs), full parameter fine-tuning becomes prohibitively expensive and practically infeasible. Therefore, parameter-efficient adaptation techniques for PLMs have been proposed to learn through incremental updates of pre-trained weights, such as in low-rank adaptation (LoRA). However, LoRA relies on heuristics to select the modules and layers to which it is applied, and assigns them the same rank. As a consequence, any fine-tuning that ignores the structural information between modules and layers is suboptimal. In this work, we propose structure-aware low-rank adaptation (SaLoRA), which adaptively learns the intrinsic rank of each incremental matrix by removing rank-0 components during training. We conduct comprehensive experiments using pre-trained models of different scales in both task-oriented (GLUE) and task-agnostic (Yelp and GYAFC) settings. The experimental results show that SaLoRA effectively captures the structure-aware intrinsic rank. Moreover, our method consistently outperforms LoRA without significantly compromising training efficiency.

Keywords: pre-trained language models; parameter-efficient fine-tuning; low-rank adaptation; intrinsic rank; training efficiency (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/20/4317/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/20/4317/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:20:p:4317-:d:1261404

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().