A Federated Fine-Tuning Framework for Large Language Models via Graph Representation Learning and Structural Segmentation

Yuxin Dong, Ruotong Wang, Guiran Liu, Binrong Zhu, Xiaohan Cheng, Zijun Gao and Pengbin Feng ()
Additional contact information
Yuxin Dong: School of Business, Wake Forest University, Winston-Salem, NC 27109, USA
Ruotong Wang: Department of Computer Science, Rutgers University, Piscataway, NJ 08901, USA
Guiran Liu: College of Science & Engineering (CoSE), San Francisco State University, San Francisco, CA 94132, USA
Binrong Zhu: College of Science & Engineering (CoSE), San Francisco State University, San Francisco, CA 94132, USA
Xiaohan Cheng: D’Amore-McKim School of Business, Northeastern University, Boston, MA 02115, USA
Zijun Gao: Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115, USA
Pengbin Feng: Department of Mathematics, University of Southern California, Los Angeles, CA 90007, USA

Mathematics, 2025, vol. 13, issue 19, 1-29

Abstract: This paper focuses on the efficient fine-tuning of large language models within the federated learning framework. To address the performance bottlenecks caused by multi-source heterogeneity and structural inconsistency, a structure-aware federated fine-tuning method is proposed. The method incorporates a graph representation module (GRM) to model internal structural relationships within text and employs a segmentation mechanism (SM) to reconstruct and align semantic structures across inputs, thereby enhancing structural robustness and generalization under non-IID (non-Independent and Identically Distributed) settings. During training, the method ensures data locality and integrates structural pruning with gradient encryption (SPGE) strategies to balance privacy preservation and communication efficiency. Compared with representative federated fine-tuning baselines such as FedNLP and FedPrompt, the proposed method achieves consistent accuracy and F1-score improvements across multiple tasks. To evaluate the effectiveness of the proposed method, extensive comparative experiments are conducted across tasks of text classification, named entity recognition, and question answering, using multiple datasets with diverse structures and heterogeneity levels. Experimental results show that the proposed approach significantly outperforms existing federated fine-tuning strategies on most tasks, achieving higher performance while preserving privacy, and demonstrating strong practical applicability and generalization potential.

Keywords: federated learning; large language models; privacy preservation; graph representation learning; structure-aware fine-tuning (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/19/3201/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/19/3201/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:19:p:3201-:d:1765591

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().