Permutation mask: a combined gradient sparsification for federated learning

Shiqi Zhou and Yongdao Zhou

Journal of Nonparametric Statistics, 2025, vol. 37, issue 4, 967-989

Abstract: Large-scale distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. This situation gets even worse with training on mobile devices, which suffers from high latency, lower throughput and intermittent poor connections. Various recent works proposed to use quantisation or sparsification techniques to reduce the amount of data that need to be communicated, for instance Top-k sparsification. However, the Top-k sparsification is lack of the cooperation between nodes. In this paper, we present the permutation mask (Pmask), a novel gradient sparsification technique for distributed model training. To unite nodes and improve the convergence rate, Pmask employs non-zero average and combined gradients. The corresponding convergence property is also given. We have applied permutation mask to image classification task on Cifar10, mini-ImageNet and SVHN, and the experiments show that Pmask leads to a faster distributed training and improves the accuracy.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/10485252.2024.2370874 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:gnstxx:v:37:y:2025:i:4:p:967-989

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/GNST20

DOI: 10.1080/10485252.2024.2370874

Access Statistics for this article

Journal of Nonparametric Statistics is currently edited by Jun Shao

More articles in Journal of Nonparametric Statistics from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().