Spectral Salt-and-Pepper Patch Masking for Self-Supervised Speech Representation Learning

June-Woo Kim, Hoon Chung and Ho-Young Jung ()
Additional contact information
June-Woo Kim: Department of Artificial Intelligence, Kyungpook National University, Daegu 41566, Republic of Korea
Hoon Chung: Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea
Ho-Young Jung: Department of Artificial Intelligence, Kyungpook National University, Daegu 41566, Republic of Korea

Mathematics, 2023, vol. 11, issue 15, 1-22

Abstract: Recent advanced systems in the speech recognition domain use large Transformer neural networks that have been pretrained on massive speech data. General methods in the deep learning area have been frequently shared across various domains, and the Transformer model can also be used effectively across speech and image. In this paper, we introduce a novel masking method for self-supervised speech representation learning with salt-and-pepper (S&P) mask which is commonly used in computer vision. The proposed scheme includes consecutive quadrilateral-shaped S&P patches randomly contaminating the input speech spectrum. Furthermore, we modify the standard S&P mask to make it appropriate for the speech domain. In order to validate the effect of the proposed spectral S&P patch masking for the self-supervised representation learning approach, we conduct the pretraining and downstream experiments with two languages, English and Korean. To this end, we pretrain the speech representation model using each dataset and evaluate the pretrained models for feature extraction and fine-tuning performance on varying downstream tasks, respectively. The experimental outcomes clearly illustrate that the proposed spectral S&P patch masking is effective for various downstream tasks when combined with the conventional masking methods.

Keywords: self-supervised learning; speech representation learning; salt-and-pepper masking; spectrum patch masking (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/15/3418/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/15/3418/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:15:p:3418-:d:1211344

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().