EconPapers    
Economics at your fingertips  
 

Orthogonal position representations for transformer in neural machine translation

Yue Zhao, Qinghong Zhang and Shuhan Zhou

PLOS ONE, 2025, vol. 20, issue 10, 1-21

Abstract: In recent years, the Transformer architecture has solidified its position as the dominant model in neural machine translation (NMT), thanks to its exceptional effectiveness in capturing long-range dependencies and remarkable scalability across diverse linguistic tasks. A key characteristic of the Transformer is its reliance on self-attention mechanisms, which, while powerful, are inherently position-insensitive—treating tokens as a set rather than an ordered sequence. This limitation makes positional encoding a critical component in Transformer-based models and their variants, as it provides the necessary sequential context to differentiate token positions within a sequence. In this paper, we address this challenge by proposing a novel orthogonal fixed-dimension positional representation (OPR). This design is meticulously engineered to maximize the discrimination of positions within a sequence, ensuring that each position is uniquely and distinctively encoded. Notably, OPR introduces no additional parameters to the model and incurs no extra computational overhead, making it highly efficient for real-world applications. Our experimental evaluations, conducted across multiple standard NMT datasets, demonstrate that OPR consistently outperforms several strong baselines, including traditional sine-cosine positional encoding and learnable positional embeddings. It achieves notable improvements in both BLEU and COMET scores, with gains observed across all tested language pairs. Furthermore, when combined with relative positional encoding (RPR), the OPR method’s performance is further enhanced, highlighting its ability to effectively model both absolute and relative positional relationships—a dual capability that is crucial for nuanced sequence understanding.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0334443 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 34443&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0334443

DOI: 10.1371/journal.pone.0334443

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().

 
Page updated 2025-10-18
Handle: RePEc:plo:pone00:0334443