Enhancing Machine Translation Quality Estimation via Fine-Grained Error Analysis and Large Language Model

Dahyun Jung, Chanjun Park, Sugyeong Eo and Heuiseok Lim ()
Additional contact information
Dahyun Jung: Department of Computer Science and Engineering, Korea University, Seoul 02841, Republic of Korea
Chanjun Park: Upstage, Yongin 16942, Republic of Korea
Sugyeong Eo: Department of Computer Science and Engineering, Korea University, Seoul 02841, Republic of Korea
Heuiseok Lim: Department of Computer Science and Engineering, Korea University, Seoul 02841, Republic of Korea

Mathematics, 2023, vol. 11, issue 19, 1-9

Abstract: Fine-grained error span detection is a sub-task within quality estimation that aims to identify and assess the spans and severity of errors present in translated sentences. In prior quality estimation, the focus has predominantly been on evaluating translations at the sentence and word levels. However, such an approach fails to recognize the severity of specific segments within translated sentences. To the best of our knowledge, this is the first study that concentrates on enhancing models for this fine-grained error span detection task in machine translation. This study introduces a framework that sequentially performs sentence-level error detection, word-level error span extraction, and severity assessment. We present a detailed analysis for each of the methodologies we propose, substantiating the effectiveness of our system, focusing on two language pairs: English-to-German and Chinese-to-English. Our results suggest that task granularity enhances performance and that a prompt-based fine-tuning approach can offer optimal performance in the classification tasks. Furthermore, we demonstrate that employing a large language model to edit the fine-tuned model’s output constitutes a top strategy for achieving robust quality estimation performance.

Keywords: natural language processing; quality estimation; fine-grained error span detection (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/19/4169/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/19/4169/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:19:p:4169-:d:1253812

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().