Investigating the Decision-making Style of Translation Raters in Large-scale Language Tests

Xiaodong Li, Jie Yuan and Ying Xu

SAGE Open, 2025, vol. 15, issue 3, 21582440251379569

Abstract: Scoring constructed responses in large-scale language tests is crucial, as it affects the reliability, validity, and fairness of test results. While previous research has examined rater-related factors in explaining scoring variations, most studies focused on raters’ background variables, with limited attention to their cognitive decision-making. Although a few studies have examined raters’ decision-making styles in writing assessments, revealing the importance of raters’ cognitive differences in scoring, little is known about how such styles operate in translation assessments. This study employs a two-phase design to investigate the decision-making characteristics of translation raters. First, to assess the applicability of the General Decision-Making Style Inventory (GDMSI) in the College English Test (CET) translation rating context, responses from 469 CET raters were analyzed using exploratory factor analysis, confirmatory factor analysis, and reliability analysis. Second, a convergent parallel mixed-methods approach was used to construct decision-making profiles for ten experienced raters by integrating their GDMSI responses with qualitative think-aloud data. The results showed that, with minor adjustments, the GDMSI is a valid tool for assessing decision-making styles among CET translation raters, and the ten experienced raters exhibited three primary styles: rational, intuitive, and a hybrid of both rational and intuitive. These findings have practical implications for improving rater training and quality control in translation assessment contexts.

Keywords: decision-making style; raters’ variability; General Decision-Making Style Inventory; think-aloud protocol; translation task (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/21582440251379569 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:sagope:v:15:y:2025:i:3:p:21582440251379569

DOI: 10.1177/21582440251379569

Access Statistics for this article

More articles in SAGE Open
Bibliographic data for series maintained by SAGE Publications ().