The Influence of Translator Backgrounds and Machine Translation on Statistical Properties of Surveys: Evidence from a Survey Experiment

Chia-Jung Tsai, Clemens Lechner, Dorothée Behr, Ulrike Efu Nkong and Anke Radinger

No rw6f2_v1, SocArXiv from Center for Open Science

Abstract: Comparable questionnaire translation is essential for drawing valid conclusions in cross-cultural survey research. Sound translation methodology, including the use of adequate personnel, is seen as crucial for reaching this goal (Harkness 2003). Recommended methodology should be empirically backed and stay tuned to latest developments, such as machine translation. Against this backdrop, to investigate the potential effect of varied translators’ backgrounds and machine translation on the statistical properties of surveys, we conducted an experiment in which an English questionnaire was translated into German by 16 professional translators and 16 social scientists; translations were subsequently fielded in web surveys. We introduced two translation conditions: translation from scratch and post-editing (machine translation corrected by a human translator). To investigate the quality of the survey data from these 32 translation versions (approx. 250 responses each), we use standardized mean distance and Cohen’s d with the official translation as a benchmark. We have four key findings: First, the resulting statistical means of the survey items vary, sometimes substantially, across translations. Second, post-editing is associated with a reduced gap between the survey data from the experimental questionnaires and the official translation, and it also lowers the variability among different translations. Third, when translating from scratch, social scientists are more likely to produce translations leading to statistical outliers of survey data. Fourth, post-editing can lead to systematic bias for both social scientists and professional translators if translation errors made by the machine are not identified and corrected. This study highlights to what extent decisions concerning the choice of translators and the integration of machine translation can impact the statistical properties of survey data. We offer evidence to implement recommendations for good practices in translation protocols to enhance data comparability in cross-cultural studies.

Date: 2026-01-23
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/69724f6bb010b2f98936bd36/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:rw6f2_v1

DOI: 10.31219/osf.io/rw6f2_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().