Human Ratings and Complexity, Accuracy, and Fluency (CAF) Indices: A Correlational Study of a Standardised Monologic English-Speaking Test in China

Hengzhi Hu, Nur Ehsan Mohd Said and Harwati Hashim

SAGE Open, 2025, vol. 15, issue 2, 21582440251343944

Abstract: Foreign language (L2) learners’ speaking proficiency is often quantified using two dimensions: intuitive human ratings and analytical, linguistic complexity, accuracy, and fluency (CAF) indices. While previous research and assessment practices have predominantly focused on either the subjective approach to L2 speaking or the objective one, it is essential to establish an association between these two seemingly contradictory assessment methods to enhance and promote more credible assessment judgements. To this end, 160 recordings from a monologic task of a standardised English test in China were analysed to quantify CAF in the present study, and the scores were then compared with human ratings given by qualified examiners. Correlation and regression analyses demonstrated that human ratings were positively correlated with speaking fluency, with the number of pauses produced by a candidate being the most significant predictor of human-judged scores. Speaking complexity also positively predicted human ratings, with examiners tending to focus more on grammatical complexity than lexical complexity. In contrast, no correlations were found between human ratings and speaking accuracy. The findings of this study reinforce the possibility of “halo†effects on human raters in L2 assessment and suggest that rater training should focus on helping examiners recognise and mitigate such potential effects.

Keywords: assessment; English speaking; human ratings; CAF; standardised tests (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/21582440251343944 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:sagope:v:15:y:2025:i:2:p:21582440251343944

DOI: 10.1177/21582440251343944

Access Statistics for this article

More articles in SAGE Open
Bibliographic data for series maintained by SAGE Publications ().