Rater variability and reliability of constructed response questions in New York state high-stakes tests of English language arts and mathematics: implications for educational assessment policy

Jinyan Huang () and Patrick B. Whipple
Additional contact information
Jinyan Huang: Jiangsu University
Patrick B. Whipple: The Genesee Valley Board of Cooperative Educational Services

Palgrave Communications, 2023, vol. 10, issue 1, 1-10

Abstract: Abstract Using generalizability (G-) theory as a theoretical framework and research methodology, this study examined the impact of the current one-rater holistic scoring practice on the rater variability and reliability of constructed response questions in New York State high-stakes tests of grades four and six English language arts (ELA) and grades four and five mathematics assessments. Following the New York State scoring rubrics, a total of 36 grades four and six ELA constructed response samples and 72 grades four and five mathematics constructed response samples were marked holistically by ten independent raters having current certifications as educators in the State of New York. The results indicated that the current single-rater holistic scoring practice would not be able to yield acceptable G-coefficients for the New York State grades four and six ELA and grades four and five mathematics assessments. Implications for assessment policy making at the local and state levels are discussed.

Date: 2023
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://link.springer.com/10.1057/s41599-023-02385-4 Abstract (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:pal:palcom:v:10:y:2023:i:1:d:10.1057_s41599-023-02385-4

Ordering information: This journal article can be ordered from
https://www.nature.com/palcomms/about

DOI: 10.1057/s41599-023-02385-4

Access Statistics for this article

More articles in Palgrave Communications from Palgrave Macmillan
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().