Assessing Quality in Systematic Literature Reviews: A Study of Novice Rater Training

Sandra Acosta, Tiberio Garza, Hsien-Yuan Hsu and Patricia Goodson

SAGE Open, 2020, vol. 10, issue 3, 2158244020939530

Abstract: This study investigated performance variability when graduate students critically appraised original studies from a systematic review. Fourteen doctoral students from different academic programs, with no systematic review experience, received training on the Methodological Quality Questionnaire (MQQ) rating scale. Participants were mostly male (71%) and non-native English speakers (79%). Each rater was randomly assigned one original study to independently assess using the MQQ. Their scores were compared to an expert rater. Statistical analysis comprised the following: percentage of agreement (POA), Kappa coefficient, and Kendall’s tau-b correlation. On the completed MQQ rating scale, 43% of the novice raters had a POA of 78% or higher with the expert rater. From this case study, a guide for improving training on methodological quality assessment was developed. Benefits include the following: (a) developing and supporting critical reasoning as well as problem-solving skills and (b) increasing research skills and competencies in the systematic review process.

Keywords: Methodological Quality Questionnaire; training; systematic review; rater bias; higher education (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/2158244020939530 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:sagope:v:10:y:2020:i:3:p:2158244020939530

DOI: 10.1177/2158244020939530

Access Statistics for this article

More articles in SAGE Open
Bibliographic data for series maintained by SAGE Publications ().