Enhancing analytic rigor in qualitative analysis: developing and testing code scheme using Many Facet Rasch Model

Zuliana Mohd Zabidi, Bambang Sumintono () and Zuraidah Abdullah
Additional contact information
Zuliana Mohd Zabidi: Universiti Malaya
Bambang Sumintono: Universiti Malaya
Zuraidah Abdullah: Universiti Malaya

Quality & Quantity: International Journal of Methodology, 2022, vol. 56, issue 2, No 16, 713-727

Abstract: Abstract Performance assessments that include multiple raters using rating scales to evaluate the quality of a performance are frequently seen and reported. However, the establishment of intercoder reliability in analysing content validity of codes applied to data from transcripts in a qualitative study is not widely practiced. This paper presents intercoder reliability using Many Facet Rasch Model (MFRM) in developing, testing, and enhancing the code scheme of a qualitative study. The results suggest that although the raters were in the same field of expertise and were given the same set of code scheme, their independent characteristics were still observable because unlike the standard approach, MFRM allowed for comparability of each expert’s rating operation as opposed to group homogeneity. While all raters agreed to assign 70% of the codings in the category of very good and good, the researchers were still able to determine the raters’ severity or leniency based on the ratings they assigned. Moreover, MFRM’s ability to provide independent piece of information helped the researchers to identify which codings were accepted and which coding needed review on the researchers’ part. The evaluation has increased trust in the quality of codings and this has created a sense of confidence to the researchers in establishing the code scheme for the qualitative study.

Keywords: Interrater reliability; Qualitative; Confirm coding; Subjectivity; Many Facet Rasch Model (MFRM); Rater severity (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11135-021-01152-4 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:qualqt:v:56:y:2022:i:2:d:10.1007_s11135-021-01152-4

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11135

DOI: 10.1007/s11135-021-01152-4

Access Statistics for this article

Quality & Quantity: International Journal of Methodology is currently edited by Vittorio Capecchi

More articles in Quality & Quantity: International Journal of Methodology from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().