Assessment of observer agreement for matched repeated binary measurements

Jingjing Gao, Yi Pan and Michael Haber

Computational Statistics & Data Analysis, 2012, vol. 56, issue 5, 1052-1060

Abstract: Agreement is a broad term simultaneously covering evaluations of accuracy and precision of measurements. Assessment of observer agreement is based on the similarity between readings made on the same subject by different observers. The assessment of agreement on categorical observations is traditionally based on kappa or weighted kappa coefficients. However, kappa statistics have been criticized because they attain implausible values when the marginal distributions are skewed and/or unbalanced. New scaled indices called the coefficients of individual agreement (CIAs) have been developed for the assessment of individual observer agreement by comparing the observed disagreement between two observers to the disagreement between replicated observations made by the same observer on the same subject. This is based on the notion that under a good agreement, the disagreement between the two observers is usually not expected to exceed the disagreement between replicated observations of the same observer, and hence, a satisfactory agreement is established if these quantities are similar. This idea is extended and a new method utilizing the generalized linear mixed model is proposed to estimate the CIAs for binary data which consist of matched sets of repeated measurements made by the same observer under different conditions. The conditions may represent different time points, raters, laboratories, treatments, etc. The new approach allows the values of the measured variable and the magnitude of agreement to vary across the conditions. The reliability of the estimation method is examined via a simulation study. Data from a study aiming at determining the validity of diagnosis of breast cancer based on mammography are used to illustrate the new concepts and methods.

Keywords: Coefficient of individual agreement; Observer agreement; Binary data; Repeated measurements; Generalized linear mixed model (search for similar items in EconPapers)
Date: 2012
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0167947311004026
Full text for ScienceDirect subscribers only.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:csdana:v:56:y:2012:i:5:p:1052-1060

DOI: 10.1016/j.csda.2011.11.005

Access Statistics for this article

Computational Statistics & Data Analysis is currently edited by S.P. Azen

More articles in Computational Statistics & Data Analysis from Elsevier
Bibliographic data for series maintained by Catherine Liu ().