Using Generalizability Theory to Examine Scoring Reliability and Variability of Judging Panels in Skating Competitions

Huang Jinyan and Foote Chandra J
Additional contact information
Huang Jinyan: Niagara University
Foote Chandra J: Niagara University

Journal of Quantitative Analysis in Sports, 2011, vol. 7, issue 3, 23

Abstract: Generalizability (G-) theory is increasingly being used by assessment professionals in a variety of assessment contexts, especially those involving performance assessments. This study expands the use of G-theory and its possible application to highly competitive sporting events that involve judging panels. Specifically, using G-theory, this study examined both the scoring variability and reliability of all four events of the 2004 World Figure Skating Championships. For each event, it examined the independent sources of score variation contributing to the variability of scores assigned by the raters (judges), and the score reliability (or generalizability coefficient). For the Ice Dancing event, an additional examination of the impact of treating "rater" and "rating" as different facets of G-theory studies on the score reliability was conducted. The results show that rater/rating was identified as a component of concern in all events. For the Singles events, in particular, the percentages of the variance for rater (judge) reached approximately 10 percent of the total variance, indicating that there were serious concerns about the scoring reliability. The generalizability coefficients for the events of Men's Singles and Ladies' Singles were both 0.93. However, the generalizability coefficients for Pairs and Ice Dancing were 0.99 and 0.98, respectively, indicating that the scores for these two events were extremely reliable. Using rater vs. rating in the Ice Dancing scores appeared to result in very little difference.

Keywords: G-theory; scoring variability and reliability; figure skating (search for similar items in EconPapers)
Date: 2011
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.2202/1559-0410.1241 (text/html)
For access to full text, subscription to the journal or payment for the individual article is required.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bpj:jqsprt:v:7:y:2011:i:3:n:16

Ordering information: This journal article can be ordered from
https://www.degruyter.com/journal/key/jqas/html

DOI: 10.2202/1559-0410.1241

Access Statistics for this article

Journal of Quantitative Analysis in Sports is currently edited by Mark Glickman

More articles in Journal of Quantitative Analysis in Sports from De Gruyter
Bibliographic data for series maintained by Peter Golla ().