Testing the Relative Performance of Data Adaptive Prediction Algorithms: A Generalized Test of Conditional Risk Differences

Goldstein Benjamin A. (), Polley Eric C., Briggs Farren B. S., J. van der Laan Mark and Hubbard Alan
Additional contact information
Goldstein Benjamin A.: Department of Biostatistics & Bioinformatics, Duke University, Durham, NC, USA
Polley Eric C.: National Institute of Health, National Cancer Institute – Biometric Research Branch, 6130 Executive Blvd EPN RM 8146, Rockville, MD 20892, USA
Briggs Farren B. S.: Department of Epidemiology and Biostatistics, Case Western Reserve University, Cleveland, OH, USA
J. van der Laan Mark: Division of Biostatistics, UC Berkeley, School of Public Health, Berkeley, CA 94720, USA
Hubbard Alan: Division of Biostatistics, UC Berkeley, School of Public Health, Berkeley, CA 94720, USA

The International Journal of Biostatistics, 2016, vol. 12, issue 1, 117-129

Abstract: Comparing the relative fit of competing models can be used to address many different scientific questions. In classical statistics one can, if appropriate, use likelihood ratio tests and information based criterion, whereas clinical medicine has tended to rely on comparisons of fit metrics like C-statistics. However, for many data adaptive modelling procedures such approaches are not suitable. In these cases, statisticians have used cross-validation, which can make inference challenging. In this paper we propose a general approach that focuses on the “conditional” risk difference (conditional on the model fits being fixed) for the improvement in prediction risk. Specifically, we derive a Wald-type test statistic and associated confidence intervals for cross-validated test sets utilizing the independent validation within cross-validation in conjunction with a test for multiple comparisons. We show that this test maintains proper Type I Error under the null fit, and can be used as a general test of relative fit for any semi-parametric model alternative. We apply the test to a candidate gene study to test for the association of a set of genes in a genetic pathway.

Keywords: risk prediction; cross-validation; semi-parametric models; machine learning (search for similar items in EconPapers)
Date: 2016
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1515/ijb-2015-0014 (text/html)
For access to full text, subscription to the journal or payment for the individual article is required.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bpj:ijbist:v:12:y:2016:i:1:p:117-129:n:6

Ordering information: This journal article can be ordered from
https://www.degruyter.com/journal/key/ijb/html

DOI: 10.1515/ijb-2015-0014

Access Statistics for this article

The International Journal of Biostatistics is currently edited by Antoine Chambaz, Alan E. Hubbard and Mark J. van der Laan

More articles in The International Journal of Biostatistics from De Gruyter
Bibliographic data for series maintained by Peter Golla ().