Reply: Birnbaum’s (2012) statistical tests of independence have unknown Type-I error rates and do not replicate within participant

Yun-shil Cha, Michelle Choi, Ying Guo, Michel Regenwetter and Chris Zwilling

Judgment and Decision Making, 2013, vol. 8, issue 1, 55-73

Abstract: Birnbaum (2011, 2012) questioned the iid (independent and identically distributed) sampling assumptions used by state-of-the-art statistical tests in Regenwetter, Dana and Davis-Stober’s (2010, 2011) analysis of the “linear order model”. Birnbaum (2012) cited, but did not use, a test of iid by Smith and Batchelder (2008) with analytically known properties. Instead, he created two new test statistics with unknown sampling distributions. Our rebuttal has five components: 1) We demonstrate that the Regenwetter et al. data pass Smith and Batchelder’s test of iid with flying colors. 2) We provide evidence from Monte Carlo simulations that Birnbaum’s (2012) proposed tests have unknown Type-I error rates, which depend on the actual choice probabilities and on how data are coded as well as on the null hypothesis of iid sampling. 3) Birnbaum analyzed only a third of Regenwetter et al.’s data. We show that his two new tests fail to replicate on the other two-thirds of the data, within participants. 4) Birnbaum selectively picked data of one respondent to suggest that choice probabilities may have changed partway into the experiment. Such nonstationarity could potentially cause a seemingly good fit to be a Type-II error. We show that the linear order model fits equally well if we allow for warm-up effects. 5) Using hypothetical data, Birnbaum (2012) claimed to show that “true-and-error” models for binary pattern probabilities overcome the alleged short-comings of Regenwetter et al.’s approach. We disprove this claim on the same data.

Date: 2013
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.cambridge.org/core/product/identifier/ ... type/journal_article link to article abstract page (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cup:judgdm:v:8:y:2013:i:1:p:55-73_8

Access Statistics for this article

More articles in Judgment and Decision Making from Cambridge University Press Cambridge University Press, UPH, Shaftesbury Road, Cambridge CB2 8BS UK.
Bibliographic data for series maintained by Kirk Stebbing ().