EconPapers    
Economics at your fingertips  
 

Comparing distributions by multiple testing across quantiles or CDF values

David Kaplan () and Matt Goldman

No 18-01, Working Papers from Department of Economics, University of Missouri

Abstract: When comparing two distributions, it is often helpful to learn at which quantiles or values there is a statistically significant difference. This provides more information than the binary "reject" or "do not reject" decision of a global goodness-of-fit test. Framing our question as multiple testing across the continuum of quantiles tau in (0,1) or values r, we show that the Kolmogorov–Smirnov test (interpreted as a multiple testing procedure) achieves strong control of the familywise error rate. However, its well-known flaw of low sensitivity in the tails remains. We provide an alternative method that retains such strong control of familywise error rate while also having even sensitivity, i.e., equal pointwise type I error rates at each of n (going to infinity) order statistics across the distribution. Our one-sample method computes instantly, using our new formula that also instantly computes goodness-of-fit p-values and uniform confidence bands. To improve power, we also propose stepdown and pre-test procedures that maintain control of the asymptotic familywise error rate. One-sample and two-sample cases are considered, as well as extensions to regression discontinuity designs and conditional distributions. Simulations, empirical examples, and code are provided. Classification-JEL: C12, C14, C21

Keywords: Dirichlet; familywise error rate; Kolmogorov–Smirnov; probability integral transform; stepdown (search for similar items in EconPapers)
Date: 2013-10-31, Revised 2018-02-22
Note: Title change on 2018-02-22
References: View references in EconPapers View complete reference list from CitEc
Citations: Track citations by RSS feed

Published in Journal of Econometrics, Volume 206, Issue 1, September 2018, Pages 143-166 https://doi.org/10.1016/j.jeconom.2018.04.003

Downloads: (external link)
https://economics.missouri.edu/sites/default/files/wp-files/gk2018_dist_inf.pdf (application/pdf)

Related works:
Journal Article: Comparing distributions by multiple testing across quantiles or CDF values (2018) Downloads
Working Paper: Comparing distributions by multiple testing across quantiles or CDF values (2018) Downloads
Working Paper: Comparing distributions by multiple testing across quantiles or CDF values (2017) Downloads
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:umc:wpaper:1801

Access Statistics for this paper

More papers in Working Papers from Department of Economics, University of Missouri Contact information at EDIRC.
Bibliographic data for series maintained by Valerie Kulp ().

 
Page updated 2019-04-19
Handle: RePEc:umc:wpaper:1801