Comparing distributions by multiple testing across quantiles

Working Paper Number: 
WP 16-19

When comparing two distributions, it is often helpful to learn at which quantiles there is a statistically significant difference. This provides more information than the binary "reject" or "do not reject" decision of a global goodness-of-fit test. Framing our question as multiple testing across the continuum of quantiles, the Kolmogorov-Smirnov test (with appropriately modified interpretation) achieves strong control of the familywise error rate. However, its well-known flaw of low sensitivity in the tails remains. We provide an alternative method that retains such strong control of familywise error rate while also having even sensitivity, i.e., equal pointwise type I error rates at n (approaching infinity) quantiles across the distribution. Our method computes instantly, using our new formula that also instantly computes goodness-of-fit p-values and uniform confidence bands. To improve power, we also propose stepdown and pre-test procedures that maintain asymptotic familywise error rate control. One-sample (i.e., one known distribution, one unknown) and two-sample (i.e., two unknown distributions) cases are considered. Simulations, an empirical example, and code are provided. 

JEL Codes: 
C12, C14, C21

Matt Goldman