Asymptotic distribution of P values in composite null models

We investigate the compatibility of a null model H-0 with the data by calculating a p value; that is, the probability, under H-0, that a given rest statistic T exceeds its observed value. When the null model consists of a single distribution, the p value is readily obtained, and it has a uniform distribution under H-0. On the other hand, when the null model depends on an unknown nuisance parameter theta, one must somehow Set rid of theta, (e.g., by estimating it) to calculate a;o value. Various proposals have been suggested to "remove" theta, each yielding a different candidate p value. But unlike the simple case, these p values typically are not uniformly distributed under the null model. In this article we investigate their asymptotic distribution under H-0. We show that when the asymptotic mean of the test statistic T depends on theta, the posterior predictive p value of Guttman and Rubin, and the plug-in p value are conservative (i.e., their asymptotic distributions are more concentrated around 1/2 than a uniform), with the posterior predictive p value being the more conservative. In contrast. the partial posterior predictive and conditional predictive p values of Bayarri and Berger are asymptotically uniform. Furthermore, we show that the discrepancy p value of Meng and Gelman and colleagues can be conservative, even when the discrepancy measure has mean 0 under the null model. We also describe ways to modify the conservative p values to make their distributions asymptotically uniform.