An empirical comparison of permutation methods for tests of partial regression coefficients in a linear model

This study compared empirical type I error and power of different permutation techniques for the test of significance of a single partial regression coefficient in a multiple regression model, using simulations. The methods compared were permutation of raw data values, two alternative methods proposed for permutation of residuals under the reduced model, and permutation of residuals under the full model. The normal-theory t-test was also included in simulations. We investigated effects of (1) the sample size, (2) the degree of collinearity between the predictor variables, (3) the size of the covariable's parameter, (4) the distribution of the added random error and (5) the presence of an outlier in the covariable on these methods. We found that two methods that had been identified as equivalent formulations of permutation under the reduced model were actually quite different. One of these methods resulted in consistently inflated type 1 error. In addition, when the covariable contained an extreme outlier, permutation of raw data resulted in unstable (often inflated) type 1 error. There were no significant differences in power among the three permutation methods (raw data permutation, reduced-model permutation and full-model permutation), but all had greater power than the normal-theory t-test when errors were non-normal. The reduced model permutation method had the most consistent and reliable results of the methods investigated here for the test of a partial regression coefficient. However, reasonably extreme situations needed to be simulated in order to distinguish methods from the normal-theory t-test and from one another. Permutation of raw data, permutation under the reduced model, and permutation under the full model are generally asymptotically equivalent.