ALLOCATION OF EFFORT IN MONTE-CARLO SIMULATION FOR POWER OF PERMUTATION TESTS

Consider a randomization test in which an observed statistic is compared to a null distribution constructed by first drawing a random sample from the set of all possible permutations of the data and then calculating the statistic anew for each member of the random sample. If the power of such a test is investigated by simulation, the computer code must have two main loops. In the outer loop, which has O iterations, data are generated under a specific alternative hypothesis. For each outer iteration, there are I inner iterations in each of which the data are permuted and the statistic calculated. Assume that the costs of inner and outer iterations are known, and the total cost of the study is fixed, so that I and O are constrained by alpha-O + IO = beta for fixed (alpha, beta). This article considers what the optimal choice for I is in a mean squared sense. A general expression is derived for the mean squared error of the distribution of p values. If the alternative hypothesis is close to the null, the optimal choice is the unique positive root of a fourth-degree polynomial. If, in addition, alpha = O (implying that the cost of generating a replicate is negligible compared to the cost of permuting the data and calculating a statistic) the optimal choice is I(opt) = (4-beta)1/3.