The p-value interval as an inferential tool

In the phase III randomized clinical trial setting, design-based analyses for between-group comparisons are permutation tests which strictly preserve the type I error rate. However, the conservatism of permutation tests can cause a loss of power sufficient to prevent statistical significance from being reached. Arguments regarding the use of permutation tests thus tend to be broad brush: permutation tests should be used routinely for strict preservation of the type I error, or permutation tests should not be used at all because of the loss of power due to conservatism. Lost in these arguments is the fact that the conservatism of any particular permutation test can be measured, to allow for a more moderate decision rule: use a permutation test, but only if it is not overly conservative. We propose reversing the measure of conservatism from data independent and alpha dependent to data dependent and alpha independent, to reflect the practice of reporting p-values rather than reject or no-reject decisions at a given alpha -level. Specifically, we define the p-value interval, whose lower end point is the smallest the p-value could have been without conservatism, and whose upper end point is the (generally conservative) traditional p-value. The length of the p-value interval is the null probability of the observed outcome and measures the conservatism of the test. The p-value interval allows for an explicit quantification of the extent to which more discriminating (or secondary) test statistics help to reduce conservatism by generating more outcomes, each with a lower null probability. Higher order p-value intervals can also be used to assess the robustness of statistical significance in terms of the number of patients required to switch treatment groups to break the observed statistical significance.