A regression-based equivalence test for model validation: shifting the burden of proof

Model validation is often realized as a test of how well model predictions match a set of independent observations. One would think that the burden of proof should rest with the model, to force it to show that it can make accurate predictions. Further, one would think that increasing the sample size ought to increase the model's ability to demonstrate its utility. Traditional statistical tools are inappropriate for this because they default to the case that the model and the data are no different, and their ability to detect differences increases with the sample size. These traditional tools are optimized to detect differences, rather than similarities. We present an alternative strategy for model validation that is based on regression and statistical tests of equivalence. Equivalence tests reverse the usual null hypothesis: they posit that the populations being compared are different and use the data to prove otherwise. In this sense, equivalence tests are lumping tests, whereas the traditional statistical tests are splitting tests. To date, model validation with equivalence tests has focused on comparisons of means. Our proposed test checks not only for similarity of means, but also for similarity between individual predictions and observations. The strategy is demonstrated using three case studies that differ in their modeling objectives, and for varied sample sizes. The proposed strategy provides a formal means of model validation that is superior to traditional statistical tests in each case.