COMPARING THE PREDICTIVE ACCURACY OF MODELS USING A SIMPLE RANDOMIZATION TEST

A simple randomization t-test is proposed for testing the quality of performance of two prediction methods. The application of the test is shown to prevent unjustified conclusions about method superiority. Previous approaches to the problem of comparing predictive methods are discussed, and the proposed test is compared to other tests for paired data in a small simulation study. It is shown that the test can also be applied for classification problems where the predicted entity is qualitative rather than quantitative.