A MONTE-CARLO STUDY OF THE 632-BOOTSTRAP ESTIMATOR OF ERROR RATE

Efron (1983) proposed the 632 bootstrap error rate estimator, and demonstrated its good performance in simulations. However, further recent studies have suggested that it only performs well for a restricted range of true error rates. We conducted an intensive simulation study of the 632 estimator to investigate this hypothesis in detail. There was no evidence to support die contention, although the estimator's performance did vary with the true error rate. We also investigated some more general aspects of error rate studies. Error rate is bounded by 0 and 1, so that comparisons based on an untransformed scale may be inappropriate. We therefore explored the consequences of stretching the error rate scale. Furthermore, the results of such studies are typically generated as mixture distributions, because the performance results are averages over the true error rate values arising from underlying distributions which have a fixed optimal error rate. We discovered, rather surprisingly, that this has little effect if the conclusions are based on mean squared error.