Effects of sample size on classifier design for computer-aided diagnosis

One of the important issues in the development of computer-aided diagnosis (CAD) algorithms is the design of classifiers. A classifier is designed with case samples drawn from the patient population. Generally, the sample size available for classifier design is limited, which introduces bias and variance into the performance of the trained classifier. Fukunaga showed that the bias on the probability of misclassification is proportional to 1/N-t, where Nt is the design (training) sample size, under conditions that the higher-order terms can be neglected. For CAD applications, a commonly used performance index for a classifier is the area, A,, under the receiver operating characteristic (ROC) curve. We have studied the dependence of the bias in A, on sample size by computer simulation for a linear classifier and nonlinear classifiers such as the quadratic and the backpropagation neural network classifiers. The performance of a classifier may be evaluated in two ways: one is the resubstitution method in which the same sample set is used for training and testing, the other is the hold-out method in which the available sample set is partitioned into independent training and test subsets. For a given finite design (training) sample size, the resubstitution method optimistically biases the performance of the classifier whereas the hold-out method pessimistically biases the performance. The former estimation provides an upper bound and the latter provides a lower bound on the true classifier performance. As the design sample size increases, the upper and lower bounds converge towards the unbiased estimate. In the regime where the performance is dominated by the 1/Nt term, the classifier performance in the limit of infinite sample size (N-t -->infinity) can be estimated as the intercept (1/N-t = O) of a linear regression of the performance versus 1/N-t. We have studied the biases on A, for three types of class distributions that have specific properties: equal covariance matrices with unequal means, unequal covariance matrices with unequal means, and unequal covariance matrices with equal means. The performances of the linear discriminant, quadratic discriminant, and the backpropagation neural network classifiers were compared under the different input conditions and their dependence on 1/Nt was studied.