ON THE GENERALIZATION OF THE RECEIVER OPERATING CHARACTERISTIC ANALYSIS TO THE POPULATION OF READERS AND CASES WITH THE JACKKNIFE METHOD - AN ASSESSMENT

Rationale and Objectives. A new methodology that analyzes receiver operating characteristic (ROC) data sets based on jackknifing and that considers both case and reader variability has been proposed. The purpose of this investigation was to compare results using this method to those using commonly reported methodology. Methods. ROC data sets using discrete and continuous rating scales were analyzed using the proposed jackknifing method, and results were compared to analysis of the same data sets using the paired t test. Results. The two methodologies did not result in the same significance levels, and in some cases, the difference was sufficient to affect conclusions regarding comparisons of diagnostic modalities. The probability value for the jackknifing procedure is based on large sample distribution theory, and its appropriateness is unknown for sample sizes used in practice. Also, the jackknifing technique was found to be sensitive to outliers resulting when data from the computer programs used to estimate area under the ROC curve failed to converge. Conclusion. Although the proposed methodology yields reasonable results, several fundamental and practical issues must be addressed before it can be used widely as the analytic method of choice in ROC studies comparing different imaging techniques or reading environments.