Variance-component modeling in the analysis of receiver operating characteristic index estimates

Rationale and Objectives. The authors performed this study to clarify and systematize the large number of variances and correlations observable with variance-component models of receiver operating characteristic (ROC) index estimates. Materials and Methods. The authors present a variance-component model for ROC index estimates (and for differences between estimates) and show correspondences between the method of experimental replication and the random components in the model. The authors introduce a notation that identifies both the method of replication and, when examining estimate differences, the estimate pairing scheme. Results. For models with three factors (modality, reader, case sample), the authors delineated four methods of replication and eight pairing schemes for generating estimate differences. For each of the resulting 32 replication-pairing combinations, the authors gave expressions for the variance of the difference and for the correlation between the two ROC index estimates. Conclusion. The variance-component approach is a useful statistical tool for modeling different sources of variation that contribute to the overall variance of ROC data index estimates derived from those data.