Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis

Rationale and Objectives. Solutions have previously been presented to the problem of estimating the components of variance in the general linear model used for multivariate receiver operating characteristic (ROC) analysis. The case where the variance components do not change across the modalities under comparison was first treated, followed by the case where they are permitted to change. No analysis of uncertainties in these estimates has been presented previously. Materials and Methods, For the case where the variance components do not change across modalities, the "jackknife-after-bootstrap" resampling procedure can be used together with conventional linear propagation of variance to solve for the uncertainties in estimates of the components. For the case where the components are permitted to change across modalities, a slight elaboration of this procedure is presented. Results. The approach was validated by Monte Carlo simulations, where uncertainties in estimates of the variance components calculated by the jackknife-after-bootstrap procedure were found to converge in the mean to the Monte Carlo results over many independent trials. The method is exemplified with data from a study of readers-with and without the aid of a computer-assist modality-given the task of discriminating benign from malignant masses in mammography. Conclusion. The present approach is relevant to a broad class of problems where estimates of multiple contributions to the variance observed in ROC assessment of diagnostic modalities are desired, in particular, for the assessment of multiple-reader studies of computer-aided diagnosis in radiology where the variance components may change across reading modalities (eg, unaided vs computer-aided reading).