Independent versus sequential reading in ROC studies of computer-assist modalities: Analysis of components of variance

Rationale and Objectives. The authors analyzed two methods for arranging the temporal sequencing of unaided versus computer-assisted reading in multiple-reader, multiple-case receiver operating characteristic studies of detection of solitary pulmonary nodules on chest radiographs. In the "independent" mode the readings are separated by about 1 month; in the "sequential" mode, the computer-assisted reading immediately follows the unassisted reading. Materials and Methods. The authors used the method of Beiden, Wagner, and Campbell (BWC) to decompose the components of variance of receiver operating characteristic accuracy measures into those that are correlated and those that are uncorrelated across reading conditions. Only the latter contribute to uncertainty in estimates of the difference in accuracy measures across reading conditions (unaided vs aided). This method was used to analyze data from two independent studies of the detection of solitary pulmonary nodules on chest radiographs. Results. In the sequential reading mode the components that were Correlated across reading conditions increased compared to the independent reading mode, as might be expected. What was not anticipated was the fact that the total reader variance was approximately the same for the two reading modes. The results were remarkably similar across the two independent studies analyzed. Conclusion. The sequential reading mode may thus be the more sensitive probe of the difference between unassisted and computer-assisted reading, if the mean effect is unperturbed (as here). It is also the least demanding on the logistics and investment of reader time. (C) AUR, 2002.