MULTIREADER, MULTICASE RECEIVER OPERATING CHARACTERISTIC METHODOLOGY - A BOOTSTRAP ANALYSIS

Rationale and Objectives. We evaluated by bootstrapping the conclusions obtained by the Dorfman-Berbaum-Metz (DBM) receiver operating characteristic (ROC) method and by the Toledano-Gatsonis (TG) method on a well-known data set. Methods. We bootstrapped in two ways: resampled cases while holding readers fixed and resampled both cases and readers. Results. When an analysis of variance of pseudovalues implies that reader variance and all random interactions with treatment are essentially zero, then case-resampling bootstrap and the DBM and TG methods should give the same results. Case-resampling bootstrap and the DBM and TG methods did give highly similar results for both individual readers and the averages over all readers. Both the case-resampling bootstrap and the reader-case resampling bootstrap gave smaller standard errors for group than for individual reader means, thereby providing evidence for a trade-off of readers and cases with regard to precision and power in this data set. Conclusion. Case-resampling bootstrap provides some justification for the DBM and TG methods.