Variations in measured performance of CAD schemes due to database composition and scoring protocol

There is now a large effort towards developing computer-aided diagnosis (CAD) techniques. It is important to be able to compare performance of different approaches to be able to determine which ones are the most efficacious. There are currently a number of barriers preventing meaningful (statistical) comparisons, two of which are discussed in this paper: database composition and scoring protocol. We have examined how the choice of cases used to test a CAD scheme can affect its performance. We found that our computer scheme varied between a sensitivity of 100% to 77%, at a false-positive rate of 1.0 per image, with only 10% change in the composition of the database. To evaluate the performance of a CAD scheme the output of the computer must be graded. There are a number of different criteria that are being used by different investigators. We have found that for the same set of detection results, the measured sensitivity can be betwen 40-90% depending on the scoring methodology. Clearly consensus must be reached on these two issues in order for the field to make rapid progress. As it stands now, it is not possible to make meaningful comparisons of different techniques.