Comparing classifiers when the misallocation costs are uncertain

Receiver Operating Characteristic (ROC) curves are popular ways of summarising the performance of two class classification rules. In fact, however, they are extremely inconvenient. If the relative severity of the two different kinds of misclassification is known, then an awkward projection operation is required to deduce the overall loss. At the other extreme, when the relative severity is unknown, the area. under an ROC curve is often used as an index of performance. However, this essentially assumes that nothing whatsoever is known about the relative severity - a situation which is very rare in real problems. We present an alternative plot which is more revealing than an ROC plot and we describe a comparative index which allows one to take advantage of anything that may be known about the relative severity of the two kinds of misclassification. (C) 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.