THE CRITICAL OPERATING CHARACTERISTIC AND THE EVALUATION OF EXPERT JUDGMENT

The situation in which an expert makes a judgment or renders an opinion about the occurrence of an event, and the event either occurs or does not, may fruitfully be considered from two points of view: considering the expert as a signal detection system; and considering the consumer of the expert opinion who must decide whether or not to take some specific action based on that opinion. The dual-Gaussian, variable-criterion signal detection model provides a good description of expert behavior and forms the basis for distinguishing between accuracy of the expert and performance of the expert. Accuracy of the expert is defined in terms of the receiver operating characteristic (ROC) and is independent of decision criteria and prior probability of the event. Accuracy is measured either by da or Az. The performance of the expert, on the other hand, depends not only on the accuracy of the expert, but also on both the decision criteria used by the expert and the prior probability of the event. Two new concepts, the critical operating characteristic (COC) and the critical hit rate, HRcr, are introduced to allow the performance of experts to be meaningfully compared. Different ways in which expert judgment may be used by the consumer are also discussed. © 1992.