Knowledge acquisition from multiple experts: An empirical study

Expert systems often employ a weight on rules to capture conditional probabilities. For example, in classic rule-based settings, Pr(h \ e) = x is used to mean "If e is known to be true then conclude h is true with probability x." Further, other probability-based approaches, such as influence diagrams and Bayes' Nets are increasingly being used to support decision making through decision support systems. Although algorithms for these systems have received substantial attention, less attention has been given to knowledge acquisition of probabilities used in these systems. However, the underlying probabilities are critical because they lead the user to particular solutions. Accordingly, the purpose of this paper is to investigate the quality of probability knowledge when it is acquired from groups or individuals. This paper summarizes the results of an empirical cognitive study on the ability of individuals and groups to provide consistent sets of probabilities Pr(A), Pr(B), Pr(A \ B) and Pr(B \ A). The analysis of these probabilities allowed the study of the ability of subjects to account for Bayes' theorem reversals, a basic assumption made by virtually all algorithms. It was found that knowledge acquisition from groups provided more correct orderings to the probabilities than knowledge acquisition from individuals. This suggests that knowledge acquisition from groups is more likely to obtain correct probability knowledge.