ELICITING PROBABILITIES FROM EXPERTS IN COMPLEX TECHNICAL PROBLEMS

Expert judgment is used throughout all technical analyses of complex problems. In recent years the need for defensibility and accountability of these analyses has increased, leading, among other things, to an increased review and scrutiny of the expert judgments used. Too often, both the process and procedures for obtaining and using expert judgments are found to be inadequate. A formal elicitation process is summarized to obtain probabilities from experts in a large-scale study involving nuclear safety. A first expert elicitation, using primarily in-house experts and staff members of the project team was strongly criticized in peer review. Subsequently, many changes in the process and procedures were made including training experts in making probability judgments, providing assistance in decomposing judgments, formally eliciting probabilities in individual sessions, and documenting the results. Approximately 1000 probability distributions were assessed from about 40 experts from universities, consulting firms, and national laboratories. This second elicitation was much better received by peer review. Based on this experinence, a comprehensive process to elicit probability judgments is outlined in detail.