Aggregate scoring derives weights for the responses to test questions that are the proportion of a criterion group of experts; examinees’ test scores are simply the sum of the weights of the responses they choose. This study applied aggregate scoring to a recertifying examination where it is particularly useful. It is an efficient means of generating an answer key, it ensures that the answer key reflects differences in practice, and examinees may find it reassuring to be judged against the performance of their peers. Results indicated considerable agreement between the traditional answer key and the aggregate answer key. Although the scores produced by the two answer keys were similar, aggregate scoring slightly favored individuals out of training longer. Generalizability analyses (Brennan, 1983) produced the expected results. The use of several experts in aggregate scoring made a sizeable contribution to reduction in measurement error. The choice of either completely crossed designs or nested designs for collecting the responses of experts depends on the resources available. © 1990, SAGE PUBLICATIONS, INC. All rights reserved.
