USING AN EXTENDED ANGOFF PROCEDURE TO SET STANDARDS ON COMPLEX PERFORMANCE ASSESSMENTS

The number of assessments that are performance based is increasing rapidly, but to date there is not an established procedure for setting standards on these assessments. The purposes of this article are to describe several extensions to the Angoff procedure to accommodate the characteristics of a performance-based assessment and to present the results of our research to apply this new procedure. The extensions included a revised task for panelists, which involved the specification of (a) expected scores for just barely certifiable candidates on polytomously scored exercises and (b) weights to reflect the relative importance of scoring dimensions at the exercise level and the exercises themselves. The results obtained from the new procedure were mixed. On the one hand, panelists seemed to find the new procedure straightforward to apply and were able to reach a high level of agreement among themselves about the standards. Confidence levels in the resulting standards were high. However, when given the choice, panelists definitely preferred a standard-setting procedure that was at least partly conjunctive, as opposed to the extended Angoff procedure that was completely compensatory in its nature. Suggestions for follow-up research are offered in the article.