Alternative approaches to standard setting for licensing and certification examinations

In the most frequently used of Angoff's standard setting methodologies, judges develop a common understanding of minimally competent candidates and estimate the percentage of minimally competent candidates who would answer each item of an examination correctly. This method (percentage) and another Angoff method (yes-no) were explored in this study. Four separate groups of judges were provided behavioral descriptors or incidents to use when making ratings. Judges were asked to estimate the percentage of minimally competent candidates who would answer each item of an examination correctly (percentage) or to provide dichotomous ratings that affirmed or disaffirmed that minimally competent candidates would answer each item correctly (yes-no). The groups were allowed to make ratings in two stages: initial ratings without discussion and final ratings after presentation of item difficulty data and discussion with group members. Results indicated that the passing scores based on percentage estimates were stable from initial to final ratings. However, passing scores based on dichotomous (yes-no) ratings declined from initial to final ratings. Apparently, the judges using the yes-no method changed their judgments systematically after provision of item difficulty data and opportunity for discussion. The nature of the differences and the implications for setting passing scores for licensing examinations are discussed.