A comparison of standard-setting procedures for an OSCE in undergraduate medical education

Purpose. To compare four standard-setting procedures for an objective structure clinical examination (OSCE). Methods. A 12-station OSCE was administered to 84 students in each of the final (fourth-) year medical classes of 1996 and 1997 at Dalhousie University Faculty of Medicine. Four standard-setting procedures (Angoff, borderline, relative, and holistic) were applied to the data to establish a cutoff score for a pass/fail decision. Results. The procedures yielded highly inconsistent results. The Angoff and borderline procedures gave similar results; however, the relative and holistic methods gave widely divergent results. The Angoff procedure yielded results reliable enough to use in decision making for a high-stakes examination, but would have required more judges or more stations. Conclusions. The Angoff and borderline procedures provide reasonable and defensible approaches to standard setting and are practical to apply by non-psychometricians in medical schools. Further investigation of the other procedures is needed.