Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment

Performance assessments typically are scored by having experts rate individual performances. In contexts such as medical licensure, where the examinee population is large and the pool of expert raters is limited practically, this approach may be unworkable. This article describes an automated scoring algorithm for a computer simulation-based examination of physicians' patient-management skills. The algorithm is based on the policy used by clinicians in rating case performances. The results show that scores produced using this algorithm are highly correlated to actual clinician ratings. These scores also are shown to be effective in discriminating between case performance judged to be passing or failing by an independent group of clinicians.