A MULTISTAGE DOMINANT PROFILE METHOD FOR SETTING STANDARDS ON COMPLEX PERFORMANCE ASSESSMENTS

During pilot testing of standard-setting procedures for use with complex performance assessments, panelists were asked to construct standards-related policies schematically and/or textually. Schematic constructions consisted of profiles characterizing their judgments of minimally acceptable performance on a multi-exercise performance assessment, in the sense that such performance barely warranted certification as a highly accomplished teacher. Some panelists also wrote descriptions of minimally acceptable performance policies. Written and constructed policies were compared to panelists' judgments when using the judgmental policy capturing standard-setting procedure for consistency between the two methods, essentially a comparison of panelists' explicit and implicit policies. The multi-stage dominant profile method shows promise for training panelists in standard setting, demonstrating implications of panelists' policies for certification decisions, and determining the policy most preferred by panelists.