Assessing the generalizability of OSCE measures across content domains

Purpose. To assess the degree to which OSCE measures generalize across multiple administrations to the same students, Method. Students' scores from three OSCEs at one institution were correlated to determine the generalizability of the scoring systems across course domains. Results. Analysis revealed that while checklist scores showed quite low correlations across examinations from different domains (ranging from 0.14 to 0.25), global process scores showed quite reasonable correlations (ranging from 0.30 to 0.44), with the correlations for global scores being significantly higher than those for checklist scores in all three comparisons. Conclusion. These data would seem to confirm the intuitions about each of these measures: the checklist scores are highly content-specific, while the global scores are evaluating a more broadly based set of skills. Implications for the use of these scales are discussed.