The educational value of an OSCE in a family practice residency

Purpose. To assess the educational value of an objective structured clinical examination (OSCE) administered in three consecutive years (1992-93 to 1994-95) to first-year residents in a family practice residency. Method. Each year an OSCE was administered early in a family practice residency based at the University of North Carolina at Chapel Hill School of Medicine. The OSCE encompassed eight to ten stations, each 15 minutes long: ten minutes for tasks and five minutes for feedback. After the OSCE, focus groups with participating faculty and residents assessed the strengths and weaknesses of specific stations as well as the overall OSCE structure. Each year the residents' OSCE scores were correlated with their scores on the American Board of Family Physicians In-Training Examination (ITE). Pearson product-moment correlation coefficients were calculated for both the composite ITE scores and the clinical set problem scores. Test reliability was measured by Cronbach's alpha. In the spring of 1995 the faculty completed global evaluations of all the residents in the program at that time, including 16 of the 17 residents who had participated in the OSCEs. The faculty rated the residents in four areas, and a four-point scale was based on the percentile ranges a resident's rating could fall into for each area. Correlation coefficients were calculated for the OSCE scores and the faculty's perceptions of the residents' overall performance. Results. The OSCE scores correlated significantly with the faculty's global assessments of knowledge and clinical skills, but not with in-training examination scores. Concurrent validity and reliability estimates did not support using an OSCE for decisions about the residents' competency. The faculty valued the opportunity to observe basic skills for the whole group of residents; the residents found the experience educational but stressful and valued the immediate performance feedback included at each station, Conclusion. The OSCE is a useful tool for teaching basic clinical skills and for forming initial impressions of interns' clinical styles and abilities, but it should be used with caution for summative assessments.