Analysis of quality and feasibility of an objective structured clinical examination (OSCE) in preclinical dental education

Introduction: An objective structured clinical examination (OSCE) has been implemented in preclinical dentistry. It was taken at an early stage (propaedeutics course). The objectives of this study were to evaluate the reliability, validity, and feasibility of the examination, and the effect of circuit number on OSCE score. Methods: The OSCE was designed by an expert committee on the basis of pre-reviewed blueprints and checklists. Eleven stations formed an interdisciplinary circuit. Six groups of students (n = 62) passed sequentially round the same circuit. Statistical analysis was performed by using SPSS. Reliability was determined by measurement of internal consistency (Cronbach's alpha, Guttman's lambda(2)), standard error of measurement (SEM) (comprising generalisability index a, dependability index phi and passfail reliability pc), consistency coefficient kappa, itemscale correlation (Pearson correlation), and, because the unidimensionality of the stations could not be assumed, factor analysis including varimax rotation. Convergent validity (Pearson correlation, t-test), and predictive validity for future preclinical courses and the final preclinical examination were assessed by analysis of variance (ANOVA). The effect of the circuit number on score improvement was calculated, including a correction for the general competence of the students (ANOVA). Cost was calculated on the basis of the time invested. Results: Fifty-three out of sixty-two students passed the OSCE (mean score: 67%, SD 7.7, range, 47-81). Scores for each station correlated significantly with total scores (r = 0.35-0.54, P < 0.01). For internal consistency, alpha = 0.75 (relative SEM 3.8) and lambda(2) = 0.766. The dependability index was phi = 0.694 (absolute SEM 4.4), p(c) = 0.89 and kappa = 0.61. Factor analysis yielded two components: dental-materials-oriented stations and all other stations (explained variance 43%). Scores correlated significantly with success in passing practical tests (i.e. performing dental procedures under examination conditions) (known group validity, P < 0.01) and with scores for subsequent courses and the final preclinical examination (Physikum) (predictive validity, P < 0.001). Later groups performed 4% better on average (CI 95%: 1.2-6.8%; P < 0.01). The cost was 181 Euro per student. Conclusions: The OSCE is reliable and valid in the context of preclinical dentistry. The cost is substantial. The problem of improvement of students' results with ascending circuit number has to be addressed.