Development of an evaluation of medical student competence in evidence-based medicine using a computer-based OSCE station

Background: Instruction in evidence-based medicine (EBM) has been widely incorporated into medical school curricula with little evidence of its effectiveness. Our goal was to create, implement. and validate a computer-based assessment tool that measured medical students' EBM skills. Description: As part of a required objective structured clinical examination, we developed a specific case scenario in which students (a) asked a structured clinical question using a standard framework. (b) generated effective MEDLINE search terms to answer a specific question. and (c) elected the most appropriate of 3 abstracts generated from a search justifying which best applies to the patient scenario. Evaluation: Between the 3 blinded raters, there was very good interrater reliability with 84, 94, and 96% agreement on the scoring for each component. respectively (kappa = .64, .82, and .91, respectively). In addition. students found the station appropriately difficult for their level of training. Conclusions: This computer-based tool appears to measure several EBM skills independently and combines simple administration and scoring. Its generalizability to other cases and settings requires further study.