Surgical skills assessment: A blinded examination of obstetrics and gynecology residents

OBJECTIVE: We have previously shown that objective structured assessment of technical skills (OSATS) is an innovative, reliable, and valid method of assessing surgical skills. Our goal was to establish the feasibility, reliability, and validity of our surgical skills assessment instrument when administered in a blinded fashion. STUDY DESIGN: A 7-station OSATS was administered to 16 obstetric and gynecology residents from Madigan Army Medical Center. The test included laparoscopic (salpingostomy, intracorporeal knot, and ligation of vessels with clips) and open abdominal procedures (subcuticular closure, bladder neck suspension, enterotomy repair, and abdominal wall closure). All tasks were performed with lifelike surgical models. Residents were timed and assessed at each station with 3 methods of scoring: task-specific checklist, global rating scale, and pass/fail grade. Each resident was evaluated by one examiner blinded as to the postgraduate year level and one examiner who had previously worked with the resident. RESULTS: Assessment of construct validity (the ability to distinguish between resident levels) found significant differences on the checklist, global rating scale, and pass/fail grade by residency level for both blinded and unblinded examiners. Reliability indices calculated with Cronbach's alpha were .82 for the checklists and .93 for the global rating scale. Overall interrater reliability between blinded and unblinded examiners was 0.95 for global rating scale and ranged from 0.74 to 0.97 for the checklists. The cost to administer the exam for the 16 residents was approximately $1000. CONCLUSIONS: OSATS administered in either a blinded or unblinded fashion can assess residents' surgical skills with a high degree of reliability and validity. This study provides further evidence that OSATS can be used to establish surgical competence.