Construct and face validity of MIST-VR, Endotower, and CELTS - Are we ready for skills assessment using simulators?

Background: Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators. Methods: Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training). Results: Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two parameters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 +/- 0.3 for MIST-VR, 7.9 +/- 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 +/- 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6%, 0%, and 36% of "expert" participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively. Conclusions: All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.