Evaluation of web-based learning tools: Lessons learned from the Thoracic Surgery Directors Association curriculum project three-year experience

Background. In 2001 the Thoracic Surgery Directors Association implemented a web-based curriculum for prematriculated TS residents containing 75 learning modules organized into 13 sections Each module contains an audio/video presentation, a summary, frequently asked questions, references, a self-assessment, and a critique. Curriculum content was provided as a randomized prospective trial one year prior to matriculation to residents matching into a TS program from 2001 to 2004. Methods. Over 3 years the utilization and impact of this curriculum was evaluated by 8 different methods including resident surveys, faculty surveys, web tracking, examination scores, and individual online learning module critiques. Each completed critique evaluated the resident's perception of the relevance, content quality and presentation quality of the module on a scale corresponding to low, average and high. Results. A total of 11,117 learning modules were used through the end of 2004 with complete critiques available on 1458 (13%). Utilization and measured quality continue to increase over the 3-year interval. The average rating scores for relevance, content and presentation tracked together and were overall rated as good or excellent (92%). There were large variations in utilization and perceived quality by section subject. As the average module rating score improved, the utilization increased and the variability of the rating decreased substantially. Conclusions. Module specific learner evaluation of educational content is one of several important feedback tools for monitoring ongoing curricular development and refinement. These studies, based upon web tracking, demonstrate increasing resident use and strong resident satisfaction. This educational format will hopefully lend itself to many important improvements in web-based Thoracic Surgery education.