The reliability of multiple objective measures of surgery and the role of human performance

Background: There is a need for reliable and valid objective methods of technical skills in surgery. Six-bench surgical top stations have been combined to assess basic surgical trainees (BSTs) objectively. The current study examines its reliability and validity across repeat sittings. Methods: Eleven surgical trainees (6 senior BSTs and 5 higher surgical trainees [HSTs]) undertook 5 sittings of the 6-station assessment designed to be completed within 90 minutes. The 6 stations consisted of knot tying, suturing, closure of enterotomy, excision of sebaceous cyst, laparoscopic task, and instrument examination, Methods of analysis employed were motion analysis, observation with criteria, and inbuilt simulation metrics. Results: On analysis 3 knot tying and suturing stations exhibited significant differences in either time or movement; any difference was over by the second run. The intertest reliabilities were .66,.74,.55,.51, and .65 for the 5 runs. The intratest reliability across repeated sittings varied from .56 to .96. The inter-rater reliability for video assessment varied from .77 to .94. Conclusion: The assessment is reliable and valid across repeated sittings. Its use in assessment of basic technical skills needs to be encouraged. (c) 2005 Excerpta Medica Inc. All rights reserved.