Simplifying the 360-Degree Peer Evaluation in a Physical Medicine and Rehabilitation Residency Program

Sorg JC, Wilson RD, Perzynski AT, Tran D, Vargo MM: Simplifying the 360-degree peer evaluation in a physical medicine and rehabilitation residency program. Am J Phys Med Rehabil 2012; 91: 797-803. Objective: A previous study determined that multiple questions on a 360-degree evaluation instrument were highly correlated, suggesting possible redundancy in what was being measured, and that some questions may be eliminated. The current study uses factor analysis and examines a larger data set to further explore this question. Design: To evaluate the structure of the questionnaire, a factor analysis was performed on 3 yrs of data from a 19-item resident 360-degree evaluation. The number of factors was determined by examining the scree plot of eigenvalues for each item in the instrument, with a cutoff where the slope changes from rapid to slow decline. A reliability analysis was performed with the indicated number of factors, with deletion of each variable to evaluate its influence on overall reliability (Cronbach alpha). Results: There were 299 evaluations with complete responses to all 19 questions. The scree plot supported a single factor model. The reliability of the full, single factor survey was excellent (Cronbach alpha = 0.99). The three items with the highest loading on the factor were retained, which related to humanistic, moral/ethical, and professional responsibility behaviors. The reliability for these final three items remained excellent (Cronbach alpha = 0.96). Conclusions: The factor analysis suggests that one overall opinion of the evaluated resident was informing the responses of the evaluator. Shortening the instrument to the three items responsible for the greatest influence on the survey does not result in a large decrease in reliability as measured by Cronbach alpha. It is possible that time limitations prevent residents from putting thought into the evaluation of their peers, which results in unidimensional responses. Shortening the instrument may improve evaluations and should be studied in the future.