A pilot study of the use of the analytic hierarchy process for the selection of surgery residents

Purpose. To compare a specific decision-making process-the analytic hierarchy process (AHP)-with the traditional informal selection process in the selection of general surgery residents. Method. The study focused on 1994 and 1995 applicants for the four positions in the five-year general surgery residency program at the Graduate Hospital in Philadelphia. Three criteria were used: academic performance, personal fit, and surgical appropriateness. The relative importance of each was determined by pairwise comparison. For each hierarchy level, these comparisons were combined into a pairwise comparison matrix, and weights were deter mined for each criterion and rating category. The rating category weights for each criterion were scaled so that outstanding received the full criterion weight. Each applicant was interviewed by three committee members and rated with both the AHP system and the traditional 0-10 scoring system. In both cases the rating scores were averaged to create a single score for each applicant. The final ranking list (advocacy ranking) was compiled at a meeting of the entire selection committee, during which each member spoke on behalf of the candidates he or she had interviewed. Results. Significant Spearman correlations were found between the AHP ranking and the traditional ranking in both years (1994: n = 26, r = .63, p = .0005; 1995: n = 25, r = .061, P = .0012). The AHP ranking was also significantly correlated with the advocacy ranking in 1994 (n = 26, r = .43, P = .0273); however, there was no significant correlation found in 1995. In 1994 the traditional ranking significantly correlated with the advocacy ranking (n = 26, r = .40, p = .0423). This was not the case in 1995, suggesting that the results of the interviewing process had minimal influence on the outcome of the selection process that year. Conclusion. The findings from this pilot study support the use of the AHP as a viable alternative for the selection of surgical residents. Although the small sample size limits the generalizability of the results, the AHP is a quantitative alternative to the traditional, unwieldy and subjective selection process. Quantitative assessment and ranking of all aspects of a candidate's attributes and performance allow a program to more closely match a candidate to that particular institution.