Quantitative validation of a general competency composite assessment evaluation

Objectives: The authors sought to modify and validate a composite assessment evaluation process that assesses resident acquisition of the Accreditation Council for Graduate Medical Education (ACGME) general competencies (GCs). Methods: This study critically analyzed the evaluation process used in a multicenter study (150 emergency medicine resident evaluations) to determine whether the procedure was psychometrically valid. For each GC, principal component analysis (PCA) was used to determine whether certain evaluation items could be eliminated, as well as to determine the magnitude of variability explained by up to three linear combinations or "principal components." The factor proportions (factor loadings) of various eigenvectors were measured to determine the degree of variability (determined by the square of the factor proportion) within a data or item set. The factor proportions essentially measure the length of the eigenvector as determined from a correlation matrix. Results: The first three principal components are reported as factor proportion sum (% of total variability) as follows: patient care 0.91 (83%), medical knowledge 0.87 (76%), practice-based learning and improvement 0.90 (81%), interpersonal and communication skills 0.84 (71%), professionalism 0.74 (55%), and systems-based practice 0.80 (64%). PCA showed that evaluating certain traditional categories such as medical knowledge seemed to capture a single element, whereas professionalism appeared to measure a more complex, multidimensional phenomenon. Conclusions: By using a structured development process, the authors were able to create valid evaluation items for determining resident acquisition of the ACGME GCs.