Performance assessment of community-based physicians: Evaluating the reliability and validity of a tool for determining CME needs

Purpose. To evaluate the reliability, validity, and feasibility of the Physician Assessment in Medical Practice (PAMP) as a means of determining the CME needs of practicing, community-based physicians. Method. A group of 45 randomly selected community-based physicians (19 certified family physicians and 26 general practitioners) affiliated with the Department of Family Medicine at Bruce Rappaport Faculty of Medicine, Technion Institute of Technology, Haifa, Israel, volunteered to participate in the study, conducted in 1997. All participants took a ten-station, performance-based examination designed to closely represent the physicians' work settings. At each station, a different medical problem was presented by a standardized patient. Physician-candidates' performances were assessed by physician-examiners using global ratings. The following performance domains were assessed: information gathering, diagnosis and management plan, and communication skills. A CME needs assessment scone was determined for each of the participants and a CME level to meet the needs of the physician was recommended. Results. Overall reliability of the examination was high (.87), with domain reliabilities ranging from .76 to .87. Reliability of the examiners' judgments of the physicians' competence was .66. All the stations' validity scores were significant, and differences in performances between family physicians and general practitioners demonstrated construct validity of the test results. Overall, the cost of running the examination was U.S. $250 per physician-candidate. Conclusions. Using the PAMP to determine CME needs of community-based physicians was found to reliable, valid, and feasible, and the cost per physician-candidate was not excessive. Performance results provided indepth information for use by both the individual physician and providers of CME programs.