Identifying adverse events caused by medical care: Degree of physician agreement in a retrospective chart review

Objective: To 1) assess the degree of agreement among physicians on the cause of previously flagged adverse outcomes and 2) relate the findings to systems of quality assurance and performance assessment and proposals for no-fault compensation for medical injuries. Design: Observational study of 7533 pairs of ''structured implicit'' reviews (subjective opinions based on guidelines) of medical records done by 127 physicians working independently. Setting: Random sample of 51 inpatient facilities in New York State. Patients: Random sample of inpatient medical records from the selected facilities. Measurements: 1) Number of agreed-upon adverse events compared with the number of cases of extreme disagreement and 2) internally and indirectly standardized rates at which physician reviewers found adverse events (injuries to patients caused at least in part by medical management). Results: In 12.9% of cases (971 of 7533), the two physicians in a pair had extreme disagreement about the occurrence of an adverse event. These cases outnumbered those in which both reviewers found an adverse event (10%; n = 757). Agreement was highest for wound infections and lowest for adverse events attributed to failure to diagnose or lack of therapy. The amount of experience the physicians had in reviewing records tended to increase the level of agreement. Even after standardization to the results of the entire sample, individual physicians' rates of finding at least slight evidence of an adverse event varied widely (range, 9.9% to 43.7%) (P < 0.001). Conclusions: Structured implicit reviews produced disagreement on the causes of adverse patient outcomes. If systems of quality assurance, performance audits, or no-fault patient compensation are to succeed, methods for overcoming the common tendency toward disagreement among experts must be developed.