Assessment of the validity and reliability of three systems of medical record screening for quality of care assessment

OBJECTIVES, The authors compared the validity and reliability of three methods of medical record screening for detecting problems with quality of care of patients undergoing cardiovascular procedures: (1) the Uniform Clinical Data Set System (UCDSS), a software-based system developed by the Health Care Financing Administration (HCFA), (2) the HCFA generic screens, and (3) the Harvard Medical Practice Study (HMPS) screening criteria. METHODS. A random sample of 451 medical records of inpatients who underwent cardiovascular procedures were screened by trained nurses and were also examined by physicians masked to screening results. Physicians' judgment provided a reference standard for validity assessment of screening applied by nurses. RESULTS. agreement between pairs of physician reviews about the quality of care was fair or slight (Kappa < 0.40). Agreement between nurses about presence of any screening flags was excellent for UCDSS (0.93), slight for HCFA (0.11), and fair for HMPS (0.41). The UCDSS was more sensitive to quality problems agreed on by two physicians than HCFA or HMPS (85%, 74% and 69%, respectively), but was also less specific (48%, 57%, and 56%, respectively). The UCDSS was much more labor-intensive with an average of 6.7 hours of nurse review per quality problem disclosed versus 2 hours required by HCFA and HMPS. CONCLUSIONS. The UCDSS computerized screens brought about substantial improvements in reliability of medical record screening, but needed to reduce its false-positive rate and to increase efficiency before it is used. From the perspective of continuous quality improvement, the UCDSS computerized algorithms provided a starting point to developing disease-specific patterns of practice to guide interventions.