How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patient visits

OBJECTIVES. This study was designed to determine the optimal nonobservational method of measuring the delivery of outpatient medical services. METHODS. AS part of a multimethod study of the content of primary care practice, research nurses directly observed consecutive patient visits to 138 practicing family physicians. Data on services delivered were collected using a direct observation checklist, medical record review, and patient exit questionnaires. For each medical service, the sensitivity, specificity, and Kappa statistic were calculated for medical record review and patient exit questionnaires compared with direct observation. Interrater reliability among eight research nurses was calculated using the Kappa statistic for a separate sample of videotaped visits and medical records. RESULTS. Visits by 4,454 patients were observed. Exit questionnaires were returned by 75% of patients. Research nurse interrater reliabilities were generally high. The specificity of oth the medical record and the patient exit questionnaire was high for most services. The sensitivity of the medical record was low for measuring health habit counseling and moderate for physical examination, laboratory testing, and immunization. The patient exit questionnaire showed moderate to high sensitivity for health habit counseling and immunization and variable sensitivity for physical examination and laboratory services. CONCLUSIONS. The validity of the medical record and patient questionnaire for measuring delivery of different health services varied with the service. This report can be used to choose the optimal nonobservational method of measuring the delivery of specific ambulatory medical services for research and physician profiling and to interpret existing health services research studies using these common measures.