A comparison of the National Ambulatory Medical Care Survey (NAMCS) measurement approach with direct observation of outpatient visits

Background: The National Ambulatory Medical Care Survey (NAMCS) informs a wide range of important policy and clinical decisions by providing nationally representative data about outpatient practice. However, the validity of the NAMCS methods has not been compared with a reference standard. Methods: Office visits of 549 patients visiting 30 family physicians in Northeastern Ohio were observed by trained research nurses. Visit content measured by direct observation was compared with data reported by physicians using the 1993 NAMCS form. Results: Outpatient visit physician reports of procedures and examinations using the NAMCS method showed generally good concordance with direct observation measures, with kappas ranging from 0.39 for ordering a chest x-ray to 0.86 for performance of Pap smears. Concordance was generally lower for health behavior counseling, with kappas ranging from 0.21 for alcohol counseling to 0.60 for smoking cessation advice. The NAMCS form had high specificity (range, 0.90-0.99) but variable (range, 0.12-84) sensitivity compared with direct observation, with the lowest sensitivities for health behavior counseling. The NAMCS physician report method overestimated visit duration in comparison with direct observation (16.5 vs. 12.8 minutes). Conclusions: Compared with direct observation of outpatient visits, the NAMCS physician report method is more accurate for procedures and examinations than for health behavior counseling. Under-reporting of behavioral counseling and overreporting of visit duration should lead to caution in interpreting findings based on these variables.