Underestimation of standard errors in multi-site time series studies

Multi-site time series studies of the association of air pollution with mortality and morbidity have figured prominently in the literature as comprehensive approaches for estimating short-term effects of air pollution on health. Hierarchical models are generally used to combine site-specific information and to estimate pooled air pollution effects while taking into account both within-site statistical uncertainty and across-site heterogeneity. Within a site, characteristics of time series data of air pollution and health (small pollution effects, missing data, and highly correlated predictors) make the modeling of all sources of uncertainty challenging. One potential consequence is underestimation of the statistical variance of the site-specific effects to be combined. In this paper, we investigate the impact of variance underestimation on the pooled relative rate estimate. We focused on two-stage normal-normal hierarchical models and on underestimation of the statistical variance at the first stage. By mathematical considerations and simulation studies, we found that variance underestimation did not affect the pooled estimate substantially. However, the pooled estimate was somewhat sensitive to variance underestimation when the number of sites was small and underestimation was severe. These simulation results are applicable to any two-stage normal-normal hierarchical model for combining information of site-specific results (including meta-analyses), and they can easily be extended to more general hierarchical formulations. We also examined the impact of variance underestimation on the national average relative rate estimate from the National Morbidity, Mortality and Air Pollution Study. We found that variance underestimation as large as 40% had little effect on the national average.