Myths and truths concerning estimation of power spectra: the case for a hybrid estimator

It is widely believed that maximum-likelihood estimators must be used to provide optimal estimates of power spectra. Since such estimators require the inversion and multiplication of N-d x N-d matrices, where N-d is the size of the data vector, maximum-likelihood estimators require at least of order N-d(3) operations and become computationally prohibitive for N-d greater than a few tens of thousands. Because of this, a large and inhomogeneous literature exists on approximate methods of power-spectrum estimation. These range from manifestly suboptimal but computationally fast methods, to near-optimal but computationally expensive methods. Furthermore, much of this literature concentrates on the power-spectrum estimates rather than the equally important problem of deriving an accurate covariance matrix. In this paper, I consider the problem of estimating the power spectrum of cosmic microwave background (CMB) anisotropies from large data sets. Various analytic results on power-spectrum estimators are derived, or collated from the literature, and tested against numerical simulations. An unbiased hybrid estimator is proposed that combines a maximum-likelihood estimator at low multipoles and pseudo-Cl estimates at high multipoles. The hybrid estimator is computationally fast (i.e. it can be run on a laptop computer for Planck-sized data sets), nearly optimal over the full range of multipoles, and returns an accurate and nearly diagonal covariance matrix for realistic experimental configurations (provided certain conditions on the noise properties of the experiment are satisfied). It is argued that, in practice, computationally expensive methods that approximate the O(N-d(3)) maximum-likelihood solution are unlikely to improve on the hybrid estimator, and may actually perforin worse. The results presented here can be generalized to CMB polarization and to power-spectrum estimation using other types of data, such as galaxy clustering and weak gravitational lensing.