Difficulties with recovering the masses of supermassive black holes from stellar kinematical data

We investigate the ability of three-integral, axisymmetric, orbit-based modeling algorithms to recover the parameters defining the gravitational potential (mass-to-light ratio T and black hole mass M-circle) in spheroidal stellar systems using stellar kinematical data. We show that the potential estimation problem is generically underdetermined when applied to long-slit kinematical data of the kind used for most black hole mass determinations to date. A range of parameters (Y, M-circle) can provide equally good fits to the data, making it impossible to assign best-fit values. The indeterminacy arises from the large variety of orbital solutions that are consistent with a given mass model. We demonstrate the indeterminacy using a variety of data sets derived from realistic models, as well as published observations of the galaxy M32. The indeterminacy becomes apparent only when a sufficiently large number of distinct orbits are supplied to the modeling algorithm; if too few orbits are used, spurious minima appear in the chi(2)(Y, M-circle) contours, and these minima do not necessarily coincide with the parameters defining the gravitational potential. We show that the range of degeneracy in M-circle depends on the degree to which the data resolve the radius of influence r(h) of the black hole. For FWHM/2r(h)greater than or similar to0.5, where FWHM refers to the instrumental resolution, we find that only very weak constraints can be placed on M-circle. In the case of M32, our reanalysis demonstrates that when a large orbit library is used, data published prior to 2000 (FWHM/2r(h)approximate to0.25) are equally consistent with black hole masses in the range 1.5x10(6) M-circle dot<M-circle<5x10(6) M-circle dot, with no preferred value in that range. Exactly the same data can reproduce previous published results with smaller orbit libraries. While the Hubble Space Telescope (HST) Space Telescope Imaging Spectrograph (STIS) data for this galaxy (FWHM/2r(h)approximate to 0.06) may overcome the degeneracy in M-circle, HST data for most galaxies do not resolve the black hole's sphere of influence and in these galaxies the degree of degeneracy allowed by the data may be greater than previously believed. We investigate the effect of regularization, or smoothness constraints, on the degree of degeneracy of the solutions. Enforcing smoothness reduces the range of acceptable models, but we find no indication that the true potential can be recovered simply by enforcing smoothing. For a given smoothing level, all solutions in the minimum chi(2) valley exhibit similar levels of noise; as the smoothing is increased, there is a systematic shift in the midpoint of the chi(2) valley, until at a high level of smoothing the solution is biased with respect to the true solution. These experiments suggest both that the indeterminacy is real (i.e., that it is not an artifact associated with nonsmooth solutions) and that there is no known empirical way to choose the smoothing parameter to ensure that the correct solution is selected.