Adaptive optics observations of Vega: Eight detected sources and upper limits to planetary-mass companions

From adaptive optics observations with the Palomar 5 m telescope we place upper limits on the masses of any planetary companions located between similar to30 and 230 AU away from Vega, where our data are sensitive to depths ranging from H=12.5-19.0 mag fainter than Vega itself. Our observations cover a plus-shaped area with two 25" x 57" elements, excluding 7" x 7" centered on the star. We have identified two double and four single point sources. These projected companions are 14.9-18.9 mag fainter than Vega and, if physically associated, would have masses ranging from 4 to 35 M-J and orbital radii of 170-260 AU. Recent simulations of dusty rings around Vega predict the presence of a perturbing body with a mass of less than 2-3 M-J and an orbital radius of similar to40-100 AU, and more massive (less than or similar to10 M-J) planets cannot be excluded. None of the detected objects are this predicted planet. Based on a color-magnitude, spectroscopic, and proper motion analysis, all objects are consistent with being background sources. Given the glare of Vega, a 2 M-J object near the expected orbital radii would not have been visible at the 5 sigma level in our data, though any brown dwarf with mass greater than 10 M-J could have been seen at a separation greater than 80 AU.