Survey for transiting extrasolar planets in stellar systems. III. A limit on the fraction of stars with planets in the open cluster NGC 1245

We analyze a 19 night photometric search for transiting extrasolar planets in the open cluster NGC 1245. An automated transit search algorithm with quantitative selection criteria finds six transit candidates; none are bona fide planetary transits. We characterize the survey detection probability via Monte Carlo injection and recovery of realistic limb-darkened transits. We use this to derive upper limits on the fraction of cluster members with close-in Jupiter radii, R-J, companions. The survey sample contains similar to 870 cluster members, and we calculate 95% confidence upper limits on the fraction of these stars with planets by assuming that the planets have an even logarithmic distribution in semimajor axis over the Hot Jupiter (HJ; 3.0 < P day(-1) < 9.0) and Very Hot Jupiter (VHJ; 1.0 < P day(-1) < 3.0) period ranges. For 1.5R(J) companions we limit the fraction of cluster members with companions to < 6.4% and < 52% for VHJ and HJ companions, respectively. For 1.0R(J) companions we find that < 24% have VHJ companions. We do not reach the sensitivity to place any meaningful constraints on 1.0RJ HJ companions. From a careful analysis of the random and systematic errors of the calculation, we show that the derived upper limits contain a +13%/-7% relative error. For photometric noise and weather properties similar to those of this survey, observing NGC 1245 twice as long results in a tighter constraint on HJ companions than observing an additional cluster of richness similar to that of NGC 1245 for the same length of time as this survey. If 1% of stars have 1.5RJ HJ companions (as measured in radial velocity surveys), we expect to detect one planet for every 5000 dwarf stars observed for a month. To reach an similar to 2% upper limit on the fraction of stars with 1.5R(J) companions in the 3.0 < P day(-1) < 9.0 range, we conclude that a total sample size of similar to 7400 dwarf stars observed for at least a month will be needed. Results for 1.0RJ companions, without substantial improvement in the photometric precision, will require a small factor larger sample size.