Frequentist coverage properties of uncertainty intervals for weak Poisson signals in the presence of background

We construct uncertainty intervals for weak Poisson signals in the presence of background. We consider the case where a primary experiment yields a realization of the signal plus background, and a second experiment yields a realization of the background. The data acquisition times, for the background-only experiment, T-bg, and the primary experiment, T, are selected so that their ratio, T-bg/T, varies from 1 to 25. The upper choice of 25 is motivated by an experimental study at the National Institute of Standards and Technology (NIST). The expected number of background counts in the primary experiment varies from 0.2 to 2. We construct 90% and 95% confidence intervals based on a propagation-of-errors method as well as two implementations of a Neyman procedure where acceptance regions are constructed based on a likelihood-ratio criterion that automatically determines whether the resulting confidence interval is one-sided or two-sided. In one of the implementations of the Neyman procedure due to Feldman and Cousins (FC), uncertainty in the expected background contribution is neglected. In the other implementation, we account for random uncertainty in the estimated expected background with a parametric bootstrap implementation of a method due to Conrad. We also construct minimum length Bayesian credibility intervals. For each method, we test for the presence of a signal based on the value of the lower endpoint of the uncertainty interval. In general, the propagation-of-errors method performs the worst compared to the other methods according to frequentist coverage and detection probability criteria, and sometimes produces nonsensical intervals where both endpoints are negative. The Neyman procedures generally yield intervals with better frequentist coverage properties compared to the Bayesian method except for some cases where T-bg/T = 1. In general, the Bayesian method yields intervals with lower detection probabilities compared to Neyman procedures. One of the main conclusions is that when T-bg/T is 5 or more and the expected background is 2 or less, the FC method outperforms the other methods considered. For T-bg/T = 1, 2 we observe that the Neyman procedure methods yield false detection probabilities for the case of no signal that are higher than expected given the nominal frequentist coverage of the interval. In contrast, for T-bg/T = 1, 2, the false detection probability of the Bayesian method is less than expected according to the nominal frequentist coverage.