Estimation of the reliability of ensemble-based probabilistic forecasts

Reliability is an essential attribute of the quality of probabilistic forecasts. It is traditionally estimated by defining a number of arbitrary probability categories. Reliability is often difficult to estimate accurately with a small sample size. This occurs, for example, when evaluating high probabilities of rare events. Significance tests are used in this study in order to determine an appropriate categorization of forecast probabilities for the estimation of reliability. For events occurring frequently, this method leads to credible estimates of the performance for the whole range of forecast probabilities. On the other hand, the reliability of higher probabilities for infrequent events cannot be estimated with confidence. A statistical scheme has been designed for estimating reliability from limited samples, even in the case of rare events and higher probabilities. The procedure consists of fitting a Relative Operating Characteristic (ROC) curve under the bi-normal assumption. The validity of the method is discussed by testing its ability to estimate reliability from truncated verification samples. The positive impact of a basic method of calibration is increased when it is applied after an estimation of reliability through a fitting of the ROC curve.