Neighborhood-Based Verification of Precipitation Forecasts from Convection-Allowing NCAR WRF Model Simulations and the Operational NAM

Since 2003 the National Center for Atmospheric Research (NCAR) has been running various experimental convection-allowing configurations of the Weather Research and Forecasting Model (WRF) for domains covering a large portion of the central United States during the warm season (April-July). In this study, the skill of 3-hourly accumulated precipitation forecasts from a large sample of these convection-allowing simulations conducted during 2004-05 and 2007-08 is compared to that from operational North American Mesoscale (NAM) model forecasts using a neighborhood-based equitable threat score (ETS). Separate analyses were conducted for simulations run before and after the implementation in 2007 of positive-definite (PD) moisture transport for the NCAR-WRF simulations. The neighborhood-based ETS (denoted < ETS >(r)) relaxes the criteria for "hits" (i.e., correct forecasts) by considering grid points within a specified radius r. It is shown that < ETS >(r) is more useful than the traditional ETS because < ETS >(r) can be used to diagnose differences in precipitation forecast skill between different models as a function of spatial scale, whereas the traditional ETS only considers the spatial scale of the verification grid. It was found that differences in < ETS >(r) between NCAR-WRF and NAM generally increased with increasing r, with NCAR-WRF having higher scores. Examining time series of < ETS >(r) for r - 100 and r - 0 km (which simply reduces to the "traditional" ETS), statistically significant differences between NCAR-WRF and NAM were found at many forecast lead times for < ETS >(100) but only a few times for < ETS >(0). Larger and more statistically significant differences occurred with the 2007-08 cases relative to the 2004-05 cases. Because of differences in model configurations and dominant large-scale weather regimes, a more controlled experiment would have been needed to diagnose the reason for the larger differences that occurred with the 2007-08 cases. Finally, a compositing technique was used to diagnose the differences in the spatial distribution of the forecasts. This technique implied westward displacement errors for NAM model forecasts in both sets of cases and in NCAR-WRF model forecasts for the 2007-08 cases. Generally, the results are encouraging because they imply that advantages in convection-allowing relative to convection-parameterizing simulations noted in recent studies are reflected in an objective neighborhood-based metric.