The use of protocols to select exponential smoothing procedures: A reconsideration of forecasting competitions

Forecasting competitions have evaluated the accuracy of extrapolative methods unselectively; that is, by applying each method to an entire batch of time series without asking whether the method is appropriate for each individual series. For example, simple exponential smoothing was applied to trended as well as untrended time series. In consequence, the calculated error measures do not unambiguously portray the accuracy one can expect in using a given procedure selectively. In this study, we employ three different protocols for selecting appropriate exponential smoothing procedures. Then, based on 132 time series from previous forecasting competitions, we compare forecasting accuracy in cases for which the procedures are designated (by each protocol) as appropriate with accuracy achieved in applications designated as inappropriate. The selection protocols are a variance analysis proposed by Gardner and McKenzie, a set of visual rules adapted from Collopy and Armstrong's rule-based forecasting system, and a method-switching procedure developed by Goodrich. We found the selection protocols to be effective in selecting appropriate applications of strong-trend (linear and exponential trend) procedures. For example, when the Holt (linear trend) procedure is applied selectively; its forecasting accuracy is much superior to that suggested by the forecasting competitions. On the other hand, the protocols are not effectively distinguishing appropriate from inappropriate applications of damped and no-trend procedures. Further, our results reveal considerable disagreement among the protocols themselves, making it apparent that much work remains to be done to improve the state of the art in forecasting-method selection.