A critique of present procedures used to compare performance of rainfall-runoff models

The purpose of this paper is to stimulate debate on how to compare models of watershed behavior. It is argued that procedures commonly used at present to compare the performance of rainfall-runoff models are unsatisfactory for several reasons, but principally because they provide no measure of the uncertainty (as measured by an estimate of residual variation between experimental units) in differences between measures of model performance. It is also concluded that the present procedures, in which each model is usually calibrated and also validated by persons most familiar with its use, do not provide a sound basis for recommending any particular model for use by the hydrological community at large. Whilst the principles of good experimental design (replication, randomization) are widely practised in other fields of applied science, they are not yet widely practised in hydrology (and other geophysical sciences where models are essential tools). It is argued that these principles can and should be applied where experiments are designed to compare the performance of hydrological models. Designs are proposed in which different periods of record from within each watershed are used to calibrate (or validate) each of the models being compared; subject to the random allocation of models to test periods, the designs provide a valid measure of the uncertainty in measures of model performance. The basic design can be extended to explore whether models perform differently on different types of watershed; it is also shown how it can be adapted, through the use of "balanced incomplete blocks", to avoid the need to calibrate (and validate) every model on every watershed. (C) 2008 Elsevier B.V. All rights reserved.