Distributions of statistics used for the comparison of models of sequence evolution in phylogenetics

Asymptotic statistical theory suggests that when two nested models are compared by a likelihood ratio test, a chi(2) distribution, with number of degrees of freedom equal to the difference in numbers of free parameters of the two models, can be used for significance testing. This asymptotic result has been assumed to apply in phylogenetics with the support of only a few studies. In this paper, 12 comparisons among a selection of commonly used models of nucleotide substitution were examined to see whether this assumption is reasonable. The true distributions of likelihood ratio statistics were estimated by computer simulation and compared with the appropriate chi(2) distributions. It was found that chi 2 distributions are adequate for significance testing in the comparison of models differing by parameters describing transition/transversion bias and/or unequal base frequencies when these parameters have been estimated by maximum likelihood. The chi(2) distribution was, however, found to be significantly different from the true distributions in the comparison of models differing by parameters describing rate variation across sites (estimated by maximum likelihood) or unequal base frequencies (estimated as the observed base frequencies in an alignment). These last findings may have important consequences for real-model comparisons and for the construction of increasingly complex and realistic models of nucleotide sequence evolution.