The effect of taxon sampling on estimating rate heterogeneity parameters of maximum-likelihood models

As maximum-likelihood approaches to the study of molecular systematics and evolution become both more flexible and more accessible, the importance of understanding the statistical properties of parameter estimation becomes critical. Using variation in NADH-2 sequences for 40 species of requiem sharks, we illustrate that estimates of rate heterogeneity parameters are highly sensitive to taxon sampling when the data are best explained by a mixed-distribution model of among-site rate variation (invariable sites plus gamma distribution [I+Gamma]). Using computer simulation, we attempt to differentiate two possible causes of this sensitivity. While the possibility of nonstationarity cannot be definitively rejected, our results suggest that sampling error alone provides an adequate explanation for the pattern of uncertainty observed in estimates from the real data. Furthermore, we illustrate that two parameters estimated under the I+Gamma model (the proportion of sites not free to change and the gamma distribution shape parameter) are highly correlated and that the likelihood surface across the rate heterogeneity parameter space can be poorly behaved when only a small number of sequences (taxa) are considered.