Statistics for microsatellite variation based on coalescence

The stepwise mutation model, which was at one time chiefly of interest in studying the evolution of protein charge-states, has recently undergone a resurgence of interest with the new popularity of microsatellites as phylogenetic markers. In this paper we describe a method which makes it possible to transfer many population genetics results from the standard infinite sites model to the stepwise mutation model. We study in detail the properties of pairwise differences in microsatellite repeat number between randomly chosen alleles. We show that the problem of finding the expected squared distance between two individuals and finding the variance of the squared distance can be reduced for a wide range of population models to finding the mean and mean square coalescence times. In many cases the distributions of coalescence times have already been studied For infinite site problems. In this study we show how to calculate these quantities for several population models. We also calculate the variance in mean squared pairwise distance (an estimator of mutation rate x population size) for samples of arbitrary size and show that this variance does not approach zero as the sample size increases. We can also use our method to study alleles at linked microsatellite loci. We suggest a metric which quantifies the level of association between loci-effectively a measure of linkage disequilibrium. It is shown that there can be linkage disequilibrium between partially linked loci at mutation-drift equilibrium. (C) 1996 Academic Press, Inc.