Probability of a segregating pattern in a sample of DNA sequences

Mutations that result in segregating sites (polymorphic sites) in a sample of DNA sequences can be classified into different types. A pattern of segregating sites is an array of the numbers of various types of mutations. Using an urn model, the probability of a pattern of segregating sites can be expressed as a recurrence equation and its value can be computed sequentially. Among those that can be computed by this method are the probability of obtaining k external mutations (mutations that occur in external branches of the genealogy of a sample), the probability of obtaining k internal mutations (mutations that occur in internal branches), the probability of obtaining k singletons (segregating sites at which one of the two segregating nucleotides is present in on ly one sequence), and the probability of obtaining k non-singletons. Two applications of the method are discussed. One is a maximum likelihood estimation of theta and another is a Bayesian statistical test of the hypothesis of neutral mutations. (C) 1998 Academic Press.