What is a population? An empirical evaluation of some genetic methods for identifying the number of gene pools and their degree of connectivity

We review commonly used population definitions under both the ecological paradigm (which emphasizes demographic cohesion) and the evolutionary paradigm (which emphasizes reproductive cohesion) and find that none are truly operational. We suggest several quantitative criteria that might be used to determine when groups of individuals are different enough to be considered 'populations'. Units for these criteria are migration rate (m) for the ecological paradigm and migrants per generation (Nm) for the evolutionary paradigm. These criteria are then evaluated by applying analytical methods to simulated genetic data for a finite island model. Under the standard parameter set that includes L = 20 High mutation (microsatellite-like) loci and samples of S = 50 individuals from each of n = 4 subpopulations, power to detect departures from panmixia was very high (similar to 100%; P < 0.001) even with high gene flow (Nm = 25). A new method, comparing the number of correct population assignments with the random expectation, performed as well as a multilocus contingency test and warrants further consideration. Use of Low mutation (allozyme-like) markers reduced power more than did halving S or L. Under the standard parameter set, power to detect restricted gene flow below a certain level X (H-0: Nm < X) can also be high, provided that true Nm <= 0.5X. Developing the appropriate test criterion, however, requires assumptions about several key parameters that are difficult to estimate in most natural populations. Methods that cluster individuals without using a priori sampling information detected the true number of populations only under conditions of moderate or low gene flow (Nm <= 5), and power dropped sharply with smaller samples of loci and individuals. A simple algorithm based on a multilocus contingency test of allele frequencies in pairs of samples has high power to detect the true number of populations even with Nm = 25 but requires more rigorous statistical evaluation. The ecological paradigm remains challenging for evaluations using genetic markers, because the transition from demographic dependence to independence occurs in a region of high migration where genetic methods have relatively little power. Some recent theoretical developments and continued advances in computational power provide hope that this situation may change in the future.