Morphological and molecular evidence for a stepwise evolutionary transition from teeth to baleen in mysticete whales

The origin of baleen in mysticete whales represents a major transition in the phylogenetic history of Cetacea. This key specialization, a keratinous sieve that enables filter-feeding, permitted exploitation of a new ecological niche and heralded the evolution of modern baleen-bearing whales, the largest animals on Earth. To date, all formally described mysticete fossils conform to two types: toothed species from Oligocene-age rocks (similar to 24 to 34 million years old) and toothless species that presumably utilized baleen to feed (Recent to similar to 30 million years old). Here, we show that several Oligocene toothed mysticetes have nutrient foramina and associated sulci on the lateral portions of their palates, homologous structures in extant mysticetes house vessels that nourish baleen. The simultaneous occurrence of teeth and nutrient foramina implies that both teeth and baleen were present in these early mysticetes. Phylogenetic analyses of a supermatrix that includes extinct taxa and new data for 11 nuclear genes consistently resolve relationships at the base of Mysticeti. The combined data set of 27,340 characters supports a stepwise transition from a toothed ancestor, to a mosaic intermediate with both teeth and baleen, to modern baleen whales that lack an adult dentition but retain developmental and genetic evidence of their ancestral toothed heritage. Comparative sequence data for ENAM (enamelin) and AMBN (ameloblastin) indicate that enamel-specific loci are present in Mysticeti but have degraded to pseudogenes in this group. The dramatic transformation in mysticete feeding anatomy documents an apparently rare, stepwisemode of evolution in which a composite phenotype bridged the gap between primitive and derived morphologies; a combination of fossil and molecular evidence provides a multifaceted record of this macroevolutionary pattern.