The Princeton Guide to Evolution

IV.8

Evolution of Mating Systems: Outcrossing versus Selfing

Spencer C. H. Barrett

OUTLINE

1. Definitions and measurement

2. Variation in mating patterns

3. Evolution of self-fertilization

4. Mechanisms of selection

5. The problem of mixed mating

6. Evolutionary history

Mating systems vary enormously among groups of organisms. This has led to diverse definitions and approaches for investigating their evolution and maintenance. Animal mating systems are characterized by different patterns of parental investment in offspring and variation in the extent to which sexual selection shapes male and female traits (see chapter VII.4). A primary focus of most studies is determining the causes and consequences of variation in mate number for females and males. In contrast, in hermaphrodite organisms, particularly plants, the emphasis is largely on determining the incidence of cross- and self-fertilization and its fitness consequences. Most studies of mating-system evolution have been conducted on plants and largely concern the ecological and genetic mechanisms responsible for evolutionary transitions from outcrossing to predominant self-fertilization, and the extent to which mixed mating can be maintained as a stable strategy. Models of mating-system evolution involve the balance between the transmission advantage of alleles affecting the selfing rate and the reduced fitness of self-fertilized offspring because of inbreeding depression. Selfing provides reproductive assurance whenever outcrossing is limited by the availability of pollinators or mates and thus low density often plays an important role in mating-system transitions. Reconstruction of the evolutionary history of mating systems using phylogenies indicates that transitions to predominant selfing have occurred on numerous occasions, although selfing lineages are often short-lived because of the negative effects of selfing on the genome.

GLOSSARY

Floral Design and Display. The morphological features of flowers and inflorescences that influence pollen dispersal and mating patterns in flowering plants; floral design involves characteristics of individual flowers, including their size, structure, color, and the spatial and temporal presentation of female (pistil) and male (stamens) sex organs, and floral display concerns the number of open flowers on a plant and their arrangement within and among inflorescences.

Inbreeding Depression. The reduction in viability and/or fertility of inbred offspring in comparison with outbred offspring. It results primarily from the expression of deleterious recessive alleles in homozygous genotypes and is expressed most strongly when inbreeding occurs in primarily outcrossing populations; a key parameter (δ) in models of mating-system evolution.

Mating System. Broadly defined, the mode of transmission of genes from one generation to the next through sexual reproduction; however, in species with separate sexes, it largely concerns the quantity and quality of mates obtained by males and females. In contrast, in hermaphrodites, it is usually defined as the relative frequency of cross-fertilization (outcrossing) and self-fertilization (selfing) in a population. In this article the last definition is used.

Modes of Self-pollination. The various ways in which self-pollination occurs in flowering plants, distinguished primarily by reproductive expenditure, whether a pollen vector is involved, and timing, relative to cross-pollination. The evolution of selfing depends critically on the particular mode of self-pollination.

Pollen and Seed Discounting. Important reproductive parameters for determining whether selfing will evolve in flowering plants; pollen discounting is the loss in outcrossed siring success caused by self-pollination, whereas seed discounting is the reduction in outcrossed seed production caused by selfing, either because selfing preempts ovules that could have been outcrossed, or because self-fertilized seeds consume resources that could have been allocated to outcrossed seeds.

Pollen Limitation. The reduction in potential seed production that occurs when ovules remain unfertilized and too few embryos survive genetic death to compete for maternal resources; results from insufficient delivery of pollen quantity and quality to flowers. Persistent pollen limitation can result in the evolution of selfing.

Reproductive Assurance. An increase in seed production caused by selfing when conditions for outcrossing are unfavorable because of an absence of pollinators or mates; requires plants to be self-compatible and generally capable of autonomous self-pollination.

Self-compatible and Self-incompatible. The two conditions that determine whether fertile hermaphrodites have the potential to produce offspring from self-fertilization. In plants, self-compatibility is the ability to produce abundant seed following self-pollination, whereas in a self-incompatible plant, few, if any, seeds are produced by self-pollination. Self-incompatibility is the most common antiselfing mechanism in flowering plants.

1. DEFINITIONS AND MEASUREMENT

The mating system is a key life history trait because it determines the quantity and genetic quality of offspring produced by an individual and thus the individual’s reproductive fitness. One of the most striking features of organismal diversity is the existence of numerous mating strategies, even though all serve the same basic function: to promote parental reproductive success. Why do such diverse adaptations exist for an activity crucial for persistence of species of sexual organisms? Organismal variation in mating systems is determined, in part, by the biological features that characterize a particular taxon. For example, whether a group is sedentary or mobile and what type of sexual system it possesses (e.g., hermaphroditism versus separate sexes) influences how and with whom a plant or animal can mate. But although the types of mating systems are closely linked to the distinctive features of particular taxa, it is not uncommon to find quite different mating strategies among closely related species, especially in flowering plants. This indicates that mating systems are evolutionarily labile and can respond to natural selection; therefore, understanding mating-system diversity has been a major theme in evolutionary biology since Charles Darwin’s influential work on the topic.

Reproductive biology is replete with terms for what are seemingly similar phenomena. Before we begin, it is important to clarify what is meant by “mating system” and how it relates to other terms associated with sexual reproduction. This is important because the term has various definitions and different usages, particularly in the botanical versus zoological literature. The distinctions depend largely on whether biologists are working on species with separate sexes (dioecy) versus those studying hermaphrodite (cosexual) organisms. Terms that are commonly used in reproductive biology involve the relative importance of sexual versus asexual reproduction (reproductive system) (see chapter III.9), the occurrence of separate sexes versus hermaphroditism (sexual system), and within a population, who mates with whom and how often (mating system) (see chapter VII.4); however, workers studying hermaphroditic organisms usually define mating system more specifically as the relative frequency of cross-fertilization and self-fertilization, and this is the definition used here, because the main focus of this article is mating patterns in hermaphrodites. It is also important to note that the term breeding system is often used synonymously with mating system in the animal and plant literature but usually more broadly, to include mating behavior and parental care in animals, and the diverse reproductive traits that promote particular mating patterns in plants.

Animal biologists in the field of behavioral ecology classify mating systems largely on the basis of mate number per male and female and are particularly interested in determining the variance in reproductive success of the sexes and thus the scope of sexual selection. Here, the mating system is viewed as a behavioral strategy for obtaining mates, which also encompasses the nature of parental care. Typical classes of animal mating systems include monogamy, in which each sex mates with only one partner during its lifetime, polygyny, where females mate with a single partner but males are variable in mate number, and polyandry, in which the reverse pattern occurs. Information on the types of animal mating systems provides a powerful tool for predicting the degree and direction of sexual dimorphism, as Darwin originally pointed out. These topics are discussed in more detail in Section VII: Evolution of Behavior, Society, and Humans.

Darwin wrote extensively about animal mating systems and sexual selection, but he also published three books on the reproductive biology of plants. This body of work initiated contemporary studies of mating-system evolution in plants. Because of the predominantly hermaphroditic sexual systems of plants, Darwin was particularly interested in how they avoid self-fertilization and its harmful effects on progeny fitness (see chapter IV.6), which he demonstrated in numerous species through controlled pollinations and comparisons of the performance of self- and cross-fertilized progeny. He demonstrated that plants possess numerous structural contrivances (floral designs) that reduce the incidence of self-pollination and promote cross-pollination. Darwin also identified remarkable diversity in floral biology and mating systems among closely related species. This variation has enabled investigation of the ecological and genetic mechanisms responsible for evolutionary transitions in mating systems. In the following sections we will explore this topic further, focusing in particular on plants, about which the most is known, but also drawing parallels where possible with hermaphroditic animal groups.

Early efforts to understand mating patterns in hermaphrodites depended on inferences from observations of behavior and the morphology of reproductive organs. For example, in plants, the relative positions of female and male sexual organs along with observations of pollinator foraging provided clues to the likely occurrence of cross-versus self-fertilization; however, such inferences were often unreliable, and more importantly, they did not provide quantitative estimates of the types of mating events that occur. In the 1970s, this situation changed following demonstration of the utility of genetic markers for measuring mating parameters in populations. Since then, estimates of the proportion of offspring produced by selfing, s, or its complement, the outcrossing rate (t = 1 – s), have been obtained for a wide variety of species, providing unprecedented insights into the extent of mating-system variation. Estimates of t and s portray selfing and outcrossing through female function and provide population-level estimates of mating patterns. In addition, the degree of biparental inbreeding and the extent to which progeny are full sibs can also be estimated. More recent developments using hypervariable genetic markers such as microsatellites now enable measurements of mating patterns at the individual level and also estimates of outcrossed mating success through paternal function. The assessment of paternity and male siring success is important because it provides an opportunity to evaluate the importance of sexual selection in plant populations.

2. VARIATION IN MATING PATTERNS

The use of genetic markers for measuring mating patterns in populations has uncovered considerable variation among plant species, and this diversity often provides the template for evolutionary transitions. Most observed variation in outcrossing rate occurs in self-compatible species in which environmental and demographic factors can play an important role in determining the amount of outcrossed pollen delivered to flowers by pollen vectors (animals, wind, and water). Species with strong self-incompatibility systems, or those that are highly selfing (autogamous), exhibit much less variation in outcrossing rate; because of this, they have tended to be underrepresented in mating-system surveys. Nevertheless, a picture is emerging of the distribution of outcrossing rates among seed plants, and this has stimulated much interest and controversy because of its relevance to theoretical models of mating-system evolution.

Marker-based estimates of outcrossing rate from 345 species representing 78 families by Goodwillie and colleagues (2005) demonstrated near-continuous variation from highly selfing to obligately outcrossed species; however, among this sample, 42 percent of the species could be classified as exhibiting mixed mating (a mixture of both outcrossing and selfing) in which t ranged from 0.2 to 0.8. As we will see later, the occurrence of mixed mating presents an important challenge for theoretical models of mating-system evolution. A particularly striking pattern is the finding that the distribution of outcrossing rate differs significantly between animal-pollinated (biotic) species and those pollinated by wind or water (abiotic). Biotically pollinated species are almost twice as likely to exhibit mixed mating compared to those that are abiotically pollinated. Why this difference occurs is at present unclear but may be associated with stronger selection in abiotically pollinated plants for mechanisms that reduce the incidence of self-pollination owing to the huge quantities of pollen typically produced, particularly in wind-pollinated species. Regardless of the mechanisms responsible, this difference does point to the important influence of pollination systems on mating patterns.

There are much fewer data available on mating-system variation in hermaphroditic animals. It has been estimated that about 5 percent of animal species are hermaphroditic, although this value is much higher, about one-third, if arthropods are excluded. Hermaphroditism occurs in the majority of animal phyla and is especially common in Annelida (ringed worms), Cnidaria (particularly corals), Mollusca (particularly the gastropods: snails and slugs), Nematoda (roundworms), and Playhelminthes (flatworms), where most mating-system work has been conducted. Estimates of outcrossing rate for animals are based largely on indirect measures from population structure data on the inbreeding coefficient (F_IS). This approach assumes that populations are at inbreeding equilibrium and that selfing is the sole of source of inbreeding. Despite this limitation, the available data are not dissimilar to those reported for plants with a near-continuous distribution, from complete outcrossing to predominant selfing, and a significant representation of species with mixed mating.

Figure 1. The distribution of outcrossing rates in plants (n = 342 species) and animals (n = 142 species); plant data from Goodwillie et al. (2005), animal data from Jarne and Auld (2006). The outcrossing rate (t) for each species is plotted. (Modified from Jarne and Auld [2006] with permission.)

Species-level surveys of the distribution of outcrossing rates, like those illustrated in figure 1, typically use average values across populations, masking any intraspecific variation that occurs. This variation is important because it provides opportunities to determine the proximate mechanisms governing mating-system variation and can provide important clues to the causes of mating-system transitions. For example, the annual insect-pollinated tropical aquatic plant Eichhornia paniculata exhibits a wide range of outcrossing rates, from near zero to complete outcrossing. Populations in Brazil are largely outcrossing, whereas those that have colonized Caribbean islands are highly selfing. This variation results from an evolutionary transition in mating system and is governed by the interaction of stochastic forces (genetic drift and founder effects) and natural selection favoring selfing variants when pollinator service is unreliable. Small population size plays an important role in this mating-system transition. Investigation of intraspecific variation in mating patterns can reveal much about the demographic and genetic factors responsible for the evolution of selfing.

3. EVOLUTION OF SELF-FERTILIZATION

The evolution of predominant self-fertilization (autogamy) from high levels of outcrossing is the most frequent mating-system transition in hermaphroditic organisms. No accurate estimates have been made of the total number of origins of selfing, but it is likely that in large groups, such as the angiosperms, this transition has occurred many hundreds of times. The evolution of selfing from outcrossing has attracted the attention of evolutionary biologists because it has diverse biological consequences. First, the effects of selfing on relative fitness through inbreeding depression are well established, and as we shall see, these effects play an important role in determining the dynamics of mating-system evolution. Second, the shift to selfing profoundly influences many aspects of sex allocation and life-history evolution. Third, selfing individuals have the ability to establish colonies at low density or following long-distance dispersal, a phenomenon known as Baker’s law (after the plant evolutionist Herbert G. Baker), and this capability has significant ecological, demographic, and biogeographic implications. Finally, high selfing rates have profound genetic and evolutionary consequences, influencing population genetic structure, evolutionary rates, and patterns of evolutionary diversification.

Evolutionary transitions to selfing decrease genome-wide diversity compared with that of outcrossing populations. Homozygosity increases with the selfing rate, causing reductions in effective population size (N_e) up to twofold with complete selfing; moreover, because of higher linkage disequilibrium in selfing populations, other processes in the genome, including selective sweeps and background selection, further reduce genetic variation. These processes can also be influenced by the demographic and life history characteristics of species, especially genetic bottlenecks, when single individuals found colonies. Populations of many selfing species are characterized by frequent colonization-extinction cycles, and these demographic processes can lead to strong population subdivision. Thus selfing species are characterized by limited genetic variation within populations but a high degree of differentiation among populations, a pattern that is the reverse of what is normally found in outcrossing species.

Most groups have striking ecological and life history correlates of mating-system variation, indicating that not all species are likely to experience selection for increased selfing rates. In flowering plants, the distribution of mating systems is correlated with longevity and size; long-lived woody plants are more outcrossing than herbaceous perennials, and the highest selfing rates occur in short-lived annual species. Variation in plant life history is associated with important genetic and reproductive consequences that can explain these correlations. Long-lived species tend to maintain higher genetic loads, perhaps owing to higher per generation mutation rates because of an increased number of somatic mutations, resulting in strong selection against inbred offspring. This limits opportunities for the spread of genetic modifiers influencing the selfing rate and probably explains why there are few reports of highly selfing trees. Long-lived species also tend to be large in size, growing either vertically, as occurs in trees, or laterally, as in highly clonal plants. One consequence of large size is that during reproduction, floral displays also tend to be very large, resulting in extensive pollen transfer between flowers on the same individual, resulting in a mode of self-pollination known as geitonogamy, with genetic consequences identical to those for intraflower selfing. Because of strong inbreeding depression, antiselfing mechanisms such as self-incompatibility and dioecy are especially well developed in long-lived trees and clonal plants.

By contrast, the uncertain conditions typical of ephemeral habitats occupied by species with very short life cycles favor a mating system that provides reproductive assurance. This explains why selfing is very common in annual plants, which have only a single opportunity for mating. Annuals commonly possess well-developed powers of seed dispersal and frequently establish colonies at low density, a demographic context that selects against obligate outcrossing and favors individuals with a capacity for self-fertilization. Not surprisingly, many invasive annual plants and hermaphroditic animals are highly selfing.

These associations between mating systems and life history are particularly evident in herbaceous flowering plant families in which perennials tend to be outcrossing and annuals are more likely selfers. These patterns are well exemplified in the Polemoniaceae (Phlox family), which exhibits considerable ecological versatility, particularly in western North America, where species occupy montane, forest, and desert environments. Reconstructions of the evolutionary history of the life cycle (annual versus perennial) and mating system (outcrossing versus selfing) using molecular phylogenies of the family indicate repeated transitions from outcrossing to selfing, either coincidentally with or following transitions from perennial to annual life history. The lability of reproductive mode and life history in many plant families suggests that phylogenetic constraints rarely limit opportunities when ecological conditions require evolutionary shifts in mating system.

Hermaphroditic animals capable of selfing are typically associated with low population densities and a sessile, sluggish, or parasitic lifestyle. Under these demographic conditions, opportunities for encountering mating partners are much more limited than for mobile animals with separate sexes, and this favors self-fertilization. Indeed, comparative evidence has been found among multicellular eukaryotic organisms for the correlated evolution of the sexual system with the mode of locomotion for searching for a mate. Hermaphroditic groups capable of selfing are more likely to have low mate searching efficiency than species with separate sexes. Evidence has also been found among hermaphroditic snails that individuals can respond to variation in mate availability by delaying the time before selfing occurs. This indicates that mate availability is a key target of natural selection on the mating system in animal populations.

4. MECHANISMS OF SELECTION

The most appropriate theoretical framework for understanding the selective mechanisms governing mating-system evolution is to use a population genetics approach that considers the conditions that might favor the spread of a variant capable of self-fertilization in an obligately outcrossing population. The mathematical geneticist Ronald Fisher first pointed out that all else being equal, a gene causing self-fertilization increases in frequency in each generation, because, on average, selfers generally contribute more gene copies to the next generation than outcrossers. In plants this arises because selfers are the maternal and paternal parent to the self-fertilized seed they produce, and their pollen can also participate in outcrossing to other plants; this gene transmission advantage can be seen by considering the scenario shown in table 1.

Table 1. Average Gene Contribution

	Outcrosser	Selfer
Ovule parent	1	1
Pollen parent	1	2
Total	2	3

Not all species are selfing, however; indeed, many more are predominantly outcrossing, indicating that other factors must play a role in determining the direction of mating-system evolution. Today, it is recognized that three principal factors serve to limit the spread of selfing genes in outcrossing populations: inbreeding depression (δ), pollen discounting, and seed discounting, and each factor has the potential to reduce the gene transmission advantage of a selfing variant. Inbreeding depression causes reduced fitness of self-fertilized offspring. If they survive and reproduce only half as well as outcrossed offspring, the advantage of selfing disappears. Thus, when δ > 0.5, outcrossing is favored, but when δ < 0.5, selfing should spread. Fisher’s model also does not consider whether self-pollination reduces the amount of pollen available for outcrossed siring success. If there is a reduction in pollen available for outcrossing, a selfer’s fitness through male function is reduced proportionately (pollen discounting). Similarly, seed discounting, the reduction in outcrossed seed production caused by selfing, also reduces the transmission advantage of a selfing variant. Current work is attempting to measure these three parameters in an effort to test models and predict the course of mating-system evolution. Measurements of inbreeding depression are relatively straightforward, but pollen and seed discounting are more challenging to estimate, and the extent to which they influence selection on the selfing rate in plant populations is at present unclear.

The other general explanation for the evolution of selfing involves the advantage of selfing individuals over outcrossing individuals when pollinators or mates are scarce, such as commonly occurs under low density. This is referred to as the reproductive assurance hypothesis and was originally suggested by Charles Darwin and the German naturalist Hermann Müller. The reproductive assurance hypothesis differs from Fisher’s automatic selection hypothesis because it predicts increased seed production in selfers compared to outcrossers and requires that self-compatible plants have the ability to self-pollinate autonomously in the absence of a pollinator. Considerable biogeographical evidence indicates that selfing populations occupy range margins, ecologically marginal sites, or areas with reduced pollinator densities where outcrossers are absent, all circumstances predicted by the reproductive assurance hypothesis; however, surprisingly few field studies have provided experimental evidence in support of this hypothesis, despite the widespread occurrence of pollen limitation of seed set in animal-pollinated species. It seems likely that both automatic selection and reproductive assurance will turn out to be involved in many transitions from outcrossing to selfing in flowering plants, with demographic context determining their relative importance.

5. THE PROBLEM OF MIXED MATING

Most genetic models of mating-system evolution, based on the balance between the transmission advantage of selfing and the costs of inbreeding depression, result in populations in which either predominant selfing or outcrossing is an alternative stable state. Thus, theory predicts a bimodal distribution of outcrossing rates resulting from selection for the maintenance of outcrossing in historically large populations with inbreeding depression greater than 0.5, and selection for selfing when demographic factors such as population bottlenecks reduce inbreeding depression to less than 0.5, and partial selfing purges deleterious recessive alleles; however, as we have seen, surveys of outcrossing rates in plants and animals indicate that many species exhibit mixed mating. What accounts for this apparent discrepancy and is there evidence that mixed mating can be a stable mating strategy?

Several explanations help to reconcile the mismatch between theory and empirical evidence on the distribution of mating systems. First, existing estimates of the distribution of outcrossing rates are likely biased in terms of taxonomic representation (many estimates are concentrated in some well-studied taxa, e.g., pines, gum trees, and grasses) and, as discussed earlier, an underrepresentation of the two ends of the distribution, thus inflating the frequency of species with mixed mating. Second, theoretical models predict the equilibrium mating system in a population, and it is probable that some species are not at equilibrium when sampled and, given sufficient time, could be driven to predominant selfing or outcrossing. Third, in many animal-pollinated species, the selfing component of mixed mating is likely a nonadaptive cost that plants pay by having large floral displays to attract pollinators. Geitonogamy occurs because pollinators transfer pollen between flowers on a plant; this provides little benefit to fitness because of strong inbreeding depression and pollen discounting. Fourth, where pollinators exhibit spatial and temporal fluctuations in density, mixed mating can arise by modes of self-pollination that provide reproductive assurance. For example, delayed selfing arises when flowers self-pollinate after opportunities from cross-pollination have passed, and models indicate that this will always be adaptive. Finally, more elaborate theory, integrating more ecological and genetic details, has been able to show stable mixed mating, albeit often under restricted conditions. Nevertheless, the adaptive significance of mixed mating remains problematic for most species, and the search for a general explanation continues to be elusive, probably because the phenomenon has diverse causes.

6. EVOLUTIONARY HISTORY

The recent availability of phylogenetic information for groups with variation in mating systems has enabled the reconstruction of the evolutionary history of outcrossing and selfing, leading to several generalizations: (1) the transition from outcrossing to predominant self-fertilization often occurs repeatedly within lineages and is usually unidirectional; (2) selfing species most often occur at the tips of phylogenies and are therefore relatively young in age; (3) species-rich groups and higher taxa are usually composed of predominantly outcrossing species. These findings support an early view proposed by the plant evolutionist G. Ledyard Stebbins that the evolution of selfing represents an evolutionary dead end. Why should this be so, given that selfing clearly has short-term advantages in some ecological contexts?

High rates of self-fertilization are associated with negative genomic consequences that make it unlikely that selfing species will persist over long timescales. These effects include a reduction in the amount of adaptive genetic variation in populations, limiting their capacity to adapt to environmental change, low recombination and a reduced efficacy of natural selection, lowering opportunities for purging deleterious mutations as well as for fixing adaptive mutations. Thus, weak purifying selection and/or the operation of Muller’s ratchet (the irreversible accumulation of deleterious mutations) should lead over time to mutational decay and the ultimate extinction of selfing lineages compared with those that habitually outcross.

Evidence supporting the ephemerality of selfing lineages comes from comparative work on the tomato family, Solanaceae, where the distribution of self-incompatibility (SI) and self-compatibility (SC), traits that directly influence outcrossing and selfing, respectively, are associated with contrasting rates of species diversification. Many independent transitions from SI to SC occur in this group, but significantly, SI is never regained, a pattern commonly observed in other plant families. This raises the question of how SI can be maintained over long timescales if it is being continuously broken down because of selection for selfing. The answer lies in the higher extinction rates of SC lineages compared to SI lineages, counterbalancing the frequent loss of SI. Transitions from outcrossing to selfing are driven ultimately by short-term ecological and genetic mechanisms in local populations; however, they can also have long-term macroevolutionary consequences for patterns of biodiversity, because mating systems influence character evolution and species diversification.