Imbalanced Segregation of Recombinant Haplotypes in Hybrid Populations Reveals Inter- and Intrachromosomal Dobzhansky-Muller Incompatibilities

Dobzhansky-Muller incompatibilities (DMIs) are a major component of reproductive isolation between species. DMIs imply negative epistasis and are exposed when two diverged populations hybridize. Mapping the locations of DMIs has largely relied on classical genetic mapping. Approaches to date are hampered by low power and the challenge of identifying DMI loci on the same chromosome, because strong initial linkage of parental haplotypes weakens statistical tests. Here, we propose new statistics to infer negative epistasis from haplotype frequencies in hybrid populations. When two divergent populations hybridize, the variance in heterozygosity at two loci decreases faster with time at DMI loci than at random pairs of loci. When two populations hybridize at near-even admixture proportions, the deviation of the observed variance from its expectation becomes negative for the DMI pair. This negative deviation enables us to detect intermediate to strong negative epistasis both within and between chromosomes. In practice, the detection window in hybrid populations depends on the demographic scenario, the recombination rate, and the strength of epistasis. When the initial proportion of the two parental populations is uneven, only strong DMIs can be detected with our method unless migration prevents parental haplotypes from being lost. We use the new statistics to infer candidate DMIs from three hybrid populations of swordtail fish. We identify numerous new DMI candidates, some of which are inferred to interact with several loci within and between chromosomes. Moreover, we discuss our results in the context of an expected enrichment in intrachromosomal over interchromosomal DMIs.