Matrix games between full siblings in Mendelian populations.

We develop a model that integrates evolutionary matrix game theory with Mendelian genetics. Within this framework, we define the genotype dynamics that describes how the frequencies of genotypes change in sexual diploid populations. We show that our formal definition of evolutionary stability for genotype distributions implies the stability of the corresponding interior equilibrium point in the genotype dynamics. We apply our findings to a model of familial selection, where the survival rates of siblings in monogamous families are determined by a matrix game between them. According to Mendelian inheritance, the behaviour associated with each genotype is uniquely determined by an autosomal (recessive-dominant or intermediate) allele pair. We provide general conditions for the evolutionary stability of homozygote populations. We find that the payoff matrix and the genotype-phenotype map together determine this stability. In numerical examples we consider the prisoner's dilemma between siblings. Based on the evolutionary stability of the pure cooperator and defector states, we provide a potential classification of the genotype dynamics. We distinguish between two cases: one in which the total survival rate is higher in cooperator-cooperator interactions ("coordinated" case), and another in which it is higher in cooperator-defector interactions ("anti-coordinated" case). In the coordinated case, global stability of cooperator homozygote population is possible but not necessary, since bistability, stable coexistence of cooperators and defectors, and even global stability of the defector homozygote state are all possible, depending on the interaction between the phenotypic payoff matrix and the genotype-phenotype mapping. In the anti-coordinated case, the cooperator homozygote population cannot be stable. Thus, similarly to the group selection theory, the welfare of the family (the sum of the survival rates of siblings) governs the emergence of cooperative behavior among family members. Finally, in the case of the donation game, the classical Hamilton's rule determines whether the homozygous cooperator or the homozygous defector population is stable; bistability or stable coexistence are impossible.