Incompatibility between two major innovations shaped the diversification of fish feeding mechanisms.

Innovations often shape the trajectory of macroevolution, yet their effects are usually considered independently, thus ignoring the functional and evolutionary interactions between them. Two innovations that have underpinned the ecological and evolutionary success of ray-finned fishes (Actinopterygii) are large teeth and highly protrusible jaws, which independently expanded the diversity of prey capture strategies. Here, we explore the functional relationship between these innovations across actinopterygians using high-speed videography and phylogenetic comparative methods. We find that these two innovations are functionally and evolutionarily incompatible because there is an overarching tradeoff between jaw protrusion and tooth size. Having large teeth decreases the kinematic diversity of prey capture by restricting species to overtake prey predominantly by swimming, while highly protrusible jaws are only found in species with small teeth. The space within tooth-bearing bones may impose this constraint, by limiting the maximum tooth size of species with gracile jaws adapted for high mobility and jaw protrusion. Nevertheless, some species break this constraint on tooth size through novel adaptations that accommodate exceptionally large teeth, unlocking new feeding modes which may have expanded the nature of aquatic feeding and influenced the ecosystems themselves. Although both high jaw protrusion and large teeth separately expanded prey capture strategies in fishes, they are generally not found in combination and are evolutionarily incompatible.