Benchmarking foundation models for splice site and exon annotation.

Recent foundation and deep learning models have brought a generational leap in improving the quality of genome annotation, particularly in identifying genes and their structural elements, including exons and splice sites. However, they are trained on reduced datasets that may not capture biological complexity, such as differences between coding versus non-coding, terminal versus internal, constitutive versus alternatively spliced, and transposable element (TE)-derived exons. We evaluate several foundation models for gene and splice site annotation, including the transformer-based SegmentNT, Enformer and Borzoi, coupled with a segmentation head for per-base resolution, and the CNN-based SpliceAI and AlphaGenome, along with a newly developed fine-tuned model, STEP2h, on different classes of gene elements as described above. We found that the performance of all methods is highest for the class of exons found in their training data class and decreases drastically for classes of exons poorly represented. In particular, performance is highest for protein-coding genes, coding exons, and constitutive exons, and decreases drastically by up to 2-4 fold for non-coding internal exons, terminal exons, and exons that undergo alternative splicing. Similarly, performance is impaired on LINE-1 and