Zero-shot de novo peptide sequencing with open posttranslational modification discovery.

De novo peptide sequencing directly infers sequences from mass spectrometry data without relying on protein databases. Although recent deep learning models can also identify posttranslational modifications (PTMs), they require labeled training data for this task. Here we introduce rotary positional embedding-enhanced de novo sequencing algorithm (RNovA), a transformer-based de novo sequencing algorithm enhanced with relative positional embeddings and a reinforcement-learning-style sequential decision framework. RNovA enables open PTM discovery in a zero-shot setting-without retraining or a predefined list of candidate residues-while maintaining state-of-the-art performance on standard benchmarks. Demonstrating this capability, we successfully identified peptides modified by kynurenine-an uncommon and biologically relevant PTM-in clinical samples from patients with RA and validated this discovery with synthetically synthesized reference peptides. Furthermore, we demonstrated open de novo PTM discovery by analyzing the bacterial strain A1232E, which lacks a reference proteome, and detected an unannotated glutamic acid modification. RNovA enables exploration of previously inaccessible regions of the proteome, including peptides with unexpected or unannotated modifications.