Deep residual network enhanced with multilevel residual-of-residual for automatic classification of radio signals for 5G and beyond systems.

Automatic Modulation Classification (AMC) plays a critical role in the design of intelligent receivers for next-generation wireless systems, particularly in the context of 5G and beyond networks characterized by diverse multicarrier waveform technologies. This paper proposes a novel AMC framework based on a Deep Residual Network (DRN) architecture enhanced with multilevel Residual-of-Residual (RoR) connections, specifically designed to classify advanced modulation formats across a wide spectrum of 5G candidate waveforms, namely Orthogonal Frequency Division Multiplexing (OFDM), Filtered-OFDM (FOFDM), Filter Bank Multi-Carrier (FBMC), Universal Filtered Multi-Carrier (UFMC), and Weighted Overlap-and-Add OFDM (WOLA), modulated using both 16-QAM and 64-QAM schemes. To the best of our knowledge, this is the first work applying DRN enhanced with multilevel RoR (DRN+RoR) specifically to OFDM, FOFDM, FBMC, UFMC, and WOLA with 16/64-QAM. The proposed architecture exploits the deep hierarchical learning capabilities of DRN+RoR to extract highly discriminative features, while employing Sequential Floating Forward Selection (SFFS) for feature optimization and dimensionality reduction. Extensive simulations are conducted under realistic wireless channel conditions, including time- and frequency-selective fading. Evaluation metrics such as classification accuracy, recall, precision, and F1-score demonstrate the superior performance of the proposed method over state-of-the-art machine learning baselines. In particular, our approach achieves higher robustness and classification accuracy across all tested SNRs and waveform types, outperforming recent deep learning-based baselines. We also assess the performance of the proposed algorithm under diverse 5G channel conditions, specifically TDL-A, TDL-B, and TDL-C models. The evaluation results indicate that our proposal maintains robust classification accuracy and strong generalization across all tested channel scenarios. These findings underscore the effectiveness of DRN+RoR as a robust and scalable solution for modulation recognition, offering substantial improvements in adaptive signal processing and enabling more reliable AMC in dynamically evolving 5G and beyond communication environments.