Learnable digital signal processing: a new benchmark of linearity compensation for optical fiber communications.

The surge in interest regarding the next generation of optical fiber transmission has stimulated the development of digital signal processing (DSP) schemes that are highly cost-effective with both high performance and low complexity. As benchmarks for nonlinear compensation methods, however, traditional DSP designed with block-by-block modules for linear compensations, could exhibit residual linear effects after compensation, limiting the nonlinear compensation performance. Here we propose a high-efficient design thought for DSP based on the learnable perspectivity, called learnable DSP (LDSP). LDSP reuses the traditional DSP modules, regarding the whole DSP as a deep learning framework and optimizing the DSP parameters adaptively based on backpropagation algorithm from a global scale. This method not only establishes new standards in linear DSP performance but also serves as a critical benchmark for nonlinear DSP designs. In comparison to traditional DSP with hyperparameter optimization, a notable enhancement of approximately 1.21 dB in the Q factor for 400 Gb/s signal after 1600 km fiber transmission is experimentally demonstrated by combining LDSP and perturbation-based nonlinear compensation algorithm. Benefiting from the learnable model, LDSP can learn the best configuration adaptively with low complexity, reducing dependence on initial parameters. The proposed approach implements a symbol-rate DSP with a small bit error rate (BER) cost in exchange for a 48% complexity reduction compared to the conventional 2 samples/symbol processing. We believe that LDSP represents a new and highly efficient paradigm for DSP design, which is poised to attract considerable attention across various domains of optical communications.