A residual-learning MMSE neural detector for 6G MIMO-OTFS systems under diverse channel conditions.

Orthogonal time-frequency space modulation combined with multiple-input multiple-output transmission (MIMO-OTFS) has emerged as a strong waveform candidate for sixth-generation (6G) wireless networks because of its robustness against high mobility and doubly selective channels. However, reliable signal detection in large-scale MIMO-OTFS systems remains challenging owing to severe delay-Doppler coupling and channel state information (CSI), particularly under Rayleigh and Rician fading conditions. This paper proposes a residual-learning minimum mean square error neural detector (RL-MMSE-ND) to address these challenges under both perfect and imperfect CSI, including scenarios with up to a 20% channel estimation error. The proposed detector integrates a conventional minimum mean square error (MMSE) front-end with a lightweight residual-learning neural network that learns only the residual interference caused by CSI mismatch and delay-Doppler effects, thereby preserving MMSE stability while introducing minimal learning overhead. Extensive simulations were conducted for representative large-scale MIMO-OTFS configurations to evaluate the bit error rate (BER) versus signal-to-noise ratio (SNR), BER versus delay spread, power spectral density (PSD) characteristics, inference latency, and training convergence. Numerical results demonstrate that the proposed RL-MMSE-ND achieves 10-13 dB SNR gain at a BER of 10