Fiber Nonlinearity Compensation Method Based on Stepwise Bit Estimation Using Low Complexity CNN-BiGRU-Attention

Objective Fiber optic nonlinear effects, primarily encompassing self-phase modulation, cross-phase modulation, and four-wave mixing, represent a fundamental technical bottleneck that constrains performance enhancements in contemporary high-speed, high-capacity optical communication systems. Driven by emerging technologies, global data traffic is experiencing exponential growth, creating an urgent demand for higher transmission rates and greater spectral efficiency. This demand invariably pushes optical systems to operate in regimes where nonlinear effects are more pronounced and detrimental. Deep learning has been identified as a potent tool for mitigating such impairments, owing to its powerful nonlinear mapping capabilities. However, a major challenge is the prohibitively high computational complexity associated with advanced neural network-based equalizers, which poses a significant obstacle to their practical implementation and commercial viability despite their superior performance. Achieving an optimal trade-off between compensation performance and computational overhead is therefore a central challenge in this field. To address this issue, this paper proposes and rigorously validates a novel, low-complexity nonlinear compensation scheme. The core of this research is a hybrid neural network architecture that synergistically integrates a convolutional neural network (CNN), a bidirectional gated recurrent unit (BiGRU), and an attention mechanism, complemented by a stepwise bit estimation (SBE) output strategy. The principal objective of this study is to demonstrate that this multi-component, collaborative design can effectively counteract fiber nonlinear impairments while substantially reducing computational complexity relative to other state-of-the-art neural network models, thereby paving a viable path for the practical application of deep learning in next-generation optical networks. Methods The proposed compensation framework consists of a structured deep learning pipeline. The input signal sequence, distorted by fiber transmission, is first processed by a CNN layer that extracts local and abstract features using multiple convolutional filters. These features capture essential nonlinear patterns and prepare the data for temporal modeling. Subsequently, the extracted features are input into a BiGRU layer, which processes the sequence bidirectionally. The GRU structure includes reset and update gates that control information flow and update hidden states, effectively learning long-range dependencies without suffering from vanishing gradients. The BiGRU layer leverages both past and future context to capture the dynamic characteristics of symbol-level distortions caused by dispersion and nonlinear interaction. Following the BiGRU layer, an attention mechanism is employed to dynamically weigh the relevance of each hidden state in the sequence. This selective focus allows the model to emphasize the most informative parts of the signal while suppressing irrelevant or noisy segments. The attention scores are normalized via softmax and used to generate a context vector by weighted summation of hidden states. This context vector is then passed through fully connected layers to synthesize a comprehensive representation for final prediction.At the output, the SBE strategy is adopted. Instead of treating symbol recovery as a multi-class classification task, the model decomposes the prediction of high-order modulation symbols into several binary classification sub-tasks, each corresponding to one bit. Each bit is predicted using a sigmoid neuron and thresholded independently. This output design aligns directly with the bit error rate (BER), the key performance metric in communication systems. Results and Discussions To comprehensively validate the efficacy of the proposed scheme, we conduct systematic evaluations on a coherent optical communication simulation platform. The platform models the transmission of a 32 GBaud polarization-division multiplexed 16-ary quadrature amplitude modulation (PDM-16-QAM) signal over a 1200 km standard single-mode fiber link. Our analysis encompasses both single-channel and five-channel wavelength-division multiplexing (WDM) scenarios, and the simulations incorporate realistic hardware impairments, including laser phase noise and frequency offset. Initially, to verify the effectiveness of the model's core components, we compare the performance of the integrated CNN-BiGRU network against a baseline BiGRU network. The results unequivocally demonstrate that the inclusion of the CNN layer yields a substantial performance improvement. In the single-channel system operating at a launch power of 0 dBm, the SBE-based CNN-BiGRU network achieves a quality (Q) factor improvement of 0.77 dB over its counterpart lacking the CNN, which confirms the CNN's proficiency in extracting nonlinearity-relevant features. The central contribution of this research stems from the integration and analysis of the Attention mechanism. An investigation into the attention weight distribution of the trained network reveals that the model autonomously learns to focus its attention predominantly on the central symbol and its immediate neighbors, while paying significantly less attention to more distant symbols. This observation aligns with the physical nature of fiber nonlinear effects and provides a strong rationale for our subsequent model optimization efforts. Informed by this finding, we implement a model pruning strategy, wherein the input sequences and hidden states processed by the BiGRU are truncated to include only the temporal regions identified as important by the attention mechanism. We then compare the performance of this pruned, low-complexity CNN-BiGRU-Attention network against the unpruned CNN-BiGRU network. The results show that, across both single-channel and WDM systems, the attention-based model achieves a significant reduction in computational complexity at the cost of only a minor Q-factor penalty ranging from 0.1 to 0.4 dB. This outcome highlights the model's excellent balance between performance and efficiency. Finally, we perform a quantitative analysis of the models' computational complexity. The analysis shows that our proposed CNN-BiGRU-Attention architecture exhibits substantially lower complexity than the comparative models across all tested output strategies. When compared to the corresponding CNN-BiGRU network, the introduction of the attention mechanism reduces the complexity by 43%. Moreover, even when compared to the baseline BiGRU network, our overall optimized architecture achieves a 36% reduction in complexity. These figures provide robust evidence of the attention mechanism's exceptional effectiveness in managing the computational load of the neural network. Conclusions This paper proposes and systematically validates a deep learning-based compensation method for fiber nonlinear impairments, built upon a CNN-BiGRU-Attention architecture with a stepwise bit estimation strategy. The method leverages the powerful local feature extraction of CNNs, the precise long-range dependency modeling of BiGRUs, and the intelligent, dynamic resource allocation of an attention mechanism to create an efficient model for nonlinear impairment compensation. The use of the SBE strategy further aligns the model's training objective with practical communication system requirements. Comprehensive simulation results robustly demonstrate that the proposed method not only delivers excellent nonlinear compensation performance but also achieves a significant 43% reduction in computational complexity compared to a similar network without the attention mechanism. This research strikes an effective balance between high performance and manageable computational cost, presenting a viable and promising technical path for the deployment of advanced deep learning technologies in future high-speed, high-capacity optical communication systems.