Generalized Rayleigh quotient optimization method for inter-channel nonlinearity compensation in coherent optical communication systems

Inter -channel nonlinearity compensation plays a crucial role in wavelength division multiplexing (WDM) systems for improving transmission capacity and distance. In this work, we propose a novel, to the best of our knowledge, inter -channel nonlinearity compensation method called generalized Rayleigh quotient optimization (GRQO) method with two different working modes. In an 8 x 64 GBaud 16-ary quadrature amplitude modulation (16-QAM) experimental system over 1600 km standard single -mode fiber (SSMF), the proposed method shows a 0.40 dB Q( )(2)factor improvement over nonlinear polarization cross talk canceller (NPCC) with a moderately low computational complexity of about 2000 real multiplications per bit (RMb). (c) 2024 Optica Publishing Group