Quadrant Multiplexing-based Geometrically Shaped Four-Dimensional Modulation Format for Polarization Multiplexed Coherent Optical Communication System

•We proposed novel GS-4D modulation format based on quadrant multiplexing.•Novel SP approach is proposed to increase the Euclidean distance.•A GS scheme with binary tags completed by improved PSO is further designed.•The GS-4D can achieve a SNR gain of 3.6 dB compared with PM-16QAM.•The proposed signal has great advantages in transmission distance and data rate.