Cylindrical vector beams (CVBs), which have spatial non-uniform polarization distributions, have gained considerable interest in optical communication owing to their orthogonal vector modes. However, the identification of vector modes is always a challenge owing to the lack of effective mode extraction techniques, which hinders the practical application of CVB communications. Herein we introduce a convolutional neural network (CNN) to identify the vector modes of CVBs and an experiment to demonstrate its application in CVB shift-keying (CVB-SK) communication. The CNN model composed of convolution layers was designed to extract mode features from the petal patterns obtained by CVBs using a Glan prism. The identification accuracy of vector modes ranging from −20 to 20 reached 98.07% with an atmospheric turbulence of $C_{\textrm {n}}^{2} =1\times 10^{-14}\textrm {m}^{-2/3},\Delta \textrm {z=}2000~\textrm {m}$ . Mapping and encoding a Maxwell grayscale image with 100 $\times100$ pixels to vector modes, we constructed a CVB-SK communication system, and the CVB-SK signals were successfully demodulated by the CNN model with a pixel-error rate of $5.19\times 10^{-2}$ . Our results indicate that this CNN model can effectively recognize vector modes, which may have application potential in CVB communication, high-dimensional quantum information protocols, etc.

Fractional vortex beam (FVB) possessing helical phase can be applied in the shift-keying communication due to its fractional orbital angular momentum (FOAM) mode, which theoretically allows an infinite increase of the transmitted capacity. However, the discontinuity of spiral phase makes FVB more likely to be disturbed in turbulence environment, and the precise measurement of distorted FOAM modes is crucial for practical FOAM-based communication application. Here, we proposed a FOAM mode recognition method with feedforward neural network (FNN). Employing the diffraction preprocessing of a two-dimensional fork grating, the original optical features of FVBs can be extended along the far-field diffraction order, endowing FNN more feature information and saving calculation time, and enlarging the detection range to conjugate FOAM modes. The simulation results show that the 9-layer FNN can identify FOAM mode with interval of 0.1 with an accuracy of 99.1% under the turbulences of C-n(2) = 1 x 10(-14) m(-2/3) and Delta z = 10m. Furthermore, we experimentally constructed a 102-ary FOAM shift-keying communication link to transmit gray image, and the signals are successfully demodulated by the FNN model with the pixel-error-rate of 0.07160. It is anticipated that the proposed FNN-based FOAM recognition method will break the limitation of precision measurement under turbulence environment in practical FOAM applications.