Ocotillo‐Optimized Modulated FSO Link with OFDM‐Málaga Channel for 5G/6G under Pointing Errors and Turbulence

ABSTRACT Free‐space optical (FSO) communication systems face significant performance degradation due to atmospheric turbulence and pointing errors, which reduce spectral efficiency, limit data rates, and affect link reliability. This work proposes an Ocotillo‐optimized modulated FSO (OM‐FSO) system that employs the Málaga distribution to accurately represent turbulence and integrates the Ocotillo optimization algorithm (OcOA) with a low‐intermediate frequency (low‐IF) receiver to enhance system performance. The proposed design improves spectral efficiency, increases data transmission rates, and reduces bit error rate (BER) while maintaining robustness under severe atmospheric variations. Simulation results show that for a 20‐km link, the system achieves a reduction in required transmission power of 20.4 dB under clear conditions and 20.53 dB under strong turbulence at a BER of 10 –7.2 . Furthermore, it attains a spectral efficiency of 154 bits/s/Hz and a data rate exceeding 870 Gbps. These results demonstrate that the optimized OM‐FSO system offers a resilient and efficient solution for long‐distance FSO communication.