A Neural Network Based Trellis Coded Modulation Optimization Scheme

In this paper, trellis coded modulation (TCM) is optimized to achieve better bit error ratio (BER) performance by training a neural network-based encoder. Specifically, two blocks, i.e., the mapper and constellation modulator, are jointly optimized, while convolutional coding and Viterbi decoder are maintained in the traditional manner. For the purpose of numerical optimization, a BER upper bound expression is derived as a function of Euclidean distance, and a loss function is designed based on the upper bound expression. The optimized TCM system can achieve higher coding gain and shaping gain, and can be implemented using the traditional encoder and decoder structure. Notably, the proposed scheme supports arbitrary code lengths with guaranteed scalability and manageable computational complexity. Simulation results demonstrate that the proposed scheme outperforms not only the traditional TCM with symmetric constellations, but also other TCM optimization schemes with asymmetric constellations.