Multi-Agent Reinforcement Learning Based Energy Efficiency Optimization in NB-IoT Networks

Based on the existing Evolved Packet System (EPS) architecture, Narrowband Internet of Things (NB-IoT) has been expected as a promising paradigm to support energy-aware massive Machine Type Communications (mMTC). However, with the tremendous increase of IoT devices, as well as their requirements of energy- saving and low-cost, current power ramping and preamble allocation mechanisms in legacy long term evolution (LTE) can hardly achieve high energy efficiency in machine-to-machine (M2M) communications, mainly resulting from the significant redundancy of control signals. Due to the strict restrictions of NB-IoT, up till the present moment, the standardized preamble allocation mechanism is still randomly picking. To satisfy these constrained conditions in NB-IoT, this work proposes a joint optimization framework of power ramping and preamble picking to improve the energy efficiency of NB-IoT systems. In this optimization problem, a comprehensive energy estimation model is established, which investigates the inadequacy of random access (RA) procedure and meanwhile reveals the effects of power ramping and preamble picking on energy efficiency. In addition, to search the optimal policies of the joint optimization formulated. A distributed Multi-Agent Reinforcement Learning (MARL) algorithm based on Win-or-Learn-Fast Policy Hill-Climbing (WoLF-PHC) is proposed, in which a "stateless" modification is introduced to reduce the algorithm complexity significantly. The performance of high energy efficiency is validated in simulations, which also reveal the applicability and convergence of the designed WoLF-PHC based optimization algorithm.