Distributed management over Unmanned Aerial Vehicle (UAV) based communication networks (UCNs) has attracted increasing research attention. In this work, we study a distributed user connectivity maximization problem in a UCN. The work features a horizontal study over different levels of information exchange during the distributed iteration and a consideration of dynamics in UAV set and user distribution, which are not well addressed in the existing works. Specifically, the studied problem is first formulated into a time-coupled mixed-integer non-convex optimization problem. A heuristic two-stage UAV-user association policy is proposed to faster determine the user connectivity. To tackle the NP-hard problem in scalable manner, the distributed user connectivity maximization algorithm 1 (DUCM-1) is proposed under the multi-agent deep Q learning (MA-DQL) framework. DUCM-1 emphasizes on designing different information exchange levels and evaluating how they impact the learning convergence with stationary and dynamic user distribution. To comply with the UAV dynamics, DUCM-2 algorithm is developed which is devoted to autonomously handling arbitrary quit's and join-in's of UAVs in a considered time horizon. Extensive simulations are conducted i) to conclude that exchanging state information with a deliberated task-specific reward function design yields the best convergence performance, and ii) to show the efficacy and robustness of DUCM-2 against the dynamics.

Deep reinforcement learning (DRL) has been extensively applied to Multi-Unmanned Aerial Vehicle (UAV) network (MUN) to effectively enable real-time adaptation to complex, time-varying environments. Nevertheless, most of the existing works assume a stationary user distribution (UD) or a dynamic one with predicted patterns. Such considerations may make the UD-specific strategies insufficient when a MUN is deployed in unknown environments. To this end, this paper investigates distributed user connectivity maximization problem in a MUN with generalization to arbitrary UDs. Specifically, the problem is first formulated into a time-coupled combinatorial nonlinear non-convex optimization with arbitrary underlying UDs. To make the optimization tractable, a multi-agent CNN-enhanced deep Q learning (MA-CDQL) algorithm is proposed. The algorithm integrates a ResNet-based CNN to the policy network to analyze the input UD in real time and obtain optimal decisions based on the extracted high-level UD features. To improve the learning efficiency and avoid local optimums, a heatmap algorithm is developed to transform the raw UD to a continuous density map. The map will be part of the true input to the policy network. Simulations are conducted to demonstrate the efficacy of UD heatmaps and the proposed algorithm in maximizing user connectivity as compared to K-means methods.

Unmanned Aerial Vehicle (UAV) based communication networks (UCNs) are a key component in future mobile networking. To handle the dynamic environments and UAV topologies in UCNs, reinforcement learning (RL) has been a promising solution attributed to its strong capability of adaptive decision-making free of the environment models. However, most existing RL-based research focus on control strategy design assuming a fixed set of UAVs, i.e., the number of UAVs does not change during the mission. Few works have investigated how UCNs should be adaptively regulated when the number of serving UAVs changes dynamically. This article discusses RL-based strategy design for adaptive UCN regulation given a dynamic number of UAVs, addressing both reactive strategies in general UCNs and proactive strategies in solar-powered UCNs. An overview of the UCN and the RL framework is first provided. Potential research directions with key challenges and possible solutions are then elaborated. Preliminary works are presented as case studies to inspire innovative ways to handle dynamic UAV crew with different RL algorithms.

The increasingly frequent occurrence of natural disasters has severely interfered with the operation of fundamental infrastructures such as power, transportation, and communication systems. For these decades-old infrastructures, enhancing the system resilience requires extremely high upgrade expenditure. Therefore, more flexible and cost-efficient solutions are in urgent demand. Equipped with on-broad large-capacity batteries, electric vehicles (EVs) could serve as mobile post-disaster rescue devices, namely mobile energy storage (MES). This paper proposes a flexible post-disaster rescue scheme using mobile and connected EVs as MESs to supply emergency resources before the fundamental infrastructures fully recover. Different from existing literature, this paper uncovers the potential energy supply and communication capabilities of MESs to provide damaged areas with on-demand energy and communication resources. Specifically, the uncertainty of natural disasters of tornadoes and flooding is modelled during different scenario generations. Then, a two-stage stochastic programming problem is formulated to determine the MES deployment location in the pre-disaster stage and the MES service operation in the post-disaster stage. The generated disaster scenarios are integrated into the formulated problem to ensure a statistically optimal result. Simulation results validate the optimality of the proposed scheme compared to benchmark schemes.

With the electrification of the automobile system, the overload problem is being incurred by the increased charging demands from the electric vehicles (EVs). To avoid overloading in the power grid, microgrids (MGs) can be integrated to assist the power balancing. In addition, a coordinated charging strategy among EVs can mitigate the overload problem based on spatially and temporally varying distribution of vehicle traffic in transportation. However, few works have been done on the integration of power system and transportation system in large-scale realistic EV networks. In this paper, both the power distribution and transportation systems are integrated in the high-fidelity and at-scale co-simulation models. Specifically, an LSTM-based prediction model of vehicle traffic distribution is first built and trained over realistic vehicle trace files. The predicted vehicle traffic distribution is exploited to forecast the future EV charging demand. The distribution system is then simulated to describe how the loads (e.g., controllable loads and EVs) and supplies (e.g., distributed generations and energy storages in MGs) impact a power system across the region at scale. Based on the forecast EV loads and co-simulation results from the integrated system, a spatio-temporal coordinated fast EV charging strategy is developed and executed in a distributed way to improve the reliability and resilience of the power systems. Numerical results demonstrate that our proposed strategy can improve the total EV charging performance in the power system while maintaining the power balance of the networked MGs.

As ubiquitous interconnection becomes a reality for human beings, addressing the challenge of seamless coverage in the near future 6 G network, particularly for remote area connections, has become increasingly urgent. HAPs and UAVs can efficiently extend the network coverage which is unfeasible by ground base stations. However, due to the high dynamic nature of air ground communications, both the available network resource and channel state information (CSI) are time varying which is typically called the dilemma of information uncertainty, often making the resource allocation intractable. To overcome the challenges, first we propose an air ground integrated 6 G heterogeneous network (AGIN 6 G) leveraging the wide coverage of HAPs and the hot spot communication enhancement of UAVs. Second we formulate the challenge as a three-sided matching problem with the size and cyclic preference (TMSC) among HAPs, UAVs and users. To find a feasible solution, we transform the problem into a R-TMSC issue by applying some reasonable constraints, especially when the CSI is assumed known, we can develop the HOR $^{2}$ A-CSI algorithm. Third, to cope with the information uncertainty, we propose the CVA-UCB matching solution augmented with machine learning. Finally, we conduct extensive experiments comparing with benchmark algorithms. Our works can verify that the proposed algorithm is superior to others in terms of the data transmission rate, system revenue and throughput.

Enhancing the resilience of fundamental infrastructures such as the power system and the communication system are crucial considering the increasingly frequent occurrence of natural disasters. Unmanned aerial vehicles (UAVs) have been extensively discussed as flexible and effective disaster rescue devices for the communication system but their service quantity and quality are severely constrained by their limited battery capacities. In this paper, we leverage the power provision and computing offloading capabilities of electric vehicles (EVs) to help UAVs offload computing tasks and recharge UAVs to prolong their service period. Different from existing literature, the proposed work explores the potential of ground EVs to serve as both power source and computing offloading devices to help extend the operation period of UAVs. Specifically, a cooperative UAV-EV rescue framework is developed to characterize the cooperation procedure of UAV-EV pairs. Then, a two-tier matching problem is formulated where the lower tier maximizes the UAV operation period leveraging EV’s computing and recharging services while the upper tier matches UAVs with EVs considering the optimized UAV operation period and statuses of different outage regions. The matching problem is an integer linear programming problem in nature and can be efficiently solved by CVX. Simulation results validate the effectiveness of the cooperative framework on the service utility and UAV operation period extension compared to benchmarks.

This work studies optimal solar charging for solar-powered self-sustainable UAV communication networks, considering the day-scale time-variability of solar radiation and user service demand. The objective is to optimally trade off between the user coverage performance and the net energy loss of the network by proactively assigning UAVs to serve, charge, or land. Specifically, the studied problem is first formulated into a time-coupled mixed-integer non-convex optimization problem, and further decoupled into two sub-problems for tractability. To solve the challenge caused by time-coupling, deep reinforcement learning (DRL) algorithms are respectively designed for the two sub-problems. Particularly, a relaxation mechanism is put forward to overcome the "dimension curse" incurred by the large discrete action space in the second sub-problem. At last, simulation results demonstrate the efficacy of our designed DRL algorithms in trading off the communication performance against the net energy loss, and the impact of different parameters on the tradeoff performance.

Since the existing terrestrial fifth generation (5G) network has limited coverage, it is difficult to meet the growing demand for seamless network connection. Meanwhile, current network resource allocation methods mainly research on how to improve system performance only from the perspective of resource utilization, but rarely take users’ specific needs for network content into consideration. This brings severe challenges to efficient network service and flexible resource allocation. Therefore, we construct the content service-oriented resource allocation model for space–air–ground integrated sixth generation networks (SAGIN 6G), and formulate the three-sided matching issue among the space–air–ground integrated network equipment (SAGINE), content sources, and users. In this model, users request to establish connection with SAGIN which forwards users’ request to content service provider (CSP). CSP manages the creation of content data, and finally returns the requested content to users through SAGIN. For the content service-oriented resource allocation in SAGIN, finding the optimal stable three-sided matching with the largest cardinality is an NP-complete problem. Therefore, to efficiently solve the above issue, we design some reasonable restrictions and convert it to a restricted three-sided matching problem with size and cycle preferences. We further develop the content-oriented resource allocation algorithm (COR2A) and the user-oriented resource allocation algorithm (UOR2A) in a distributed manner. Extensive simulations verify our approach outperforms traditional benchmark resource allocation schemes in terms of system throughput, CSP revenue, and user experience.

With the popularization of artificial intelligence, 5G and other technologies, a number of emerging applications require both efficient communication and computing service, which poses enormous challenges to the computing ability and battery capacity of terminal equipment. Moreover, ground-based 5G system can not provide seamless service especially for hotspot and remote area. To tackle the above challenges, we minimize the weighting of delay and energy consumption by optimizing the task offloading decision and computing resource allocation, which, however, is a mixed integer nonlinear programming (MINLP) issue due to the strong coupling between optimization variables. Therefore, we decompose it into two subproblems and design a deep reinforcement learning-Based approach to address the first problem with offloading decision-making. For the second computing resource allocation subproblem,. a Greedy-based solution is proposed. The simulation results indicate that, in comparison to other benchmark approaches, the proposed method can achieve superior performance.

Multi-agent reinforcement learning has been applied to Unmanned Aerial Vehicle (UAV) based communication networks (UCNs) to effectively solve the problem of time-coupled sequential decision making while achieving scalability. Nevertheless, a transverse comparison on the impact of different levels of inter-agent information exchange on the learning convergence has not been well studied. In this work, we study a distributed user connectivity maximization problem in a UCN, aiming to obtain a trajectory design to optimally guide UAVs' movements in a time horizon to maximize the accumulated number of connected users. Specifically, the problem is first formulated into a time- coupled mixed-integer non-convex optimization problem. A two- stage user association policy is proposed to determine the UAV- user connectivity. A multi-agent deep Q learning algorithm is then designed to solve the optimization, featuring four different levels of information exchange and reward function design. Simulations are conducted to compare the convergence speed and total number of connected users per episode between different levels. The results show that exchanging state information with a deliberated task-specific reward function design yields the best convergence performance in both cases of stationary and dynamic user distributions.

Leveraging unmanned aerial vehicles (UAVs) for access and high altitude platform stations (HAPSs) for data backhaul to construct the Air-Ground Integrated Network (AGIN), is a feasible solution to achieve seamless network coverage for remote IoT devices in future 6G era. However, since the number of terminals increases exponentially, it is essential to improve spectrum efficiency and throughput of the system. In addition, the limited on-board energy storage of UAVs and finite battery of IoT terminals make the system energy efficiency (EE) a new concern. To cope with the above mentioned challenges, in this study we first put forward a clustered-NOMA (C-NOMA)-enabled heterogeneous AGIN model for remote areas including one HAPS for backhaul and multiple UAVs for access, where C-NOMA is used to obtain both improved system throughput and terminal complexity. Then, we study the joint UAV trajectory plan and resource allocation problem in order to maximize the system EE. Since it is a mixed integer nonlinear programming (MINLP) issue coupled with transmission power, subchannel allocation, UAV trajectory and speed control, this problem is decoupled into two subproblems and solved iteratively. For the first one, the optimal channel and power strategy is obtained by our subchannel allocation and power control for AGIN EE (SAPAE) maximum algorithm according to Lagrange dual decomposition method. For the second, for transforming the nonconvex problem into a convex one, we provide the successive convex approximation-based UAV trajectory and speed optimization for AGIN EE (SUTSAE) maximum algorithm, and then the near-optimal UAV trajectory and flight speed are obtained. We conduct extensive experiments to compare with other benchmark methods. It is shown that the proposed approach is better in EE and spectrum efficiency.

By combining information communication technology with power grid, the smart grid-oriented Power Internet of Things (PIoT) has become a critical technology to guarantee the safe and reliable power grid operation and improve system energy efficiency. Nevertheless, PIoT devices have only limited communication and computing resources since they are mostly deployed in remote areas that may be out of service coverage of existing terrestrial 5G networks. To overcome the resource limitation, we leverage Air-Ground Integrated C-NOMA Heterogeneous PIoT Networks (PAGIC HetNets), and study the core challenges in PAGIC HetNets. As PIoT devices are normally powered by battery, we aim at minimizing the energy consumption of PIoT devices and thoroughly investigate the problem of task offloading and resource allocation with minimal energy consumption. This problem belongs to a mixed integer nonlinear programming (MINLP) with extra difficulty that the long-term queuing delay and short-term constraints are coupled. To tackle the difficulty, we use Lyapunov optimization to transform this hard problem into three subproblems. The first subproblem is task splitting and local computing resource assignment at the PAGIC user side, which we solve with the Lagrangian multiplier method. The second subproblem is queue-aware channel reusing, and matching theory is adopted to solve it. The third subproblem is optimizing the aerial server resource allocation, for which we propose a greedy-based solution. Numerical simulations demonstrate that our approach can obtain excellent performance in terms of energy consumption, spectrum efficiency, task backlog, and queuing delay with lower complexity compared with several benchmark methods.

In the context of complex and dynamic marine environment, the offloading of computing tasks for ships of Internet of Things (IoT) users is a very challenging problem considering the different quality of service (QoS) requirements of maritime applications. Mobile edge computing driven by powerful computing capability and edge intelligence is taken as a promising solution, especially for the resource-constrained and delay-sensitive maritime IoT users. In this paper, we study the optimal edge server selection problem for ship IoT users to jointly minimize the latency and energy consumption for task offloading. Specifically, we first propose a novel space-air-ground-edge (SAGE) integrated maritime network architecture to offload computation-intensive IoT services at sea. Then, the latency and energy consumption of data transmission and processing during offloading are modelled. Based on the models, the edge server selection problem is formulated into a Multi-Armed Bandits learning problem, with considering the task latency requirement and energy budget. To achieve the optimal solution, a novel algorithm, referred to as UCB1-ESSS, is developed, which links the latency, energy consumption, and network constraints by introducing both reward and cost. The simulation results show that the proposed algorithm can achieve considerably lower offloading latency and weighted latency-energy cost compared with the traditional algorithms under different QoS requirements, which proves the efficacy of theproposed algorithm.

Aim: The rapid growth in the number of ground users over recent years has introduced the issues for a base station of providing more reliable connectivity and guaranteeing the reasonable quality of service (QoS). Thanks to the unique features of unmanned aerial vehicles (UAVs), such as flexibility in deployment, large coverage range and lower cost, UAVs can help the base station to provide wireless connectivity to the ground users, e.g., in rural and remote areas. As the energy limitation is the main concern for UAVs, the motivation is to provide uninterrupted connection to ground users in the next generation wireless networks using solar powered UAV-assisted air networks. Methods: The research uses global horizontal irradiance (GHI) data from the National Renewable Energy Laboratory, small cell power ratings for communication, and UAV parameters. In addition, the TensorFlow library and Python programming language were also used to develop machine learning models and simulate the UAV flying time. Results: In this paper, we develop a novel resource management system for UAVs, which consists of an energy harvesting deep learning model to predict the future power harvested from the solar panel and a consumption model which determines user arrival rate. With energy consumption and harvesting predictions, the resource management system adaptively switches the power consumed by a UAV for communication. In addition, based on the future energy availability and user's arrival rate, the resource management system communicates with other UAVs and enables energy coordinating scheduling among multiple UAVs to support user communications. The experiment results demonstrate that by using adaptive energy scheduling among UAVs, the flying time of the UAVs is improved by 40% during nighttime and by 37% when performing energy coordination among multiple UAVs. Conclusion: In this work, the UAV based communications have been researched. To understand more about UAVs and air segments, some literature review has been done based on previous works. Finally, alteration of the transmission power using several methodologies has been accomplished to increase the flying time of the UAV.

With the vigorous development of information and communication technology, mobile internet has undergone tremendous changes. How to achieve global coverage of the network has become the primary problem to be solved. GEO satellites and LEO satellites, as important components of the satellite–ground network, can offer service for hotspots or distant regions where ground-based base stations’ coverage is limited. Therefore, we build a satellite–ground network model, which transforms the satellite–ground network resource allocation problem into a matching issue between GEO satellites, LEO satellites, and users. A GEO satellite provides data backhaul for users, and a LEO satellite provides data transmission services according to users’ requests. It is important to consider the relationships between all entities and establish a distributed scheme, so we propose a three-sided cyclic matching algorithm. It is confirmed by a large number of simulation experiments that the method suggested in this research is better than the conventional algorithm in terms of average delay, satellite revenue, and number of users served.

With the rapid development of the Internet of Things, multi-access edge computing (MEC) has emerged as a key technique and intelligent edge resource allocation has become one of the important research issues in MEC. In this paper, we utilize advantages of matching game to integrate users’ needs and the position of edge computing nodes in IoT networks. The proposed GS-based two-sided matching approach simulates the competition and negotiation relationship between different user sets in IoT edge nodes, and provides feasible options based on the preference between different entities. Simulations show that it not only solves the problem of edge resource allocation in a semi-distributed manner, but also effectively improves SP’s revenue.

Traffic engineering (TE) is fundamental and important in modern communication networks. Deep reinforcement learning (DRL)-based TE solutions can solve TE in a data-driven and model-free way thus have attracted much attention recently. However, most of these solutions ignore that TE is a real-world application and there are challenges applying DRL to real-world TE like: (1) Efficiency. Existing learning-from-scratch DRL agent needs long-time interactions to find solutions better than traditional methods. (2) Safety. Existing DRL-based solutions make TE decisions without considering safety constraints, poor decisions may be made and cause significant performance degradation. In this paper, we propose a safe training approach for DRL-based TE, which tries to address the above two problems. It focuses on making full use of data and ensuring safety so that DRL agent for TE can learn more quickly and possibly poor decisions will not be applied to real environment. We implemented the proposed method in ns-3 and simulation results show that our method performs better with faster convergence rate compared to other DRL-based methods while ensuring the safety of the performed TE decisions.

Vehicular communication networks hold promise to significantly improve road safety by giving both automated vehicles and human drivers improved awareness and advanced warning to emergencies. The emergency messages are broadcasted upon emergency detection, but this does not guarantee recipients will be able to avoid collision. In this paper, we introduce a method to relate the delay tolerance of each vehicle in the network directly to the transmission range by taking into consideration the reaction time of the drivers in order to ensure each vehicle in the network can avoid a collision. The system utilizes a digital-twin system to maintain network awareness and accounts for the coexistence of automated vehicles and human driving vehicles and allows the network to minimize transmission range while effectively assuring the safety. The proposed strategy is tested in simulated road scenarios generated from measured highway traffic data. The simulation results demonstrate the efficacy of the proposed strategy through extensive evaluation of multiple traffic scenarios.

The rechargeable battery of a plug-in electric vehicle (PEV) endows the PEV with dual roles in the power grid as power load and mobile energy storage (MES). Owing to the technical advancement of autonomous driving, private PEVs that are parked most of the day can be used as private MESs (PMESs) to autonomously deliver energy for overloaded charging stations (CSs). In this paper, we investigate an energy compensation problem where PMESs are scheduled to deliver energy to overloaded CSs so that the energy balance can be achieved while the energy delivery time can be minimized. Based on the time-variant CS operation status and traffic conditions, we propose a pricing-based scheduling scheme that considers both PMES navigation and incentive price design. First, to navigate PMESs in the energy-capacitated transportation system, a minimum-cost flow problem is formulated to minimize the energy delivery time. Then, the incentive price is determined to encourage PMESs to follow the optimal navigation results for energy delivery. Simulations are conducted based on the traffic data of California highway to validate the effectiveness of the proposed scheduling scheme.