This paper focuses on one of the collision avoidance scenarios for unmanned aerial vehicles (UAVs), where the UAV needs to avoid collision with the enemy UAV during its flying path to the goal point. Such a type of problem is defined as the enemy avoidance problem in this paper. To deal with this problem, a learning based framework is proposed. Under this framework, the enemy avoidance problem is formulated as a Markov Decision Process (MDP), and the maneuver policies for the UAV are learned based on a temporal-difference reinforcement learning method called Sarsa. To handle the enemy avoidance problem in continuous state space, the Cerebellar Model Arithmetic Computer (CMAC) function approximation technique is embodied in the proposed framework. Furthermore, a hardware-in-the-loop (HITL) simulation environment is established. Simulation results show that the UAV agent can learn a satisfying policy under the proposed framework. Comparing with the random policy and the fixed-rule policy, the learned policy can achieve a far higher possibility in reaching the goal point without colliding with the enemy UAV.

This paper focuses on the conflict detection and resolution (CDR) of unmanned aerial vehicles (UAVs). Firstly, the airspace conflict problem of UAVs is studied and a taxonomy of conflict situation is presented. The multi-UAV conflict is studied in virtue of the graph theory. The CDR problem is casted to a nonlinear optimization problem. Secondly, a two layered optimization algorithm, which combines stochastic parallel gradient descent (SPGD) method and Sequential quadratic programming (SQP) algorithm, is presented to solve the nonlinear optimization problem. Numerical simulations are performed to demonstrate the computational efficiency of this solver. Thirdly, the proposed algorithm is extended to 3-D space. Finally, the proposed algorithm is demonstrated on several scenarios. The results demonstrate that the proposed method outperform the existing algorithms. It can obtain conflict free solutions that would not lead to unnecessary detors.

Transfer Learning (TL) has received a great deal of attention because of its ability to speed up Reinforcement Learning (RL) by reusing learned knowledge from other tasks. This paper proposes a new transfer learning framework, referred to as Transfer Learning via Artificial Neural Network Approximator (TL-ANNA). It builds an Artificial Neural Network (ANN) transfer approximator to transfer the related knowledge from the source task into the target task and reuses the transferred knowledge with a Probabilistic Policy Reuse (PPR) scheme. Specifically, the transfer approximator maps the state of the target task symmetrically to states of the source task with a certain mapping rule, and activates the related knowledge (components of the action-value function) of the source task as the input of the ANNs; it then predicts the quality of the actions in the target task with the ANNs. The target learner uses the PPR scheme to bias the RL with the suggested action from the transfer approximator. In this way, the transfer approximator builds a symmetric knowledge path between the target task and the source task. In addition, two mapping rules for the transfer approximator are designed, namely, Full Mapping Rule and Group Mapping Rule. Experiments performed on the RoboCup soccer Keepaway task verified that the proposed transfer learning methods outperform two other transfer learning methods in both jumpstart and time to threshold metrics and are more robust to the quality of source knowledge. In addition, the TL-ANNA with the group mapping rule exhibits slightly worse performance than the one with the full mapping rule, but with less computation and space cost when appropriate grouping method is used.

This paper presents a study on short-term cooperative conflict detection and resolution (CDR) of unmanned aerial vehicles (UAVs). A two-layered CDR mechanism is proposed, which aims at guaranteeing safe separation, minimizing the overall cost of UAVs, and improving computational efficiency. In the first layer, the information from the environment is processed. In the second layer, conflicts among UAVs are resolved by applying the local centralized optimization method, with consideration given to the dynamic constraints of UAVs. This paper studies the safe separation constraints of pairwise conflicts in virtue of a geometry-based method. A heading change and speed change mixed conflict resolution approach is applied. To meet with the online planning requirements, the vectorized stochastic parallel gradient descent-based method is proposed to find the local optimal heading change solutions. The linear safe separation constraints on speeds are derived. A periodicity feature-based method is used to depart the feasible sub-regions for each pairwise conflict. A mixed integer linear programming model is established to find the optimal speed change solutions. The experiments results show that the proposed heading change algorithm could greatly reduce the summation of additional flight distances of UAVs, and influences on air traffic, compared with other short-term algorithms; the computational efficiency of this algorithm satisfies the online planning requirement. Comparing with the existing algorithm, our speed change algorithm reduces the number of feasible sub-regions to 2n(c) times lower, where n(c) is the number of pairwise conflict, and therefore, it reduces computation time dramatically.

Though popular in many agent learning tasks, reinforcement learning still faces problems, such as long learning time in complex environment. Transfer learning could shorten the learning time and improve the performance in reinforcement learning by reusing the knowledge acquired from different but related source task. Due to the difference in state space and/or action space of the target and source task, transfer via inter-task mapping is a popular method. The design of the inter-task mapping is very critical to this transfer learning method. In this paper, we propose a linear multi-variable mapping (LMVM) for the transfer learning to make a better use of the knowledge learned from the source task. Unlike the inter-task mapping used before, the LMVM is not a one-to-one mapping but a one-to-many mapping, which is based on the idea that the element in target task is related with several similar elements from source task. We test transfer learning via our new mapping on the Keepaway platform. The experimental results show that our method could make the reinforcement learning agents learn much faster than those without transfer and those transfer with simpler mappings.

Transfer learning could speed up reinforcement learning in many applications. Toward the fully autonomous reinforcement learning transfer agent, the mapping between the source task and target task should be learned instead of human designed. To this end, this paper proposes an autonomous intertask mapping learning method via artificial neural network, so as to reduce the human intervention in the transfer process. With this learned network, the reinforcement learning agent could transfer the learned knowledge from source task to target task for initialization, and set a good prior for the learning in the target task. The method is tested on the Keepaway Soccer Platform. The results indicate that the proposed method could provide a good jumpstart in the target task when weights are properly chosen for training the network.

Task allocation is a very important part in multi-UAV system. There are a lot of factors needed to be taken into consideration when assigning a dynamic task to one of the UAV in the multi-UAV system. We aim at solving the multi-UAV dynamic task allocation problem under multiple constraints. In this paper, we propose a method which uses the auction algorithm to assign the dynamic tasks to the UAVs. We propose a multi-layer cost computation method, in which the computation of the cost is divided into four layers in accordance with the four types of constraints. In this way, we solve multiple constraints and provide a way of calculating the price for bid. We use a simulation experiment to show the effectiveness of the new method. The result indicates that our proposed method can assign the new tasks very well and solve the multiple constraints at the same time.

In this article, a tree search algorithm is proposed to find the near optimal conflict avoidance solutions for unmanned aerial vehicles. In the dynamic environment, the unmodeled elements, such as wind, would make UAVs deviate from nominal traces. It brings about difficulties for conflict detection and resolution. The back propagation neural networks are utilized to approximate the unmodeled dynamics of the environment. To satisfy the online planning requirement, the search length of the tree search algorithm would be limited. Therefore, the algorithm may not be able to reach the goal states in search process. The midterm reward function for assessing each node is devised, with consideration given to two factors, namely, the safe separation requirement and the mission of each unmanned aerial vehicle. The simulation examples and the comparisons with previous approaches are provided to illustrate the smooth and convincing behaviours of the proposed algorithm.

This paper focuses on the solution for conflict detection and resolution (CDR) of unmanned aerial vehicles (UAVs) by heading control. The cooperative method is proposed. First, the relationships between conflicts involved UAVs are described by the geometric method. The practical and potential conflicts are considered. Then, the CDR problem is formalized as a nonlinear optimization problem so as to minimize maneuver costs. Moreover, a two-layer strategy composed of stochastic parallel gradient descent (SPGD) and interior-point algorithm is designed to efficiently solve the non-convex optimization problem. Finally, our approach is demonstrated on several scenarios and the simulation results show that it can achieve high performance in obtaining near optimal maneuvers for UAVs CDR.

The recommender system is widely used in many areas in the age of information overload. Collaborative filtering (CF), as one of the most successful methods used for recommendation, recommends items based on the nearest neighbors of the target user. Thus, the performance of the recommender system depends largely on the similarity measure used for selecting neighbors. Most of the traditional similarity measures are based on the rating data that users give to the items, such as Pearson's correlation and cosine, and suffer from low performances. In order to improve the performance of CF recommender system, this paper proposes a new kind of similarity measure based on user preference models and applies it to a movie recommender system. In this paper, two user preference models are build, one is focused on percentages of different movie genres a user has watched, the other is on the average ratings a user has given to different genres of movies. Based on the two user preference models, two new similarity measures are designed. The experiments compare the performance of the two new similarity measures with the Pearson's correlation which is widely used in the traditional CF, and are carried out on the MovieLens data set. The results show that the new similarity measure based on the watched genre ratio model outperforms that of traditional CF in all aspects. While the new similarity measure based on the average genre rating model has almost the same performance with the traditional one, but has much less computing complexity for no need in finding the co-rated items.

With the development of simulation system, too much simulation resources are accumulated, most of which are reusable. Good management on these resources can fully achieve the reusability of them and facilitate the simulation process, but the distribution and complexity of simulation resource add challenge on the management in aspects like resource storage, computing speed and resource sharing. The cloud computing integrates mature technologies to provide infinite storage and computing resource on users' demand, which is needed in the simulation resource management. To use these merits of cloud, this paper proposed a cloud-based simulation resource management model and produced a detailed design of the prototype system. This model is universal and provides a framework for cloud-based system; the prototype system is flexible for any scale of simulation system, it offer mechanism in data security, storage and usage, and can manage the simulation resources effectively and economically.

The present simulation resource management systems are full of all kinds of simulation resources; it is inefficient to get the needed simulation resource with the traditional search methods. To solve this problem, the recommendation system based on collaborative filtering is applied to the simulation resource management system, which can recommend the most relative simulation resource to the user according to user's previous preference. After analyzing the necessity of combining the recommendation system with the simulation resource system, the simulation resource recommendation system is designed and realized. The realization includes three main procedures: collecting user preferences, finding neighbor users, recommending simulation resources. The recommendation system collects users' grading on used simulation resources as user preferences, and uses the Pearson correlation to calculate the similarity between users and then finds out the neighbor users; then it bases on the neighbor users to predict the user's grade of the resource and then gets the recommended resources. The test result shows that the recommended resources have strong similarity with the user' previous preference. The recommendation system improves the efficiency of the resource obtaining and the use frequency of the recommended resource.