Machine unlearning refers to the process of mitigating the influence of specific training data on machine learning models based on removal requests from data owners. However, one important area that has been largely overlooked in the research of unlearning is reinforcement learning. Reinforcement learning focuses on training an agent to make optimal decisions within an environment to maximize its cumulative rewards. During the training, the agent tends to memorize the features of the environment, which raises a significant concern about privacy. As per data protection regulations, the owner of the environment holds the right to revoke access to the agent's training data, thus necessitating the development of a novel and pressing research field, known as \emph{reinforcement unlearning}. Reinforcement unlearning focuses on revoking entire environments rather than individual data samples. This unique characteristic presents three distinct challenges: 1) how to propose unlearning schemes for environments; 2) how to avoid degrading the agent's performance in remaining environments; and 3) how to evaluate the effectiveness of unlearning. To tackle these challenges, we propose two reinforcement unlearning methods. The first method is based on decremental reinforcement learning, which aims to erase the agent's previously acquired knowledge gradually. The second method leverages environment poisoning attacks, which encourage the agent to learn new, albeit incorrect, knowledge to remove the unlearning environment. Particularly, to tackle the third challenge, we introduce the concept of ``environment inference attack'' to evaluate the unlearning outcomes.

Reinforcement learning (RL) is a principled AI framework for autonomous, experience-driven learning. Deep reinforcement learning (DRL) enhances this by incorporating deep learning models, promoting a higher-level understanding of the visual world. However, privacy concerns are emerging in RL applications that involve vast amounts of private information. Recent studies have demonstrated that DRL can leak private information and be vulnerable to attacks aiming to infer the training environment from an agent’s behaviors without direct access to the environment. To address these privacy concerns, we propose a differentially private DRL approach that obfuscates the agent’s observations from each visited state. This defends against privacy leakage attacks and prevents the inference of the agent’s training environment from its optimized policy. We provide a theoretical analysis and design comprehensive experiments to thoroughly reproduce the privacy leakage attack. Both the theoretical analysis and experimental results demonstrate that our method effectively defends against privacy leakage attacks while maintaining the model utility of the RL agent.

In federated learning (FL), local models are trained independently by clients, local model parameters are shared with a global aggregator or server, and then the updated model is used to initialize the next round of local training. FL and its variants have become synonymous with privacy-preserving distributed machine learning. However, most FL methods have maximization of model accuracy as their sole objective, and rarely are the clients' needs and constraints considered. In this paper, we consider that clients have differing performance expectations and resource constraints, and we assume local data quality can be improved at a cost. In this light, we treat FL in the training phase as a game in satisfaction form that seeks to satisfy all clients' expectations. We propose two novel FL methods, a deep reinforcement learning method and a stochastic method, that embrace this design approach. We also account for the scenario where certain clients can adjust their actions even after being satisfied, by introducing probabilistic parameters in both of our methods. The experimental results demonstrate that our proposed methods converge quickly to a lower cost solution than competing methods. Furthermore, it was found that the probabilistic parameters facilitate the attainment of satisfaction equilibria (SE), addressing scenarios where reaching SEs may be challenging within the confines of traditional games in satisfaction form.

Multi-agent learning involves the interaction of multiple agents with the environment to learn an optimal policy. To enhance learning performance, a commonly used approach is the teacher-student framework, which enables agents to seek advice from other agents. However, the current literature suffers from a common limitation, wherein a student agent can only receive advice from a single teacher agent at each time step, thus constraining the knowledge acquired. Although some methods allow advice from multiple teachers, they require pre-training of the teachers, which is impractical when agents concurrently learn from scratch simultaneously. Additionally, the importance of the agents' connection structures is often disregarded, despite its critical role in the advice-giving process. To overcome these limitations, we propose a novel advising approach utilizing graph neural networks (GNNs). This method models the agents' connection structures, learns the weight of advice, and aggregates inputs from multiple teachers to generate refined advice. Our experiments show that this proposed approach has superior learning performance compared to advising baseline methods.

Federated learning (FL) is a privacy-preserving machine learning (ML) approach that allows clients to perform joint model training without needing to share their individual training data with the central server. When deploying FL for wireless clients with bandwidth and energy constraints, efficient utilization of communication resources is a significant challenge. In this paper, we propose a Deep Reinforcement Learning (DRL)-empowered FL framework for wireless clients that utilizes a Deep Deterministic Policy Gradient (DDPG) agent at the central server to allocate communication bandwidth to each client. The DRL aims to reduce each clients’ transmission energy by considering their respective channels to the server. Model partitioning is used to control bandwidth utilization. The proposed method trades-off the amount of information transmitted by each client and the accuracy of the central model. Simulation results demonstrate that our proposed framework outperforms baseline FL methods operating under the same constraints.

The growth of the Internet of Things (IoT) and the increasing demand for real-time networking have brought about a growing necessity for multiple reinforcement learning (RL) agents to collaboratively train within a shared environment, all working towards common objectives. The multi-agent Advantage Actor-Critic (A2C) algorithm is gaining popularity in Multi-Agent Reinforcement Learning (MARL) systems. However, this approach requires agents to share policy components among neighboring agents due to observations being only partially available to each agent. This practice increases communication overhead and raises privacy concerns. Federated learning (FL), recognized as a privacy-preserving machine learning method, can be applied in the MARL context with a central server aggregating the weights of the agents' actor and critic models. However, this technique assumes that all agents are capable of executing identical actions, which may be impractical. To overcome the aforementioned shortcomings, we introduce a novel FL A2C algorithm called "Advantage Actor Federated Critic (A2FC)". The proposed algorithm streamlines the aggregation of agents' critic models while offloading the training of actor models to the individual agents' local machines. An empirical experiment conducted in an adaptive traffic signal control (ATSC) system demonstrates the method's effectiveness in personalizing agents' actions, preserving agents' privacy during training, and mitigating communication overhead issues.

Generative AI (GenAI) based tutoring assistance has shown great potential in delivering adaptive and personalized education. The programming subjects, boasting a large on-campus student body and an active online presence via discussion boards, are well-suited for evaluating the accessibility, engagement, and satisfaction of GenAI-based tutoring assistance. In this paper, we explore the pros and cons of a GenAI-based feedback system, specifically fine-tuning a Large Language Model (LLM), as a potential tool for providing personalized programming subject matter expertise and guiding students towards solutions with interpretability and visualization. To better address the technical issues students encounter in programming and ensure our model provides accurate and professional debugging solutions, we further develop a visualized interaction tool for code interpretation and prompts optimization. Additionally, this paper compares the effectiveness of online discussion boards and the developed LLM in assisting students’ questions and personalized support.

The utility of multiple reinforcement learning (RL) agents collaboratively training within a shared environment, all working towards common objectives, is increasingly evident within the Internet of Vehicles (IoV). The multi-agent Advantage Actor-Critic (MA2C) algorithm is a prominent example of such a Multi-Agent Reinforcement Learning (MARL) system. However, MA2C requires agents to share policies, such as pairs of states and actions and even trained models, among neighboring agents, to overcome the challenge of agents having only partial observations. Unfortunately, this requirement amplifies the communication overhead and raises privacy concerns. Federated learning (FL), as a privacy-preserving machine learning method, can be applied in the MARL context with a central server aggregating the weights of the agents' models. However, this technique assumes that all agents are capable of executing identical actions, which may be impractical. In this paper, we introduce a novel FL A2C algorithm called Advantage Actor Federated Critic (A2FC). The proposed algorithm streamlines the aggregation of agents' critic models while offloading the training of actor models to the individual agents' local machines. An experiment conducted in an adaptive traffic signal control (ATSC) system demonstrates the method's effectiveness in personalizing agents' actions, preserving agents' privacy during training, and mitigating communication overhead issues.

Online food delivery services today are considered an essential service that gets significant attention worldwide. Many companies and individuals are involved in this field as it offers good income and numerous jobs to the community. In this research, we consider the problem of online food delivery services and how we can increase the number of received orders by couriers and thereby increase their income. Multi-agent reinforcement learning (MARL) is employed to guide the couriers to areas with high demand for food delivery requests. A map of the city is divided into small grids, and each grid represents a small area of the city that has different demand for online food delivery orders. The MARL agent trains and learns which grid has the highest demand and then selects it. Thus, couriers can get more food delivery orders and thereby increase long-term income. While increasing the number of received orders is important, protecting customer location is also essential. Therefore, the Protect User Location Method (PULM) is proposed in this research in order to protect customer location information. The PULM injects differential privacy (DP) Laplace noise based on two parameters: city area size and customer frequency of online food delivery orders. We use two datasets—Shenzhen, China, and Iowa, USA—to demonstrate the results of our experiments. The results show an increase in the number of received orders in the Shenzhen and Iowa City datasets. We also show the similarity and data utility of courier trajectories after we use our obfuscation (PULM) method.

Machine Learning (ML) is a critical tool for data-driven classification and regression tasks. With the increasing availability of computationally powerful wireless edge devices, distributed ML has been in the spotlight in recent years for largescale wireless applications. In that domain, Federated Learning (FL) is attractive as a means to preserve data privacy while enabling joint model training amongst multiple clients with their own private data. However, traditional FL has limitations such as a single point of failure, high communication overhead and uncertain trustworthiness of both clients and aggregator. Decentralized FL methods have been proposed to address these concerns by enabling each client to aggregate model updates from other clients. In this paper, we investigate decentralized FL in a wireless system, taking into account the transmission energy budget of each client which fundamentally limits the range and bandwidth of data communications. To do so, we propose a model partitioning method that highlights the design choices available within the energy constraint - sharing of a larger partition of the model among clients requires transmission range to shrink and therefore sharing with fewer neighboring nodes. It is non-obvious what the best setting of partition size/transmission range is, and we demonstrate that such a setting can be found for particular deployments. We further conducted simulations to validate the method we proposed, and the results demonstrated the practicality of our method in distributed systems operating within an energy budget.

GAN-generated image detection now becomes the first line of defense against the malicious uses of machine-synthesized image manipulations such as deepfakes. Although some existing detectors work well in detecting clean, known GAN samples, their success is largely attributable to overfitting unstable features such as frequency artifacts, which will cause failures when facing unknown GANs or perturbation attacks. To overcome the issue, we propose a robust detection framework based on a novel multi-view image completion representation. The framework first learns various view-to-image tasks to model the diverse distributions of genuine images. Frequency-irrelevant features can be represented from the distributional discrepancies characterized by the completion models, which are stable, generalized, and robust for detecting unknown fake patterns. Then, a multi-view classification is devised with elaborated intra- and inter-view learning strategies to enhance view-specific feature representation and cross-view feature aggregation, respectively. We evaluated the generalization ability of our framework across six popular GANs at different resolutions and its robustness against a broad range of perturbation attacks. The results confirm our method’s improved effectiveness, generalization, and robustness over various baselines.

Machine learning models are vulnerable to data inference attacks, such as membership inference and model inversion attacks. In these types of breaches, an adversary attempts to infer a data record's membership in a dataset or even reconstruct this data record using a confidence score vector predicted by the target model. However, most existing defense methods only protect against membership inference attacks. Methods that can combat both types of attacks require a new model to be trained, which may not be time-efficient. In this paper, we propose a differentially private defense method that handles both types of attacks in a time-efficient manner by tuning only one parameter, the privacy budget. The central idea is to modify and normalize the confidence score vectors with a differential privacy mechanism which preserves privacy and obscures membership and reconstructed data. Moreover, this method can guarantee the order of scores in the vector to avoid any loss in classification accuracy. The experimental results show the method to be an effective and timely defense against both membership inference and model inversion attacks with no reduction in accuracy.

Due to the rapid development of the cloud computing environment, it is widely accepted that cloud servers are important for users to improve work efficiency. Users need to know servers' capabilities and make optimal decisions on selecting the best available servers for users' tasks. We consider the process of learning servers' capabilities by users as a multiagent reinforcement learning process. The learning speed and efficiency in reinforcement learning can be improved by sharing the learning experience among learning agents which is defined as advising. However, existing advising frameworks are limited by the requirement that during advising all learning agents in a reinforcement learning environment must have exactly the same actions. To address the above limitation, this article proposes a novel differentially private advising framework for multiagent reinforcement learning. Our proposed approach can significantly improve the application of conventional advising frameworks when agents have one different action. The approach can also widen the applicable field of advising and speed up reinforcement learning by triggering more potential advising processes among agents with different actions.

Planning is one of the main approaches used to improve agents' working efficiency by making plans beforehand. However, during planning, agents face the risk of having their private information leaked. This paper proposes a novel strong privacy-preserving planning approach for logistic-like problems. This approach outperforms existing approaches by addressing two challenges: 1) simultaneously achieving strong privacy, completeness and efficiency, and 2) addressing communication constraints. These two challenges are prevalent in many real-world applications including logistics in military environments and packet routing in networks. To tackle these two challenges, our approach adopts the differential privacy technique, which can both guarantee strong privacy and control communication overhead. To the best of our knowledge, this paper is the first to apply differential privacy to the field of multi-agent planning as a means of preserving the privacy of agents for logistic-like problems. We theoretically prove the strong privacy and completeness of our approach and empirically demonstrate its efficiency. We also theoretically analyze the communication overhead of our approach and illustrate how differential privacy can be used to control it.

Federated learning is an improved version of distributed machine learning that further offloads operations which would usually be performed by a central server. The server becomes more like an assistant coordinating clients to work together rather than micro-managing the workforce as in traditional DML. One of the greatest advantages of federated learning is the additional privacy and security guarantees it affords. Federated learning architecture relies on smart devices, such as smartphones and IoT sensors, that collect and process their own data, so sensitive information never has to leave the client device. Rather, clients train a sub-model locally and send an encrypted update to the central server for aggregation into the global model. These strong privacy guarantees make federated learning an attractive choice in a world where data breaches and information theft are common and serious threats. This survey outlines the landscape and latest developments in data privacy and security for federated learning. We identify the different mechanisms used to provide privacy and security, such as differential privacy, secure multi-party computation and secure aggregation. We also survey the current attack models, identifying the areas of vulnerability and the strategies adversaries use to penetrate federated systems. The survey concludes with a discussion on the open challenges and potential directions of future work in this increasingly popular learning paradigm.

Reinforcement learning is one of the most important branches of AI. Due to its capacity for self-adaption and decision-making in dynamic environments, reinforcement learning has been widely applied in multiple areas, such as healthcare, data markets, autonomous driving, and robotics. However, some of these applications and systems have been shown to be vulnerable to security or privacy attacks, resulting in unreliable or unstable services. A large number of studies have focused on these security and privacy problems in reinforcement learning. However, few surveys have provided a systematic review and comparison of existing problems and state-of-the-art solutions to keep up with the pace of emerging threats. Accordingly, we herein present such a comprehensive review to explain and summarize the challenges associated with security and privacy in reinforcement learning from a new perspective, namely that of the Markov Decision Process (MDP). In this survey, we first introduce the key concepts related to this area. Next, we cover the security and privacy issues linked to the state, action, environment, and reward function of the MDP process, respectively. We further highlight the special characteristics of security and privacy methodologies related to reinforcement learning. Finally, we discuss the possible future research directions within this area.

Cyber deception is one of the key approaches used to mislead attackers by hiding or providing inaccurate system information. There are two main factors limiting the real-world application of existing cyber deception approaches. The first limitation is that the number of systems in a network is assumed to be fixed. However, in the real world, the number of systems may be dynamically changed. The second limitation is that attackers' strategies are simplified in the literature. However, in the real world, attackers may be more powerful than theory suggests. To overcome these two limitations, we propose a novel differentially private game theoretic approach to cyber deception. In this proposed approach, a defender adopts differential privacy mechanisms to strategically change the number of systems and obfuscate the configurations of systems, while an attacker adopts a Bayesian inference approach to infer the real configurations of systems. By using the differential privacy technique, the proposed approach can 1) reduce the impacts on network security resulting from changes in the number of systems and 2) resist attacks regardless of attackers' reasoning power. The experimental results demonstrate the effectiveness of the proposed approach.

The protection of vehicle trajectory in Vehicular ad hoc network is facing many challenges. Among these challenges, one of the most critical issues is to keep the balance between geographical location protection and semantic location protection. Traditional trajectory protection schemes either only focus on geographical location protection or only semantic location protection. Moreover, when trajectory privacy protection is carried out, each location is often given the same protection. This may lead to sensitive locations under insufficient protection and unimportant locations under overprotection. In this paper, based on differential privacy, we propose an optimized privacy differential privacy scheme with reinforcement learning in vehicular ad hoc network. The proposed scheme can dynamically optimize the privacy budget allocation for each location on the vehicle trajectory to reach a better balance between geolocation obfuscation and semantic security. Experiments results demonstrate that the proposed scheme can reduce the risk of geographical and semantic location leakage, and therefore ensure the balance between the utility and privacy.

Due to the rapid development of the cloud computing environment, it is widely accepted that cloud servers are important for users to improve work efficiency. Users need to know servers’ capabilities and make optimal decisions on selecting the best available servers for users’ tasks. We consider the process that users learn servers’ capabilities as a multi-agent Reinforcement learning process. The learning speed and efficiency in Reinforcement learning can be improved by transferring the learning experience among learning agents which is defined as advising. However, existing advising frameworks are limited by a requirement during experience transfer, which all learning agents in a Reinforcement learning environment must have the completely same available choices, also called actions. To address the above limit, this paper proposes a novel differential privacy agent advising approach in Reinforcement learning. Our proposed approach can significantly improve the conventional advising frameworks’ application when agents’ choices are not the completely same. The approach can also speed up the Reinforcement learning by the increase of possibility of experience transfer among agents with different available choices.

With the rapid development of IoT and smart homes, smart meters have received extensive attention. The third-party applications, such as smart home controlling, dynamic demand-response, power monitoring, etc., can provide services to users based on consumption data of household electricity collected from smart meters. With the emergence of non-intrusive load monitoring, privacy issues from the data of smart meters become more and more severe. Differential privacy is a recognized concept that has become an important standard of privacy preservation for data with personal information. However, the existing privacy protection methods for the data of smart meters that are based on differential privacy sacrifices the actual energy consumption to protect the privacy of users, thus affecting the charging of power suppliers. To solve this problem, we propose a group-based noise adding method, so as to ensure the correct electricity billing. The experiments with two real-world data sets demonstrate that our approach can not only provide a strict privacy guarantee but also improve performance significantly.