We present a novel Multi-Agent Reinforcement Learning approach to understand and improve policy development by land-shaping agents, such as governments and institutional bodies. We derive the underlying policy decisions by analyzing the land and developing an intelligent system that proposes optimal land conversion strategies. The aim is an efficient method for allocating residential spaces while considering the dynamic population influx in different regions, jurisdictional constraints, and the intrinsic characteristics of the land. Our main goal is to be sustainable, preserving desirable land types such as forests and fluvial lands while optimizing land organization. We introduce an attractiveness metric that quantifies the proximity to different land types and other factors to optimize land usage. It distinguishes two types of agents: "top-down" agents, which are policymakers and shareholders, and "bottom-up" agents representing individuals or groups with specific housing preferences. Our main objective is to create a synergistic environment where the top-down policy meets the bottom-up preferences to devise a comprehensive land use and conversion strategy. This paper, thus, serves as a pivotal reference point for future urban planning and policy-making processes, contributing to a sustainable and efficient landscape design model.

Real-time strategy (RTS) games present a unique challenge for AI agents due to the combination of several fundamental AI problems. While Deep Reinforcement Learning (DRL) has shown promise in the development of autonomous agents for the genre, existing architectures often struggle with games featuring maps of varying dimensions. This limitation hinders the agent's ability to generalize its learned strategies across different scenarios. This paper proposes a novel approach that overcomes this problem by incorporating Spatial Pyramid Pooling (SPP) within a DRL framework. We leverage the GridNet architecture's encoder-decoder structure and integrate an SPP layer into the critic network of the Proximal Policy Optimization (PPO) algorithm. This SPP layer dynamically generates a standardized representation of the game state, regardless of the initial observation size. This allows the agent to effectively adapt its decision-making process to any map configuration. Our evaluations demonstrate that the proposed method significantly enhances the model's flexibility and efficiency in training agents for various RTS game scenarios, albeit with some discernible limitations when applied to very small maps. This approach paves the way for more robust and adaptable AI agents capable of excelling in sequential decision problems with variable-size observations.

Federated Learning (FL) enables collaborative model training across multiple devices while preserving data privacy. However, it remains susceptible to backdoor attacks, where malicious participants can compromise the global model. Existing defence methods are limited by strict assumptions on data heterogeneity (Non-Independent and Identically Distributed data) and the proportion of malicious clients, reducing their practicality and effectiveness. To overcome these limitations, we propose Robust Knowledge Distillation (RKD), a novel defence mechanism that enhances model integrity without relying on restrictive assumptions. RKD integrates clustering and model selection techniques to identify and filter out malicious updates, forming a reliable ensemble of models. It then employs knowledge distillation to transfer the collective insights from this ensemble to a global model. Extensive evaluations demonstrate that RKD effectively mitigates backdoor threats while maintaining high model performance, outperforming current state-of-the-art defence methods across various scenarios.

We introduce DAMAS (Dynamic Adaptation through Multi-Agent Systems), a novel framework for decision-making in non-stationary environments characterized by varying reward distributions and dynamic constraints. Our framework integrates a multi-agent system with Multi-armed Bandits (MAB) algorithms and Bayesian updates. Each agent in DAMAS specializes in a particular environmental state. The system employs Bayesian estimation to continuously update the probabilities of being in each environmental state, enabling rapid adaptation to changing conditions. Our evaluation of DAMAS included both synthetic environments and real-world web server workloads.

Ad-hoc teamwork models are crucial for solving distributed tasks in environments with unknown teammates. In order to improve performance, agents may collaborate in the same environment, trusting each other and exchanging information. However, what happens if there is an impostor among the team? In this paper, we present BAE, a novel and efficient framework for online planning and estimation within ad-hoc teamwork domains where there is an adversarial agent disguised as a teammate. Our approach considers the identification of the impostor through a process we term "Q-valued Bayesian Estimation''. BAE can identify the adversary at the same time the agent performs ad-hoc estimation in order to improve coordination. Our results show that BAE has superior accuracy and faster reasoning capabilities in comparison to the state-of-the-art.

This paper introduces Dynamic Bayesian Optimisation for Multi-Arm Bandits (DBO-MAB), an algorithm that dynamically adapts hyperparameters of multi-arm bandit algorithms using incremental Bayesian optimisation. DBO-MAB addresses the challenge of tuning hyperparameters in uncertain and dynamic environments, particularly for applications like web server optimisation. It uses a dynamic range adjustment approach based on the interquartile mean (IQM) of observed rewards to focus the search space on promising regions. Evaluated across diverse static and dynamic environments, DBO-MAB outperforms state-of-the-art algorithms such as Boot-strapped UCB and f-Discounted-Sliding-Window Thompson Sampling, reducing average response time by approximate to 55%.

We propose ORAA, a novel online incentive algorithm that guides agents in a property-based MARL domain to act sustainably with a common pool of resources. ORAA, uses our proposed P-MADDPG model to learn and make decisions over the decentralised agents. We test our solutions in our novel domain, the "Pollinators' Game'', which simulates a property-based MARL scenario and its incentivisation dynamics. We show significant improvement in the incentives' cost-efficiency when using learned models that approximate the behaviour of each agent instead of simulating their true models.

Reinforcement Learning (RL) has achieved remarkable success in safety-critical areas, but it can be weakened by adversarial attacks. Recent studies have introduced "smoothed policies" to enhance its robustness. Yet, it is still challenging to establish a provable guarantee to certify the bound of its total reward. Prior methods relied primarily on computing bounds using Lipschitz continuity or calculating the probability of cumulative reward being above specific thresholds. However, these techniques are only suited for continuous perturbations on the RL agent's observations and are restricted to perturbations bounded by the l(2)-norm. To address these limitations, this paper proposes a general blackbox certification method, called ReCePS, which is capable of directly certifying the cumulative reward of the smoothed policy under various l(p)-norm bounded perturbations. Furthermore, we extend our methodology to certify perturbations on action spaces. Our approach leverages f-divergence to measure the distinction between the original distribution and the perturbed distribution, subsequently determining the certification bound by solving a convex optimisation problem. We provide a comprehensive theoretical analysis and run experiments in multiple environments. Our results show that our method not only improves the tightness of certified lower bound of the mean cumulative reward but also demonstrates better efficiency than state-of-the-art methods.

Video stabilization plays a central role to improve videos quality. However, despite the substantial progress made by these methods, they were, mainly, tested under standard weather and lighting conditions, and may perform poorly under adverse conditions. In this paper, we propose a synthetic-aware adverse weather robust algorithm for video stabilization that does not require real data and can be trained only on synthetic data. We also present Silver, a novel rendering engine to generate the required training data with an automatic ground-truth extraction procedure. Our approach uses our specially generated synthetic data for training an affine transformation matrix estimator avoiding the feature extraction issues faced by current methods. Additionally, since no video stabilization datasets under adverse conditions are available, we propose the novel VSAC105Real dataset for evaluation. We compare our method to five state-of-the-art video stabilization algorithms using two benchmarks. Our results show that current approaches perform poorly in at least one weather condition, and that, even training in a small dataset with synthetic data only, we achieve the best performance in terms of stability average score, distortion score, success rate, and average cropping ratio when considering all weather conditions. Hence, our video stabilization model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.

Flood risk management usually hinges on accurate water level identification in urban streams such as rivers or creeks. Although research has emphasised the applicability of ultrasonic sensors as a contactless technology for sensor-based water level monitoring, Light Detection and Ranging (LiDAR) sensors are less sensitive to weather conditions that typically happen during flood events, such as dust, fog and rainfall. However, there has been little research on the applicability of LiDAR sensors in this field. No previous literature has analysed the impact of complicating variables on the quality of predictions or evaluated the possible benefits of using a combined approach with Inertial Measurement Units (IMU) and machine learning to produce superior predictions. In this work, we collected a dataset in a laboratory condition synchronising data from a LiDAR, an ultrasonic sensor and an IMU in an experimental device. We controlled the incidence angle, the distance, and the water turbidity to analyse their effect on the predictions. Traditional machine-learning techniques were evaluated as models to combine data from distance and inertial sensors, reducing the error rates compared to individual sensors’ predictions. Results indicated a sharp drop in the mean absolute error, root mean squared error and coefficient of determination for all water turbidity and incidence angles considered, especially when tree-based ensembles were used. The ultrasonic sensor led to improved results for low water turbidity and increased incidence angle, but statistically significant differences were not found in the other cases.

The investigation of stateful service mobility across networking infrastructures is becoming increasingly important as applications require stateful services capable of migrating from centralized cloud data centers to edge computing infrastructures. State-of-the-art approaches propose either machine learning solutions for stateless service placement or stateful service mobility using static and inflexible state management strategies. We believe these approaches fall short of addressing the full length of the stateful service mobility problem. In this paper, we revisit an emerging concept named self-distributing systems, where a local executing application manages to detach some of its constituent (often stateful) components and place them in remote machines as a solution for stateful service mobility. In previous work, a machine learning approach to support self-distributing systems has not been thoroughly investigated. We model the distribution of stateful components across networking infrastructures as a multi-armed bandits problem and use the UCB1 algorithm to solve it as a first attempt at a flexible solution for stateful service mobility. We conclude the paper by discussing the main challenges and opportunities in this area.

Supervised machine learning methods require large-scale training datasets to perform well in practice. Synthetic data has been showing great progress recently and has been used as a complement to real data. However, there is yet a great urge to assess the usability of synthetically generated data. To this end, we propose a novel UCB-based training procedure combined with a dynamic usability metric. Our proposed metric integrates low-level and high-level information from synthetic images and their corresponding real and synthetic datasets, surpassing existing traditional metrics. By utilizing a UCB-based dynamic approach ensures continual enhancement of model learning. Unlike other approaches, our method effectively adapts to changes in the machine learning model's state and considers the evolving utility of training samples during the training process. We show that our metric is an effective way to rank synthetic images based on their usability. Furthermore, we propose a new attribute-aware bandit pipeline for generating synthetic data by integrating a Large Language Model with Stable Diffusion. Quantitative results show that our approach can boost the performance of a wide range of supervised classifiers. Notably, we observed an improvement of up to 10 traditional approaches, demonstrating the effectiveness of our approach. Our source code, datasets, and additional materials are publically available at https://github.com/A-Kerim/Synthetic-Data-Usability-2024.

We propose ORAA, a novel incentive-driven algorithm that guides agents in a property-based Multi-Agent Reinforcement Learning domain to act sustainably considering a common pool of resources in an online manner. ORAA implements our proposed P-MADDPG model to learn and make decisions over the decentralised agents. We test our solutions in our novel domain, the “Pollinators’ Game”, which simulates a property-based scenario and the incentivisation dynamics. We show significant improvement in the incentives’ cost-efficiency, reducing the budget spent while increasing the collection of rewards by individual agents. Besides that, our application shows better results when using learned (approximated) models instead of using and simulating the true models of each agent for planning, saving up to 50

The growth of videos in our digital age and the users' limited time raise the demand for processing untrimmed videos to produce shorter versions conveying the same information. Despite the remarkable progress that summarization methods have made, most of them can only select a few frames or skims, creating visual gaps and breaking the video context. This paper presents a novel weakly-supervised methodology based on a reinforcement learning formulation to accelerate instructional videos using text. A novel joint reward function guides our agent to select which frames to remove and reduce the input video to a target length without creating gaps in the final video. We also propose the Extended Visually-guided Document Attention Network (VDAN+), which can generate a highly discriminative embedding space to represent both textual and visual data. Our experiments show that our method achieves the best performance in Precision, Recall, and F1 Score against the baselines while effectively controlling the video's output length.

This work presents an autonomous approach that explores the dynamic generation of relaxing soundscapes for games and artistic installations. Differently from past works, this system can generate music and images simultaneously, preserving human intent and coherency. We present our algorithm for the generation of audiovisual instances and also a system based on this approach, verifying the quality of the outcomes it can produce in light of current approaches for the generation of images and music. We also instigate the discussion around the new paradigm in arts, where the creative process is delegated to autonomous systems, with limited human participation. Our user study (N=74) shows that our approach overcomes current deep learning models in terms of quality, being recognized as human production, as if the outcome were being generated out of an endless musical improvisation performance.

Abstract Coordination algorithms are required to minimise congestion when every robot in a robotic swarm has a common target area to visit. Some of these algorithms use artificial potential fields to enable path planning to become distributed and local. An efficiency measure for comparing them is the time to complete a task in relation to the number of individuals in the swarm. In order to compare distinct solutions as the swarm grows, experiments with different numbers of robots must be simulated to form a plot of the function of the task completion time versus the number of robots or other parameters. Nevertheless, plotting it for many robots through simulation is time-consuming. Additionally, the inference of a global swarm behaviour as the task completion time from the local individual robot motion controller based on potential fields and other dynamical variables is intractable and requires experimental analysis. Based on that, we present equations for estimating the expected task completion time of state-of-the-art algorithms, robots using only attractive and repulsive force fields and mixed teams for the common target area problem in robotic swarms with not only the number of robots as input but also environment- and algorithm-related global variables, such as the size of the common target area and the working area, average speed and average distance between the robots. This paper is a fundamental first step to start a discussion on how better approximations can be achieved and which mathematical theories about local-to-global analysis are better suited to this problem.

Real-time strategy games present a significant challenge for artificial game-playing agents by combining several fundamental AI problems. Despite the difficulties, attempts to create autonomous agents using Deep Reinforcement Learning have been successful, with bots like AlphaStar beating even expert human players. Many RTS games include several distinct world maps with different dimensions, which may affect the agent's observation and the representation of game states. However, most current architectures suffer from fixed input sizes or require extensive and complex training. In this paper, we overcome these limitations by combining Grid-Wise Control with Spatial Pyramid Pooling (SPP). Specifically, we employ the encoder-decoder framework provided by the GridNet architecture and enhance the critic component of PPO by adding an SPP layer to it. The new layer generates a standardized representation of any game state regardless of the initial observation dimensions, allowing the agent to act on any map. Our evaluation demonstrates that our proposed method improves the models' flexibility and provides a more effective and efficient solution for training autonomous agents in multiple RTS game scenarios.

When a large number of robots try to reach a common area, congestions happen, causing severe delays. To minimise congestion in a robotic swarm system, traffic control algorithms must be employed in a decentralised manner. Based on strategies aimed to maximise the throughput of the common target area, we developed two novel algorithms for robots using artificial potential fields for obstacle avoidance and navigation. One algorithm is inspired by creating a queue to get to the target area (Single Queue Former — SQF), while the other makes the robots touch the boundary of the circular area by using vector fields (Touch and Run Vector Fields — TRVF). We performed simulation experiments to show that the proposed algorithms are bounded by the throughput of their inspired theoretical strategies and compare the two novel algorithms with state-of-art algorithms for the same problem (PCC, EE and PCC–EE). The SQF algorithm significantly outperforms all other algorithms for a large number of robots or when the circular target region radius is small. TRVF, on the other hand, is better than SQF only for a limited number of robots and outperforms only PCC for numerous robots. However, it allows us to analyse the potential impacts on the throughput when transferring an idea from a theoretical strategy to a concrete algorithm that considers changing linear speeds and distances between robots.

Federated Learning (FL) is essential for building global models across distributed environments. However, it is significantly vulnerable to data and model poisoning attacks that can critically compromise the accuracy and reliability of the global model. These vulnerabilities become more pronounced in heterogeneous environments, where clients’ data distributions vary broadly, creating a challenging setting for maintaining model integrity. Furthermore, malicious attacks can exploit this heterogeneity, manipulating the learning process to degrade the model or even induce it to learn incorrect patterns. In response to these challenges, we introduce RFCL, a novel Robust Federated aggregation method that leverages CLustering and cosine similarity to select similar cluster models, effectively defending against data and model poisoning attacks even amidst high data heterogeneity. Our experiments assess RFCL’s performance against various attacker numbers and Non-IID degrees. The findings reveal that RFCL outperforms existing robust aggregation methods and demonstrates the capability to defend against multiple attack types.

Recent years have witnessed increasing interest in adversarial attacks on images, while adversarial video attacks have seldom been explored. In this paper, we propose a sparse adversarial attack strategy on videos (DeepSAVA). Our model aims to add a small human-imperceptible perturbation to the key frame of the input video to fool the classifiers. To carry out an effective attack that mirrors real-world scenarios, our algorithm integrates spatial transformation perturbations into the frame. Instead of using the lp norm to gauge the disparity between the perturbed frame and the original frame, we employ the structural similarity index (SSIM), which has been established as a more suitable metric for quantifying image alterations resulting from spatial perturbations. We employ a unified optimisation framework to combine spatial transformation with additive perturbation, thereby attaining a more potent attack. We design an effective and novel optimisation scheme that alternatively utilises Bayesian Optimisation (BO) to identify the most critical frame in a video and stochastic gradient descent (SGD) based optimisation to produce both additive and spatial-transformed perturbations. Doing so enables DeepSAVA to perform a very sparse attack on videos for maintaining human imperceptibility while still achieving state-of-the-art performance in terms of both attack success rate and adversarial transferability. Furthermore, built upon the strong perturbations produced by DeepSAVA, we design a novel adversarial training framework to improve the robustness of video classification models. Our intensive experiments on various types of deep neural networks and video datasets confirm the superiority of DeepSAVA in terms of attacking performance and efficiency. When compared to the baseline techniques, DeepSAVA exhibits the highest level of performance in generating adversarial videos for three distinct video classifiers. Remarkably, it achieves an impressive fooling rate ranging from 99.5% to 100% for the I3D model, with the perturbation of just a single frame. Additionally, DeepSAVA demonstrates favourable transferability across various time series models. The proposed adversarial training strategy is also empirically demonstrated with better performance on training robust video classifiers compared with the state-of-the-art adversarial training with projected gradient descent (PGD) adversary.