We consider a federated reinforcement learning setting involving $M$ agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q}, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just $\tilde{O}(1)$ rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
We present a tutorial focused on the applications of control barrier functions (CBFs) to flight-critical aerospace systems. This tutorial is motivated by the need for mathematically justified approaches to state and input limiting on various aerospace systems, much of which is currently accomplished using heuristic techniques. Here, we illustrate how CBFs – originally motivated by applications in robotics – provide a natural framework for constrained control of aerospace systems. Our tutorial encompasses both the theoretical foundations of CBFs as well as real-world aerospace applications of CBFs.
Deep neural networks (DNNs) are a function approximation tool that are capable of approximating complex dynamic functions by learning complex relationships between the input-output data through training and optimization. Classical DNN training methods utilize numerical optimization tools to improve the function approximation performance. However, in most cases, DNN-based controllers are implemented in an open-loop manner, meaning that the DNN model in the controller does not adapt its weights online and therefore fails to take into consideration any unexpected behavior that could hinder system stability and robustness. To tackle this issue, the authors recently leveraged Lyapunov-based methods to develop innovative real-time update laws that train a DNN's weights in real-time, ensuring stability and improved performance. In this paper, the performance and stability result for DNN-based controllers are further improved by incorporating a Lyapunov-based concurrent learning (CL) inspired term within the real-time DNN update law. A rigorous Lyapunov-based stability analysis was performed to ensure that the proposed DNN-based controller and CL augmented update law results in global exponential convergence of the tracking errors and the DNN weight estimation errors towards an ultimate bound. Thus, for the first time, the proposed approach will ensure both trajectory tracking and convergence of the DNN weights towards their optimal values. Furthermore, simulations were performed that demonstrate the significant potential of the developed DNN-based control system.
Although Koopman operators provide a way to represent autonomous nonlinear systems as linear operators in a high-dimensional space, they do not immediately apply to non-autonomous systems with inputs. State (or output) feedback controller design therefore remains nonconvex in typical formulations, even after the use of existing approximations via bilinear control-affine terms. We address this gap by introducing the Koopman Control Parametrization, a novel parameterization of control-affine dynamical systems that directly includes a feedback controller defined as a linear combination of nonlinear measurements. With this choice, the Koopman operator of the closed-loop system is a bilinear combination of two matrices: one representing the system, and the other the controller. We propose a set of sufficient conditions such that the parametrization holds. Then, we present an algorithm that calculates the feedback matrix via semi-definite programming, producing a Lyapunov-stable closed-loop system with convex optimization. We evaluate the proposed controllers on two canonical examples of control-affine nonlinear systems (inverted pendulums), showing that our approach successfully stabilize both with a proper choice of basis functions. In summary, this manuscript introduces a control synthesis method for stabilization of nonlinear systems that is broadly applicable, can be computed efficiently, is verifiably stable and data-driven.
Recent machine learning research has increasingly focused on equilibrium analysis in non-cooperative games rather than solely on optimal solutions. Many such problems involve shared constraints and can be formulated as Generalized Nash Equilibrium Problems (GNEPs). For strongly monotone games, existing methods compute consensus-based variational GNEs (v-GNEs) by exchanging Lagrange multipliers. We propose a fully distributed continuous-time algorithm for shared linear equality constraints that converges without multiplier exchange and reaches any GNE, reducing communication overhead and improving privacy. Discrete-time schemes are also provided, and the method is validated on a multi-robot placement task.
