We consider a Markov Decision Process (MDP), over a finite or infinite horizon, augmented by so-called (sigma, rho)-burstiness constraints. Such constraints, which had been introduced within the framework of net-work calculus, are meant to limit some additive quantity to a given rate over any time interval, plus a term which allows for occasional and limited bursts. We introduce this class of constraints for MDP models, and formulate the corresponding constrained optimization problems. Due to the burstiness constraints, constrained optimal policies are generally history-dependent. We use a recursive form of the constraints to define an augmented-state model, for which sufficiency of Markov or stationary policies is recovered and the standard theory may be applied, albeit over a larger state space. The analysis is mainly devoted to a characterization of feasible policies, followed by application to the constrained MDP optimization problem. A simple queuing example serves to illustrate some of the concepts and calculations involved. (c) 2023 Elsevier B.V. All rights reserved.

Engine failure is a recurring emergency in General Aviation and fixed-wing UAVs, often requiring the pilot or remote operator to carry out carefully planned glides to safely reach a candidate landing strip. We tackle the problem of minimizing the altitude loss of a thrustless aircraft flying towards a designated target position. Extending previous work on optimal glides without obstacles, we consider here trajectory planning of optimal gliding in the the presence of ground obstacles, while accounting for wind effects. Under simplifying model assumptions, in particular neglecting the effect of turns, we characterize the optimal solution as comprising straight glide segments between iteratively-determined extreme points on the obstacles. Consequently, the optimal trajectory is included in an iteratively-defined reduced visibility graph, and can be obtained by a standard graph search algorithm, such as A$^*$. We further quantify the effect of turns to verify a safe near-optimal glide trajectory. We apply our algorithm on a Cessna 172 model, in realistic scenarios, demonstrating both the altitude-loss optimal trajectory calculation, and determination of airstrip reachability.

Network resource allocation through Network Utility Maximization (NUM) is one of the fundamental problems in the realm of networked systems. In the NUM framework, the network comprises a set of nodes each of which is associated with a utility function, and the goal is to distribute resources across nodes so as to maximize the sum of the nodes utilities. In this paper, we propose a novel backpropagation approach for distributed resource allocation. e internal ow of resources among nodes is governed by the network dynamics, assumed to be captured through a directed acyclic graph (DAG). Control is exercised as an external injection of limited resources at some nodes, where the goal is to determine the optimal amount of resources to be injected at nodes, under the NUM framework. To that aim, we present a novel forward-backward algorithm, inspired by neural network training, wherein ows of resources are transferred during the forward step, and gradients are backpropagated at the backward step. Based on such gradients, the controls are adjusted, considering two variations of the algorithm under synchronous and asynchronous se ings. e proposed algorithms are distributed, in the sense that information in transferred only between neighboring nodes in the network. In addition, they are suitable for continued operation, so that the optimum resource allocation is tracked as conditions gradually change. We formally establish convergence of the proposed algorithms, and numerically compare the speed of convergence under the asynchronous se ing against its synchronous counterpart. Together, our results advance the state-of-the-art in the realm of NUM under nonlinear constraints, indicating how to leverage a backpropagation approach for that matter.

We introduce the Cooperative Multi-Agent Path Finding (Co-MAPF) problem, an extension to the classical MAPF problem, where cooperative behavior is incorporated. In this setting, a group of autonomous agents operate in a shared environment and have to complete cooperative tasks while avoiding collisions with the other agents in the group. This extension naturally models many real-world applications, where groups of agents are required to collaborate in order to complete a given task. To this end, we formalize the Co-MAPF problem and introduce Cooperative Conflict-Based Search (Co-CBS), a CBS-based algorithm for solving the problem optimally for a wide set of Co-MAPF problems. Co-CBS uses a cooperation-planning module integrated into CBS such that cooperation planning is decoupled from path planning. Finally, we present empirical results on several MAPF benchmarks demonstrating our algorithm's properties.

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. A Backpropagation Approach for Distributed Resource Allocation Alexandre Reiffers-Masson, Nahum Shimkin, Daniel Sadoc Menasche, Eitan Altman

In this paper, we present a control strategy for a micro-electro-mechanical gyroscope with a drive mode excited through parametric resonance. The reduced order two degrees-of-freedom model of the device is built, and the drive mode control is implemented using phase-locked loop (PLL) and automatic gain control (AGC) loop. A sense mode vibration control algorithm is developed as well for enhanced sensor performance. The analysis of the drive mode control loops is conducted using the multiple scales method. The robustness of the suggested control loops to parameters perturbation is demonstrated using the model. A simplified linear model of the control loops is shown to predict the device behavior with good accuracy.

Exploration while learning representations is one of the main challenges Deep Reinforcement Learning (DRL) faces today. As the learned representation is dependant in the observed data, the exploration strategy has a crucial role. The popular DQN algorithm has improved significantly the capabilities of Reinforcement Learning (RL) algorithms to learn state representations from raw data, yet, it uses a naive exploration strategy which is statistically inefficient. The Randomized Least Squares Value Iteration (RLSVI) algorithm (Osband et al., 2016), on the other hand, explores and generalizes efficiently via linearly parameterized value functions. However, it is based on hand-designed state representation that requires prior engineering work for every environment. In this paper, we propose a Deep Learning adaptation for RLSVI. Rather than using hand-design state representation, we use a state representation that is being learned directly from the data by a DQN agent. As the representation is being optimized during the learning process, a key component for the suggested method is a likelihood matching mechanism, which adapts to the changing representations. We demonstrate the importance of the various properties of our algorithm on a toy problem and show that our method outperforms DQN in five Atari benchmarks, reaching competitive results with the Rainbow algorithm.

In recent years, there has been an increasing interest in building video summarization tools, where the goal is to automatically create a short summary of an input video that properly represents the original content. We consider shot-based video summarization where the summary consists of a subset of the video shots which can be of various lengths. A straightforward approach to maximize the representativeness of a subset of shots is by minimizing the total distance between shots and their nearest selected shots. We formulate the task of video summarization as an optimization problem with a knapsack-like constraint on the total summary duration. Previous studies have proposed greedy algorithms to solve this problem approximately, but no experiments were presented to measure the ability of these methods to obtain solutions with low total distance. Indeed, our experiments on video summarization datasets show that the success of current methods in obtaining results with low total distance still has much room for improvement. In this paper, we develop ILS-SUMM, a novel video summarization algorithm to solve the subset selection problem under the knapsack constraint. Our algorithm is based on the well-known metaheuristic optimization framework - Iterated Local Search (ILS), known for its ability to avoid weak local minima and obtain a good near-global minimum. Extensive experiments show that our method finds solutions with significantly better total distance than previous methods. Moreover, to indicate the high scalability of ILS-SUMM, we introduce a new dataset consisting of videos of various lengths.

We consider the problem of cooperative multi-agent planning (MAP) in a deterministic environment, with a completely observable state. Most tractable algorithms for MAP problems assume sparse interactions among agents and exploitable problem structure. We consider a specific model for representing interactions among agents using soft cooperation constraints (SCC), which enables a compact representation of symmetric dependencies. We present a two-step planning algorithm that breaks down a multi-agent problem with K agents, to multiple instances of independent single-agent problems, such that the aggregation of the single-agent plans is optimal for the group. We propose an efficient algorithm for computing the single-agent optimal plan under a given set of soft constraints, denoted as the response function. We then utilize a well-known graphical model for efficient min-sum optimization in order to find the optimal aggregation of the single agent response functions. The proposed planning algorithm is complete, optimal, and effective when interactions among the agents are sparse. We further indicate some useful extensions to the basic SCC formulation presented here.

In “Dynamic Scheduling of Multiclass Many-Server Queues with Abandonment: The Generalized cμ/h Rule,” Long, Shimkin, Zhang, and Zhang propose three scheduling policies to cope with any general cost functions and general patience-time distributions. Their first contribution is to introduce the target-allocation policy, which assigns higher priority to customer classes with larger deviation from the desired allocation of the service capacity and prove its optimality for any general queue-length cost functions and patience-time distributions. The Gcμ/h rule, which extends the well-known Gcμ rule by taking abandonment into account, is shown to be optimal for the case of convex queue-length costs and nonincreasing hazard rates of patience. For the case of concave queue-length costs but nondecreasing hazard rates of patience, it is optimal to apply a fixed-priority policy, and a knapsack-like problem is developed to determine the optimal priority order efficiently.

Engine cutoff is a recurring emergency in general aviation. It may be caused by an engine malfunction, fuel leak, or improper aircraft maintenance. Such an event, coupled with adverse weather, may endanger passengers and crew. This work obtains max-range optimal trajectories that exploit to the utmost the airframe dynamic capability, accounting for intense in-plane and crosswinds. This study proves that the optimal trajectory in terms of minimal altitude loss must maintain constant heading and velocity. A novel equation for the optimal glide velocity is then derived, whose unique solution depends only on the aircraft's optimal velocity in still air and the wind vector. An explicit analytical expression is obtained for the optimal velocity in strict crosswinds. The results are demonstrated by applying them to the Cessna 172 airframe, in realistic engine cutoff scenarios, and it is shown that using these results can significantly increase the maximal glide range in scenarios combining strong crosswinds and in-plane winds. This paper formulates a range-maximization algorithm that yields the reachability envelope, the optimal landing site, and flight instructions that enable the pilot to follow the optimal path. These results can also enhance the common charts in glider manuals to account for non-negligible crosswinds.

We study the problem of best arm identification with risk constraints within the setting of fixed confidence pure exploration bandits (PAC bandits). The goal is to stop as fast as possible, and with high confidence return an arm whose mean is -close to the best arm among those that satisfy a risk constraint, namely their α-quantile functions are larger than a threshold β. For this risk-sensitive bandit problem, we propose an algorithm and prove an upper-bound on its sample complexity for the general case of sub-Gaussian arms’ distributions. We also prove a lower-bound for this general case that shows our derived upper-bound is near-optimal (up to logarithmic factors). Both our upper and lower bounds have similar form to the risk-neutral PAC bandits results of (Even-Dar et al. 2006) and (Mannor and Tsitsiklis 2004), respectively. We also prove a lower-bound for our problem when the arms’ distributions are Gaussian, which is smaller than our general lower-bound, but is stronger in the sense that it applies to any instance of the (Gaussian) problem. This lower-bound is in terms of the KL divergence and has similar behavior to the risk-neutral PAC bandits results of (Kaufmann et al. 2016).

Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.

We consider the task of learning control policies for a robotic mechanism striking a puck in an air hockey game. The control signal is a direct command to the robot's motors. We employ a model free deep reinforcement learning framework to learn the motoric skills of striking the puck accurately in order to score. We propose certain improvements to the standard learning scheme which make the deep Q-learning algorithm feasible when it might otherwise fail. Our improvements include integrating prior knowledge into the learning scheme, and accounting for the changing distribution of samples in the experience replay buffer. Finally we present our simulation results for aimed striking which demonstrate the successful learning of this task, and the improvement in algorithm stability due to the proposed modifications.

The global Internet has enabled a massive access of internauts to content. At the same time it allowed individuals to use the Internet in order to distribute content. This introduced new types of competition between content over popularity, visibility, influence, reputation and user attention. The rules of these competitions are new with respect to those of traditional media, and they are determined by the way resources are allocated through network protocols (such as page rank in search engines and recommendation systems that are widely spread in social networks). In this paper we first present in the introduction an overview of some central competition issues both in the Internet as well as in other types of networks. We then describe the model of when to send content in order to maximize the exposure of the content. In the two last sections we finally describe research on two bio-inspired tools that have been used to study various competition aspects.

We consider the arrival timing problem faced by multiclass strategic customers to a single queue. The customers sensitivities to delay as well as service completion time preferences may be heterogeneous and the latter may vary non linearly with time. This captures many realistic settings where customers have preferences on when to arrive at a queue. We consider a fluid setup, so each customer is a point in a continuum and service rate is deterministic. This problem has been well studied in the transportation literature as the bottleneck model and the equilibrium customer arrival profile is shown to uniquely exist using intricate fixed point arguments. We develop a simple, elegant and geometrically insightful iterative method to arrive at this equilibrium profile, and provide an equally simple uniqueness proof. Further, under somewhat stringent assumptions, we arrive at the rate of convergence of the proposed algorithm. The simple geometric proof allows easy incorporation of useful extensions - to illustrate, we consider time varying service rates where the equilibrium profile is easily computed. Further, our results easily extend to the case of customers balking when their costs are above a class dependent threshold.

We consider a game of timing between a random number of content creators, who compete for position and exposure time over an ordered shared medium such as an online classified list. Contents (such as ads, messages, multimedia items, or comments) are ordered according to their submission times, with more recent submissions displayed at the top (and better) positions. The instantaneous effectiveness of each item depends on its current display position, as well as on a time-dependent site exposure function which is common to all. Each content creator may choose the submission time of his or her item within a finite time interval, with the goal of maximizing the total exposure of this item. We formulate the problem as a noncooperative game and analyze its symmetric Nash equilibrium. We show existence of the equilibrium profile, characterize it in terms of a differential boundary value problem, provide sufficient conditions for its uniqueness, and devise a numerical scheme for its computation. We further compute the equilibrium profile explicitly for certain special cases, which include a two-player small match and a Poisson-distributed number of players, and evaluate the social efficiency of these equilibria.

The commonly used Q-learning algorithm combined with function approximation induces systematic overestimations of state-action values. These systematic errors might cause instability, poor performance and sometimes divergence of learning. In this work, we present the AVERAGED TARGET DQN (ADQN) algorithm, an adaptation to the DQN class of algorithms which uses a weighted average over past learned networks to reduce generalization noise variance. As a consequence, this leads to reduced overestimations, more stable learning process and improved performance. Additionally, we analyze ADQN variance reduction along trajectories and demonstrate the performance of ADQN on a toy Gridworld problem, as well as on several of the Atari 2600 games from the Arcade Learning Environment.

We consider a variant of the pure exploration problem in Multi-Armed Bandits, where the goal is to find the arm for which the λ -quantile is maximal. Within the PAC framework, we provide a lower bound on the sample complexity of any (ϵ ,δ ) -correct algorithm, and propose algorithms with matching upper bounds. Our bounds sharpen existing ones by explicitly incorporating the quantile factor λ . We further provide experiments that compare the sample complexity of our algorithms with that of previous works.

The notion of approachability in repeated games with vector payoffs was introduced by Blackwell in the 1950s, along with geometric conditions for approachability and corresponding strategies that rely on computing {\em steering directions} as projections from the current average payoff vector to the (convex) target set. Recently, Abernethy, Batlett and Hazan (2011) proposed a class of approachability algorithms that rely on the no-regret properties of Online Linear Programming for computing a suitable sequence of steering directions. This is first carried out for target sets that are convex cones, and then generalized to any convex set by embedding it in a higher-dimensional convex cone. In this paper we present a more direct formulation that relies on the support function of the set, along with suitable Online Convex Optimization algorithms, which leads to a general class of approachability algorithms. We further show that Blackwell's original algorithm and its convergence follow as a special case.