The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, till now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated Q-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN is theoretically grounded and that it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari 2600 games and MuJoCo continuous control problems.

In robotics, many different approaches ranging from classical planning over optimal control to reinforcement learning (RL) are developed and borrowed from other fields to achieve reliable control in diverse tasks. To get a clear understanding of their individual strengths and weaknesses and their applicability in real-world robotic scenarios, it is important to benchmark and compare their performances not only in a simulation but also on real hardware. The second AI Olympics with RealAIGym competition was held at the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024) to contribute to this cause and evaluate different controllers according to their ability to solve a dynamic control problem on an underactuated double-pendulum system (Figure 1) with chaotic dynamics. This article describes the four different RL methods submitted by the participating teams; presents their performance in the swing-up task on a real double pendulum, measured against various criteria; and discusses their transferability from simulation to real (sim-to-real) hardware and their robustness to external disturbances.

Perception through touch relies on active exploration strategies that adapt to perceptual goals and object properties. We investigated how information gathering through exploratory procedures (EPs) is organised in haptic shape perception, in particular with respect to the selection of EPs and the prioritisation of shape features - and how these processes are influenced by material properties. In two experiments, participants used a single finger to explore rigid or deformable shapes and judged shape similarity. In Experiment 1, analysis of EPs showed that while contour-following was the dominant strategy for acquiring shape information, participants flexibly supplemented it with additional EPs such as tapping and scanning when exploring deformable shapes. In Experiment 2, trajectory and force data showed a mostly consistent prioritisation of concave regions across exploration parameters (dwell time, velocity, distance, force), regardless of material. Force was modulated by material properties, with participants applying more force to rigid shapes and less to deformable ones. Analysis of the temporal sequence of movements further showed comparable contour following patterns across materials, but slightly more fragmented movements for deformable shape and smoother, more complete contour tracing for rigid ones, and some shape-specific effects. Together, the findings demonstrate that haptic shape perception is both adaptive and structured: exploratory strategies adjust to material contexts, yet perception and exploration remain anchored by stable geometrical cues, particular concavities, supporting reliable shape perception across contexts.

Accurate in-hand pose estimation is crucial for robotic object manipulation, but visual occlusion remains a major challenge for vision-based approaches. This paper presents an approach to robotic in-hand object pose estimation, combining visual and tactile information to accurately determine the position and orientation of objects grasped by a robotic hand. We address the challenge of visual occlusion by fusing visual information from a wrist-mounted RGB-D camera with tactile information from vision-based tactile sensors mounted on the fingertips of a robotic gripper. Our approach employs a weighting and sensor fusion module to combine point clouds from heterogeneous sensor types and control each modality's contribution to the pose estimation process. We use an augmented Iterative Closest Point (ICP) algorithm adapted for weighted point clouds to estimate the 6D object pose. Our experiments show that incorporating tactile information significantly improves pose estimation accuracy, particularly when occlusion is high. Our method achieves an average pose estimation error of 7.5 mm and 16.7 degrees, outperforming vision-only baselines by up to 20 perform precise object manipulation in a real-world insertion task.

Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called Adaptive $Q$-Network (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several $Q$-functions, each one trained with different hyperparameters, which are updated online using the $Q$-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari $2600$ games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability.

Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing softbody deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we propose a machine learning approach using a U-net architecture to predict force distributions directly from the sensor’s raw images. Our model, trained on force distributions inferred from Finite Element Analysis (FEA), demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application. The codebase, dataset and models are open-sourced and available at https://feats-ai.github.io.

One of the most important object properties that humans and robots perceive through touch is hardness. This paper investigates information-theoretic active sampling strategies for sample-efficient hardness classification with vision-based tactile sensors. We evaluate three probabilistic classifier models and two model-uncertainty-based sampling strategies on a robotic setup as well as on a previously published dataset of samples collected by human testers. Our findings indicate that the active sampling approaches, driven by uncertainty metrics, surpass a random sampling baseline in terms of accuracy and stability. Additionally, while in our human study, the participants achieve an average accuracy of 48.00 the same set of objects, demonstrating the effectiveness of vision-based tactile sensors for object hardness classification.

Value-based Reinforcement Learning (RL) methods hinge on the application of the Bellman operator, which needs to be approximated from samples. Most approaches consist of an iterative scheme alternating the application of a Bellman iteration and a subsequent projection step in the considered function space. In this paper, we propose a new perspective by introducing iterated Deep $Q$-Network (iDQN), a novel DQN-based algorithm that aims to obtain an approximation of several consecutive Bellman iterations at once. To this end, iDQN leverages the online network of DQN to build a target for a second online network, which in turn serves as a target for a third online network, and so forth, thereby taking into account future Bellman iterations. This entails that iDQN allows for better learning of the Bellman iterations than DQN, while using the same number of gradient steps. We theoretically prove the benefit of iDQN in terms of error propagation under the lens of approximate value iteration. Then, we evaluate iDQN against relevant baselines on $54$ Atari $2600$ games, showing that iDQN outperforms DQN while being orthogonal to more advanced DQN-based approaches.

Training robot policies in simulation is becoming increasingly popular; nevertheless, a precise, reliable, and easy-to-use tactile simulator for contact-rich manipulation tasks is still missing. To close this gap, we develop TacEx – a modular tactile simulation framework. We embed a state-of-the-art soft-body simulator for contacts named GIPC and vision-based tactile simulators Taxim and FOTS into Isaac Sim to achieve robust and plausible simulation of the visuotactile sensor GelSight Mini. We implement several Isaac Lab environments for Reinforcement Learning (RL) leveraging our TacEx simulation, including object pushing, lifting, and pole balancing. We validate that the simulation is stable and that the high-dimensional observations, such as the gel deformation and the RGB images from the GelSight camera, can be used for training. The code, videos, and additional results will be released online https://sites.google.com/view/tacex.

As artificial intelligence gains new capabilities, it becomes important to evaluate it on real-world tasks. In particular, the fields of robotics and reinforcement learning (RL) are lacking in standardized benchmarking tasks on real hardware. To facilitate reproducibility and stimulate algorithmic advancements, we held an AI Olympics competition at IJCAI 2023 conference based on the double pendulum system in the RealAIGym project where the participants were asked to develop a controller for the swing up and stabilization task. This paper presents the methods and results from the top participating teams and provides insights into the real-world performance of RL algorithms with respect to a baseline time-varying LQR controller.

This research explores the SL-Block system within an architecture framework by embracing building modularity, combinatorial design, topological interlocking, machine learning, and tactile sensor-based robotic assembly. The SL-Block, composed of S and L-shaped tetracubes, possesses a unique self-interlocking feature that allows for reversible joining and the creation of various 2D or 3D structures. In architecture modularity, the high degree of reconfigurability and adaptability of the SL-Block system introduces a new element of interest. Unlike modularization strategies that emphasize large-scale volumetric modules or standardized building components, using small-scale generic building blocks provides greater flexibility in maximizing design variations and reusability. Furthermore, the serial repetition and limited connectivity of building elements reduce the efforts required for bespoke manufacturing and automated assembly. In this article, we present our digital design and robotic assembly strategies for developing dry-jointed modular construction with SL-Blocks. Drawing on combinatorics and graph theory, we propose computational design methods that can automatically generate hierarchical SL-Block assemblies from given shapes. To address the physical complexities of contact-rich assembly tasks, we develop robotics using two distinct methods: pre-programmed assembly and sensor-based reinforcement learning. Through a series of demonstrators, we showcase the ability of SL-Blocks not only to reconfigure conventional building tectonics but also to create new building configurations.

Value-based Reinforcement Learning (RL) methods rely on the application of the Bellman operator, which needs to be approximated from samples. Most approaches consist of an iterative scheme alternating the application of the Bellman operator and a subsequent projection step onto a considered function space. However, we observe that these algorithms can be improved by considering multiple iterations of the Bellman operator at once. Thus, we introduce iterated $Q$-Networks (iQN), a novel approach that learns a sequence of $Q$-function approximations where each $Q$-function serves as the target for the next one in a chain of consecutive Bellman iterations. We demonstrate that iQN is theoretically sound and show how it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate its advantages on Atari $2600$ games and in continuous-control MuJoCo environments.

The perception of material/object properties plays a fundamental role in our daily lives. Previous research has shown that individuals use distinct and consistent patterns of hand movements, known as exploratory procedures (EPs), to extract perceptual information relevant to specific material/object properties. Here, we investigated the variation in EP usage across different tasks involving objects that varied in task-relevant properties (shape or deformability) as well as in task-irrelevant properties (deformability or texture). Participants explored 1 reference object and 2 test objects with a single finger before selecting the test object that was most similar to the reference. We recorded their finger movements during explorations, and these movements were then categorised into different EPs. Our results show strong task-dependent usage of EPs, even when exploration was confined to a single finger. Furthermore, within a given task, EPs varied as a function of material/object properties unrelated to the primary task. These variations suggest that individuals flexibly adapt their exploration strategies to obtain consistent and relevant information.

We tackle the recently introduced benchmark for whole-body humanoid control HumanoidBench using MuJoCo MPC. We find that sparse reward functions of HumanoidBench yield undesirable and unrealistic behaviors when optimized; therefore, we propose a set of regularization terms that stabilize the robot behavior across tasks. Current evaluations on a subset of tasks demonstrate that our proposed reward function allows achieving the highest HumanoidBench scores while maintaining realistic posture and smooth control signals. Our code is publicly available and will become a part of MuJoCo MPC, enabling rapid prototyping of robot behaviors.

This paper explores active sensing strategies that employ vision-based tactile sensors for robotic perception and classification of fabric textures. We formalize the active sampling problem in the context of tactile fabric recognition and provide an implementation of information-theoretic exploration strategies based on minimizing predictive entropy and variance of probabilistic models. Through ablation studies and human experiments, we investigate which components are crucial for quick and reliable texture recognition. Along with the active sampling strategies, we evaluate neural network architectures, representations of uncertainty, influence of data augmentation, and dataset variability. By evaluating our method on a previously published Active Clothing Perception Dataset and on a real robotic system, we establish that the choice of the active exploration strategy has only a minor influence on the recognition accuracy, whereas data augmentation and dropout rate play a significantly larger role. In a comparison study, while humans achieve 66.9% recognition accuracy, our best approach reaches 90.0% in under 5 touches, highlighting that vision-based tactile sensors are highly effective for fabric texture recognition.

Humanoids have the potential to be the ideal embodiment in environments designed for humans. Thanks to the structural similarity to the human body, they benefit from rich sources of demonstration data, e.g., collected via teleoperation, motion capture, or even using videos of humans performing tasks. However, distilling a policy from demonstrations is still a challenging problem. While Diffusion Policies (DPs) have shown impressive results in robotic manipulation, their applicability to locomotion and humanoid control remains underexplored. In this paper, we investigate how dataset diversity and size affect the performance of DPs for humanoid whole-body control. In a simulated IsaacGym environment, we generate synthetic demonstrations by training Adversarial Motion Prior (AMP) agents under various Domain Randomization (DR) conditions, and we compare DPs fitted to datasets of different size and diversity. Our findings show that, although DPs can achieve stable walking behavior, successful training of locomotion policies requires significantly larger and more diverse datasets compared to manipulation tasks, even in simple scenarios.

The ``AI Olympics with RealAIGym'' competition challenges participants to stabilize chaotic underactuated dynamical systems with advanced control algorithms. In this paper, we present a novel solution submitted to IROS'24 competition, which builds upon Soft Actor-Critic (SAC), a popular model-free entropy-regularized Reinforcement Learning (RL) algorithm. We add a `context' vector to the state, which encodes the immediate history via a Convolutional Neural Network (CNN) to counteract the unmodeled effects on the real system. Our method achieves high performance scores and competitive robustness scores on both tracks of the competition: Pendubot and Acrobot.

Value-based reinforcement learning~(RL) methods strive to obtain accurate approximations of optimal action-value functions. Notoriously, these methods heavily rely on the application of the optimal Bellman operator, which needs to be approximated from samples.Most approaches consider only a single Bellman iteration, which limits their power. In this paper, we introduce iterated Deep Q-Network (iDQN), a new DQN-based algorithm that incorporates several consecutive Bellman iterations into the training loss. iDQN leverages the online network of DQN to build a target for a second online network, which in turn serves as a target for a third online network, etc., thereby taking into account future Bellman iterations. While using the same number of gradient steps, iDQN allows for better learning of the Bellman iterations compared to DQN. We evaluate iDQN against relevant baselines on 54 Atari 2600 games to showcase its benefit in terms of approximation error and performance. iDQN greatly outperforms its closest baselines, DQN and Random Ensemble Mixture, while being orthogonal to more advanced DQN-based approaches.

Solving the Hamilton-Jacobi-Bellman equation is important in many domains including control, robotics and economics. Especially for continuous control, solving this differential equation and its extension the Hamilton-Jacobi-Isaacs equation, is important as it yields the optimal policy that achieves the maximum reward on a give task. In the case of the Hamilton-Jacobi-Isaacs equation, which includes an adversary controlling the environment and minimizing the reward, the obtained policy is also robust to perturbations of the dynamics. In this paper we propose continuous fitted value iteration (cFVI) and robust fitted value iteration (rFVI). These algorithms leverage the non-linear control-affine dynamics and separable state and action reward of many continuous control problems to derive the optimal policy and optimal adversary in closed form. This analytic expression simplifies the differential equations and enables us to solve for the optimal value function using value iteration for continuous actions and states as well as the adversarial case. Notably, the resulting algorithms do not require discretization of states or actions. We apply the resulting algorithms to the Furuta pendulum and cartpole. We show that both algorithms obtain the optimal policy. The robustness Sim2Real experiments on the physical systems show that the policies successfully achieve the task in the real-world. When changing the masses of the pendulum, we observe that robust value iteration is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm. Videos of the experiments are shown at https://sites.google.com/view/rfvi

Geometrically interlocking building elements offer unique opportunities for the construction industry to minimize waste, maximize reuse and reduce its carbon footprint. Dry-joint elements allow for fast robotic assembly, disassembly, and reassembly of complex structures out of prefabricated modules. The article discusses the robotic assembly of SL blocks, which are modules that interlock with each other. The assembly process is challenging due to the intricate assembly sequences, design hierarchies, numerous potential grasping points, contact-rich assemblies, and instability until the assembly is completed. To implement the robotic assembly of self-interlocking structures, advances in several research scopes are necessary, including geometry, algorithms, and implementation. The study aimed to co-evolve SL block geometry and robotic grippers using a robot-oriented design approach to compensate for tolerances and add self-centering features. The article also presents an assembly environment that includes mechanical fixation cubes to secure the SL blocks during assembly and support cantilevered sections of the structure until the aggregation is stable. The article presents algorithms for robotic task and motion planning and the generation of assembly sequences inspired by recursive algorithms to design 3D interlocking puzzles and directional blocking graphs. The study highlights how the voxel-based representation of complex geometries can be used to prepare directional blocking graphs. The research provides insights into improving the element’s geometry and robotic assembly, which could have significant applications in other studies.