This contribution ( i ) describes an open-source, physics-based simulation infrastructure that can be used to learn and test control policies in off-road navigation; and ( ii ) demonstrates the use of the simulation platform in an end-to-end learning exercise that relies on simulated sensor data fusion (camera, GPS and IMU). For ( i ), the 0.5 million lines of open-source code support vehicle dynamics (wheeled/tracked vehicles, rovers), deformable non-deformable terrains, and virtual sensing. The library has a Python API for interfacing with existing Machine Learning frameworks. For (ii) , we use a Gator off-road vehicle to demonstrate how a policy learned on non-deformable terrain performs when used in hilly conditions while navigating around a course of randomly placed obstacles on deformable terrain. The hilly terrain covers an 80×80 m patch and the soil can be controlled by the user to assume various behavior, e.g. non-deformable, deformable hard (silt-like), deformable soft (snow-like), etc. To the best of our knowledge, there is no other open-source, physics-based engine that can be used to simulate off-road mobility of autonomous agents operating on deformable terrains. The results reported herein can be reproduced with models and data available in a public repository (UW-Madison Simulation Based Engineering Laboratory, Supporting models, scripts, data, https://go.wisc.edu/arflqq , 2021 ). Animations associated with the tests run are available online (UW-Madison Simulation Based Engineering Laboratory, Supporting simulations, https://go.wisc.edu/256xb9 , 2021 ).

This work presents a numerical method for the solution of variational inequalities arising in nonsmooth flexible multibody problems that involve set‐valued forces. For the special case of hard frictional contacts, the method solves a second order cone complementarity problem. We ground our algorithm on the Alternating Direction Method of Multipliers (ADMM), an efficient and robust optimization method that draws on few computational primitives. In order to improve computational performance, we reformulated the original ADMM scheme in order to exploit the sparsity of constraint jacobians and we added optimizations such as warm starting and adaptive step scaling. The proposed method can be used in scenarios that pose major difficulties to other methods available in literature for complementarity in contact dynamics, namely when using very stiff finite elements and when simulating articulated mechanisms with odd mass ratios. The method can have applications in the fields of robotics, vehicle dynamics, virtual reality, and multiphysics simulation in general.

We describe a simulation environment that enables the development and testing of control policies for off-road mobility of autonomous agents. The environment is demonstrated in conjunction with the design and assessment of a reinforcement learning policy that uses sensor fusion and inter-agent communication to enable the movement of mixed convoys of human-driven and autonomous vehicles. Policies are learned on rigid terrain and are subsequently shown to transfer successfully to hard (silt-like) and soft (snow-like) deformable terrains. The enabling simulation environment is developed from the high fidelity, physics-based simulation engine Chrono. Five Chrono modules are employed herein: Chrono::Engine, Chrono::Vehicle, PyChrono, SynChrono and Chrono::Sensor. Vehicle’s are modeled using Chrono::Engine and Chrono::Vehicle and deployed on deformable terrain within the training/testing environment. Utilizing the Python interface to the C++ Chrono API called PyChrono and OpenAI Gym’s supporting infrastructure, training is conducted in a GymChrono learning environment. The GymChrono-generated policy is subsequently deployed for testing in SynChrono, a scalable, cluster-deployable multi-agent testing infrastructure built on MPI. SynChrono facilitates inter-agent communication and maintains time and space coherence between agents. A sensor modeling tool, Chrono::Sensor, supplies sensing data that is used to inform agents during the learning and inference processes. The software stack and the Chrono simulator are both open source. Relevant movies: [1].

gym-chrono is a set of simulated environments for Deep Reinforcement Learning (DRL) extending OpenAI Gym (Brockman et al., Openai gym, 2016) with robotics and autonomous driving tasks. The physics of these environments is simulated thanks to Project Chrono (Tasora et al., Chrono: An open source multi-physics dynamics engine, 2016), an open-source multi-physics simulation engine capable of simulating Multibody Dynamics with constraints and smooth or non-smooth contacts. The most used Deep Learning frameworks (such as PyTorch and Tensorflow) have Python API, and thus using Python to implement DRL algorithms is the most convenient option. For this reason, a condition for the creation of these environments has been the development of PyChrono, a Python module consisting of the Python bindings to Project Chrono C++ API, to effectively interface the simulation capabilities of Project Chrono with Deep Learning frameworks.

We propose to use the alternating direction method of multipliers (ADMMs) for solving the variational inequality that arises in non-smooth contact problems in multibody simulations. The ADMM method is based on simple computational primitives and offers good convergence properties especially for scenarios where loose tolerance on the precision can be accepted, thus making it attractive also for real-time applications. The method scales well with the problem size and requires a small number of tuning parameters. Differently from other methods known in the literature, such as fixed-point iterations, it can easily handle problems that mix finite elements and rigid bodies, and it is only marginally affected by odd mass ratios or ill-posed problems. Another attractive feature is that it can be easily warm-started.

In this paper we use the Proximal Policy Optimization (PPO) deep reinforcement learning algorithm to train a Neural Network to control a four-legged robot in simulation. Reinforcement learning in general can learn complex behavior policies from simple state-reward tuples datasets and PPO in particular has proved its effectiveness in solving complex tasks with continuous states and actions. Moreover, since it is model-free, it is general and can adapt to changes in the environment or in the robot itself.

This work discusses an efficient formulation of a geometrically exact three-dimensional beam which can be used in dynamical simulations involving large displacements, collisions and non-linear materials. To this end, we base our model on the shear-flexible Cosserat rod theory and we implement it in the context of Isogeometric Analysis (IGA). According to the IGA approach, the centerline of the beam is parameterized using splines; in our work the rotation of the section is parameterized by a spline interpolation of quaternions, and time integration of rotations is performed using the exponential map of quaternions. Aiming at an efficient and robust simulation of contacts, we propose the adoption of a non-smooth dynamics formulation based on differential-variational inequalities. The model has been implemented in an open-source physics simulation library that can simulate actuators, finite elements, rigid bodies, constraints, collisions and frictional contacts. This beam model has been tested on various benchmarks in order to assess its validity in non-linear static and dynamic analysis; in all cases the model behaved consistently with theoretical results and experimental data.

The increasing complexity of dynamic simulations involving unilateral constraints, such as contacts, is pushing for new solvers that may address the problem of handling non-smooth impact events in a more efficient and accurate manner, especially in mixed rigid and flexible-bodies simulations. For this purpose, a new implementation of an interior-point solver for Quadratic Cone Programming is proposed. Even though the general idea of considering multibody system formulations as a parallel of an optimization/programming problem is already known in literature ([3, 4, 6]), still very few options are available for those problems whose complexity is due to contacts with friction and finite-elements at the same time. The opportunity to handle those problems in a unified numerical framework could trigger a novel interest both in scientific and applied researches.

In this work we focus on the role of Multibody Simulation in creating Reinforcement Learning virtual environments for robotic manipulation, showing a versatile, efficient and open source toolchain to create directly from CAD models. Using the Chrono::Solidworks plugin we are able to create robotic environments in the 3D CAD software Solidworks® and later convert them into PyChrono models (PyChrono is an open source Python module for multibody simulation). In addition, we demonstrate how collision detection can be made more efficient by introducing a limited number of contact primitives instead of performing collision detection and evaluation on complex 3D meshes, still reaching a policy able to avoid unwanted collisions. We tested this approach on a 6DOF robot Comau Racer3: the robot, together with a 2 fingers gripper (Hand-E by Robotiq) was modelled using Solidworks®, imported as a PyChrono model and then a NN was trained in simulation to control its motor torques to reach a target position. To demonstrate the versatility of this toolchain we also repeated the same procedure to model and then train the ABB IRB 120 robotic arm.