Behaviors of animal bipedal locomotion can be described, in a simplified form, by the bipedal spring-mass model. The model provides predictive power, and helps us understand this complex dynamical behavior. In this paper, we analyzed a range of gaits generated by the bipedal spring-mass model during walking, and proposed a stabilizing touch-down condition for the swing leg. This policy is stabilizing against disturbances inside and outside the same energy level and requires only internal state information. In order to generalize the results to be independent of size and dimension of the system, we nondimensionalized the equations of motion for the bipedal spring-mass model. We presented the equilibrium gaits (a.k.a fixed point gaits) as a continuum on the walking state space showing how the different types of these gaits evolve and where they are located in the state space. Then, we showed the stability analysis of the proposed touch-down control policy for different energy levels and leg stiffness values. The results showed that the proposed touch-down control policy can stabilize towards all types of the symmetric equilibrium gaits. Moreover, we presented how the peak leg force changes within an energy level and as it varies due to the type of the gait since peak force is important as a measurement of injury or damage risk on a robot or animal. Finally, we presented simulations of the bipedal spring-mass model walking on level ground and rough terrain transitioning between different equilibrium gaits as the energy level of the system changes with respect to the ground height. The analysis in this paper is theoretical, and thus applicable to further our understanding of animal bipedal locomotion and the design and control of robotic systems like ATRIAS, Cassie, and Digit.

In this paper, we explore the space of running gaits for the bipedal robot Cassie. Our first contribution is to present an approach for optimizing gait efficiency across a spectrum of speeds with the aim of enabling extremely high-speed running on hardware. This raises the question of how the resulting gaits compare to human running mechanics, which are known to be highly efficient in comparison to quadrupeds. Our second contribution is to conduct this comparison based on established human biomechanical studies. We find that despite morphological differences between Cassie and humans, key properties of the gaits are highly similar across a wide range of speeds. Finally, our third contribution is to integrate the optimized running gaits into a full controller that satisfies the rules of the real-world task of the 100m dash, including starting and stopping from a standing position. We demonstrate this controller on hardware to establish the Guinness World Record for Fastest 100m by a Bipedal Robot.

The complex dynamics of agile robotic legged locomotion requires motion planning to intelligently adjust footstep locations. Often, bipedal footstep and motion planning use mathematically simple models such as the linear inverted pendulum, instead of dynamically-rich models that do not have closed-form solutions. We propose a real-time optimization method to plan for dynamical models that do not have closed form solutions and experience irrecoverable failure. Our method uses a data-driven approximation of the step-to-step dynamics and of a failure margin function. This failure margin function is an oriented distance function in state-action space where it describes the signed distance to success or failure. The motion planning problem is formed as a nonlinear program with constraints that enforce the approximated forward dynamics and the validity of state-action pairs. For illustration, this method is applied to create a planner for an actuated spring-loaded inverted pendulum model. In an ablation study, the failure margin constraints decreased the number of invalid solutions by between 24 and 47 percentage points across different objectives and horizon lengths. While we demonstrate the method on a canonical model of locomotion, we also discuss how this can be applied to data-driven models and full-order robot models.

In this work, we propose a learning approach for 3D dynamic bipedal walking when footsteps are constrained to stepping stones. While recent work has shown progress on this problem, real-world demonstrations have been limited to relatively simple open-loop, perception-free scenarios. Our main contribution is a more advanced learning approach that enables real-world demonstrations, using the Cassie robot, of closed-loop dynamic walking over moderately difficult stepping-stone patterns. Our approach first uses reinforcement learning (RL) in simulation to train a controller that maps footstep commands onto joint actions without any reference motion information. We then learn a model of that controller's capabilities, which enables prediction of feasible footsteps given the robot's current dynamic state. The resulting controller and model are then integrated with a real-time overhead camera system for detecting stepping stone locations. For evaluation, we develop a benchmark set of stepping stone patterns, which are used to test performance in both simulation and the real world. Overall, we demonstrate that sim-to-real learning is extremely promising for enabling dynamic locomotion over stepping stones. We also identify challenges remaining that motivate important future research directions.

Recent work on sim-to-real learning for bipedal locomotion has demonstrated new levels of robustness and agility over a variety of terrains. However, that work, and most prior bipedal locomotion work, have not considered locomotion under a variety of external loads that can significantly influence the overall system dynamics. In many applications, robots will need to maintain robust locomotion under a wide range of potential dynamic loads, such as pulling a cart or carrying a large container of sloshing liquid, ideally without requiring additional load-sensing capabilities. In this work, we explore the capabilities of reinforcement learning (RL) and sim-to-real transfer for bipedal locomotion under dynamic loads using only proprioceptive feedback. We show that prior RL policies trained for unloaded locomotion fail for some loads and that simply training in the context of loads is enough to result in successful and improved policies. We also compare training specialized policies for each load versus a single policy for all considered loads and analyze how the resulting gaits change to accommodate different loads. Finally, we demonstrate sim-to-real transfer, which is successful but shows a wider sim-to-real gap than prior unloaded work, which points to interesting future research.

For legged robots to match the athletic capabilities of humans and animals, they must not only produce robust periodic walking and running, but also seamlessly switch between nominal locomotion gaits and more specialized transient maneuvers. Despite recent advancements in controls of bipedal robots, there has been little focus on producing highly dynamic behaviors. Recent work utilizing reinforcement learning to produce policies for control of legged robots have demonstrated success in producing robust walking behaviors. However, these learned policies have difficulty expressing a multitude of different behaviors on a single network. Inspired by conventional optimization-based control techniques for legged robots, this work applies a recurrent policy to execute four-step, 90 degree turns trained using reference data generated from optimized single rigid body model trajectories. We present a novel training framework using epilogue terminal rewards for learning specific behaviors from pre-computed trajectory data and demonstrate a successful transfer to hardware on the bipedal robot Cassie.

Recently, work on reinforcement learning (RL) for bipedal robots has successfully learned controllers for a variety of dynamic gaits with robust sim-to-real demonstrations. In order to maintain balance, the learned controllers have full freedom of where to place the feet, resulting in highly robust gaits. In the real world however, the environment will often impose constraints on the feasible footstep locations, typically identified by perception systems. Unfortunately, most demonstrated RL controllers on bipedal robots do not allow for specifying and responding to such constraints. This missing control interface greatly limits the real-world application of current RL controllers. In this paper, we aim to maintain the robust and dynamic nature of learned gaits while also respecting footstep constraints imposed externally. We develop an RL formulation for training dynamic gait controllers that can respond to specified touchdown locations. We then successfully demonstrate simulation and sim-to-real performance on the bipedal robot Cassie. In addition, we use supervised learning to induce a transition model for accurately predicting the next touchdown locations that the controller can achieve given the robot's proprioceptive observations. This model paves the way for integrating the learned controller into a full-order robot locomotion planner that robustly satisfies both balance and environmental constraints.

In this work, we propose a method to generate reduced-order model reference trajectories for general classes of highly dynamic maneuvers for bipedal robots for use in sim-to-real reinforcement learning. Our approach is to utilize a single rigid-body model (SRBM) to optimize libraries of trajectories offline to be used as expert references that guide learning by regularizing behaviors when incorporated in the reward function of a learned policy. This method translates the model's dynamically rich rotational and translational behavior to a full-order robot model and successfully transfers to real hardware. The SRBM's simplicity allows for fast iteration and refinement of behaviors, while the robustness of learning-based controllers allows for highly dynamic motions to be transferred to hardware. Within this work we introduce a set of transferability constraints that amend the SRBM dynamics to actual bipedal robot hardware, our framework for creating optimal trajectories for a variety of highly dynamic maneuvers as well as our approach to integrating reference trajectories for a high-speed running reinforcement learning policy. We validate our methods on the bipedal robot Cassie on which we were successfully able to demonstrate highly dynamic grounded running gaits up to 3.0 m/s.

In this letter, we describe an approach to achieve dynamic legged locomotion on physical robots which combines existing methods for control with reinforcement learning. Specifically, our goal is a control hierarchy in which highest-level behaviors are planned through reduced-order models, which describe the fundamental physics of legged locomotion, and lower level controllers utilize a learned policy that can bridge the gap between the idealized, simple model and the complex, full order robot. The high-level planner can use a model of the environment and be task specific, while the low-level learned controller can execute a wide range of motions so that it applies to many different tasks. In this letter, we describe this learned dynamic walking controller and show that a range of walking motions from reduced-order models can be used as the command and primary training signal for learned policies. The resulting policies do not attempt to naively track the motion (as a traditional trajectory tracking controller would) but instead balance immediate motion tracking with long term stability. The resulting controller is demonstrated on a human scale, unconstrained, untethered bipedal robot at speeds up to 1.2 m/s. This letter builds the foundation of a generic, dynamic learned walking controller that can be applied to many different tasks.

We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a reward-specification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits - standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.

Accurate and precise terrain estimation is a difficult problem for robot locomotion in real-world environments. Thus, it is useful to have systems that do not depend on accurate estimation to the point of fragility. In this paper, we explore the limits of such an approach by investigating the problem of traversing stair-like terrain without any external perception or terrain models on a bipedal robot. For such blind bipedal platforms, the problem appears difficult (even for humans) due to the surprise elevation changes. Our main contribution is to show that sim-to-real reinforcement learning (RL) can achieve robust locomotion over stair-like terrain on the bipedal robot Cassie using only proprioceptive feedback. Importantly, this only requires modifying an existing flat-terrain training RL framework to include stair-like terrain randomization, without any changes in reward function. To our knowledge, this is the first controller for a bipedal, human-scale robot capable of reliably traversing a variety of real-world stairs and other stair-like disturbances using only proprioception.

Recent work has demonstrated the success of reinforcement learning (RL) for training bipedal locomotion policies for real robots. This prior work, however, has focused on learning joint-coordination controllers based on an objective of following joint trajectories produced by already available controllers. As such, it is difficult to train these approaches to achieve higher-level goals of legged locomotion, such as simply specifying the desired end-effector foot movement or ground reaction forces. In this work, we propose an approach for integrating knowledge of the robot system into RL to allow for learning at the level of task space actions in terms of feet setpoints. In particular, we integrate learning a task space policy with a model-based inverse dynamics controller, which translates task space actions into joint-level controls. With this natural action space for learning locomotion, the approach is more sample efficient and produces desired task space dynamics compared to learning purely joint space actions. We demonstrate the approach in simulation and also show that the learned policies are able to transfer to the real bipedal robot Cassie. This result encourages further research towards incorporating bipedal control techniques into the structure of the learning process to enable dynamic behaviors.

The data-heavy nature of reinforcement learning often dictates that control policies must be learned in simulation and then transferred to hardware. This sim-to-real transfer is not trivial and the model differences between simulation and reality usually means that naively trained policies will not transfer. Almost all of the recent robotics reinforcement learning work has relied on some form of dynamics randomization or model adaptation in order to overcome the challenge of simto-real. However, we find that we are able to achieve sim-toreal transfer without relying on these techniques. Furthermore, we accomplish this for the difficult and highly dynamic task of bipedal locomotion, transferring high performance policies capable of walking at 2.6 m/s directly to the bipedal robot Cassie. We hypothesize and discuss several possible reasons we do not require dynamics randomization and identify possible new research directions that our results raise.

In this article, we present a new controller for stable and robust walking control of ATRIAS, an underactuated bipedal robot designed based on the spring-loaded inverted pendulum (SLIP) model. We propose a forced-oscillation scheme for control of vertical motion, which we prove to be stable and contractive. Moreover, we prove that, through some mild assumptions, the dynamics of the system can be written in a hierarchical form that decouples the stability analyses of the horizontal and vertical directions. We leverage these properties to find a stabilizing class of functions for foot placement. The torso control is also proved to be decoupled using singular perturbation theory and is stabilized through a feedback linearization controller. We also take advantage of the proposed framework’s flexibility and extend it to include a new reflex-based uneven-terrain walking control scheme. We test the controller for various desired walking speeds (0 to 2.5 m/s), for stepping up and down unexpected obstacles (15 cm), and for high-speed walking on a random uneven terrain (up to 10 cm of step-ups and step-downs and up to 1.8 m/s). The results show successful performance of the controller and its stability and robustness against various perturbations.

We propose a method to generate actuation plans for a reduced order, dynamic model of bipedal running. This method explicitly enforces robustness to ground uncertainty. The plan generated is not a fixed body trajectory that is aggressively stabilized: instead, the plan interacts with the passive dynamics of the reduced order model to create emergent robustness. The goal is to create plans for legged robots that will be robust to imperfect perception of the environment, and to work with dynamics that are too complex to optimize in real-time. Working within this dynamic model of legged locomotion, we optimize a set of disturbance cases together with the nominal case, all with linked inputs. The input linking is nontrivial due to the hybrid dynamics of the running model but our solution is effective and has analytical gradients. The optimization procedure proposed is significantly slower than a standard trajectory optimization, but results in robust gaits that reject disturbances extremely effectively without any replanning required.

Controlling a non-statically stable biped is a difficult problem largely due to the complex hybrid dynamics involved. Recent work has demonstrated the effectiveness of reinforcement learning (RL) for simulation-based training of neural network controllers that successfully transfer to real bipeds. The existing work, however, has primarily used simple memoryless network architectures, even though more sophisticated architectures, such as those including memory, often yield superior performance in other RL domains. In this work, we consider recurrent neural networks (RNNs) for sim-to-real biped locomotion, allowing for policies that learn to use internal memory to model important physical properties. We show that while RNNs are able to significantly outperform memoryless policies in simulation, they do not exhibit superior behavior on the real biped due to overfitting to the simulation physics unless trained using dynamics randomization to prevent overfitting; this leads to consistently better sim-to-real transfer. We also show that RNNs could use their learned memory states to perform online system identification by encoding parameters of the dynamics into memory.

Impact forces are a destructive, yet common occurrence in legged locomotion. Every step produces a collision when the legs inertia immediately stops upon ground contact. This results in peak impact forces and high frequency vibrations that resonate through the system, damage components, and complicate control algorithms. Rubber or other damping material is the assumed solution for mitigating these impacts. However, we show the benefit of using foot springs where both stiffness and maximum compression are customized to the leg. Such springs eliminate peak impact forces by gradually bringing the legs inertia to rest. The maximum compression point (i.e. a hard stop) then provides a rigid surface during stance. We provide a methodology for designing this passive dynamic foot that is validated through simulation and physical testing. Our results show this methodology reduces rigid body impacts and foot vibrations in a way traditional methods, reliant upon rubber or damping, are yet to achieve.

In this paper, we investigate whether applying ankle torques during mid-stance can be a more effective way to reduce energetic cost of locomotion than actuating leg length alone. Ankles are useful in human gaits for many reasons including static balancing. In this work, we specifically avoid the heel-strike and toe-off benefits to investigate whether the progression of the center of pressure from heel-to-toe during mid-stance, or some other approach, is beneficial in and of itself. We use an Ankle Actuated Spring Loaded Inverted Pendulum model to simulate the shifting center of pressure dynamics, and trajectory optimization is applied to find limit cycles that minimize cost of transport. The results show that, for the vast majority of gaits, ankle torques do not affect cost of transport. Ankles reduce the cost of transport during a narrow band of gaits at the transition from grounded running to aerial running. This suggests that applying ankle torque during mid-stance of a steady gait is not a directly beneficial strategy, but is most likely a path between beneficial heel-strikes and toe-offs.

● have shown significant performance over a period of at least five of those years. In addition, candidates for Senior Member grade must supply three refer ences from current IEEE Members holding the grade of Fellow, Senior Member, or Honorary Member. For more information, visit https:// www.ieee.org/membership/senior/ or email seniormember@ieee.org. The names and affiliations of the new Senior Members are as follows … ● Taif Al Obaidi University of Central Florida Orlando Section … ● Mohammad Al-Shabi University of Sharjah Jordan Section … ● Sven Behnke University of Bonn Germany Section … ● Mike Borowczak University of Wyoming High Plains Section … ● Pinar Boyraz Chalmers University of Tech nology Sweden Section … ● Adrijan Bozinovski University American College Skopje Republic of Macedonia Section … ● Ramachandra Budihal Indian …