Optimal Limit-Cycle Control Recast As Bayesian Inference

Abstract We introduce an algorithm that generates an optimal controller for stochastic nonlinear problems with a periodic solution, e.g. locomotion. Uniquely, the quantity we approximate is neither the Value nor Policy functions, but rather the stationary state-distribution of the optimally-controlled process. We recast the control problem as Bayesian inference over a graphical model with a ring topology. The posterior approximates the controlled stationary distribution with local gaussians along the optimal limit-cycle. Linear-feedback gains and open-loop controls are extracted from the covariances and the means, respectively. Complexity scales linearly or quadratically with the state dimension, depending on the dynamics approximation. We demonstrate our algorithm on a toy 2-dimensional problem and then on a challenging 23-dimensional simulated walking robot.