Robust Stability and Near-optimality for Policy Iteration: For Want of Recursive Feasibility, All is not Lost

We consider deterministic nonlinear discrete-time systems whose inputs are generated by PI for undiscounted cost functions. We first assume that PI is recursively feasible, in the sense that the optimization problems solved at each iteration admit a solution. In this case, we provide novel conditions to establish recursive robust stability properties for a general attractor, meaning that the policies generated at each iteration ensure a robust $\mathcal {KL}$ -stability property with respect to a general state measure. We then derive novel explicit bounds on the mismatch between the (suboptimal) value function returned by PI at each iteration and the optimal one. However, we show by a counter-example that PI may fail to be recursively feasible, disallowing the mentioned stability and near-optimality guarantees. We therefore also present a modification of PI so that recursive feasibility is guaranteed a priori under mild conditions. This modified algorithm, called $\mathrm{PI}^{+}$ , is shown to preserve the recursive robust stability when the attractor is compact. Additionally, $\mathrm{PI}^{+}$ enjoys the same near-optimality properties as its PI counterpart under the same assumptions.