MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator

Offline reinforcement learning (RL) faces a significant challenge ofdistribution shift. Model-free offline RL penalizes the Q value forout-of-distribution (OOD) data or constrains the policy closed to the behaviorpolicy to tackle this problem, but this inhibits the exploration of the OODregion. Model-based offline RL, which uses the trained environment model togenerate more OOD data and performs conservative policy optimization withinthat model, has become an effective method for this problem. However, thecurrent model-based algorithms rarely consider agent robustness whenincorporating conservatism into policy. Therefore, the new model-based offlinealgorithm with a conservative Bellman operator (MICRO) is proposed. This methodtrades off performance and robustness via introducing the robust Bellmanoperator into the algorithm. Compared with previous model-based algorithms withrobust adversarial models, MICRO can significantly reduce the computation costby only choosing the minimal Q value in the state uncertainty set. Extensiveexperiments demonstrate that MICRO outperforms prior RL algorithms in offlineRL benchmark and is considerably robust to adversarial perturbations.