Learning from Sparse Offline Datasets via Conservative Density Estimation
CoRR(2024)
摘要
Offline reinforcement learning (RL) offers a promising direction for learning
policies from pre-collected datasets without requiring further interactions
with the environment. However, existing methods struggle to handle
out-of-distribution (OOD) extrapolation errors, especially in sparse reward or
scarce data settings. In this paper, we propose a novel training algorithm
called Conservative Density Estimation (CDE), which addresses this challenge by
explicitly imposing constraints on the state-action occupancy stationary
distribution. CDE overcomes the limitations of existing approaches, such as the
stationary distribution correction method, by addressing the support mismatch
issue in marginal importance sampling. Our method achieves state-of-the-art
performance on the D4RL benchmark. Notably, CDE consistently outperforms
baselines in challenging tasks with sparse rewards or insufficient data,
demonstrating the advantages of our approach in addressing the extrapolation
error problem in offline RL.
更多查看译文
关键词
offline reinforcement learning,stationary distribution correction estimation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要