Successor-Predecessor Intrinsic Exploration
NeurIPS 2023(2023)
摘要
Exploration is essential in reinforcement learning, particularly in
environments where external rewards are sparse. Here we focus on exploration
with intrinsic rewards, where the agent transiently augments the external
rewards with self-generated intrinsic rewards. Although the study of intrinsic
rewards has a long history, existing methods focus on composing the intrinsic
reward based on measures of future prospects of states, ignoring the
information contained in the retrospective structure of transition sequences.
Here we argue that the agent can utilise retrospective information to generate
explorative behaviour with structure-awareness, facilitating efficient
exploration based on global instead of local information. We propose
Successor-Predecessor Intrinsic Exploration (SPIE), an exploration algorithm
based on a novel intrinsic reward combining prospective and retrospective
information. We show that SPIE yields more efficient and ethologically
plausible exploratory behaviour in environments with sparse rewards and
bottleneck states than competing methods. We also implement SPIE in deep
reinforcement learning agents, and show that the resulting agent achieves
stronger empirical performance than existing methods on sparse-reward Atari
games.
更多查看译文
关键词
exploration,successor-predecessor
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要