Effective Reinforcement Learning Based on Structural Information Principles
arxiv(2024)
摘要
Although Reinforcement Learning (RL) algorithms acquire sequential behavioral
patterns through interactions with the environment, their effectiveness in
noisy and high-dimensional scenarios typically relies on specific structural
priors. In this paper, we propose a novel and general Structural Information
principles-based framework for effective Decision-Making, namely SIDM,
approached from an information-theoretic perspective. This paper presents a
specific unsupervised partitioning method that forms vertex communities in the
state and action spaces based on their feature similarities. An aggregation
function, which utilizes structural entropy as the vertex weight, is devised
within each community to obtain its embedding, thereby facilitating
hierarchical state and action abstractions. By extracting abstract elements
from historical trajectories, a directed, weighted, homogeneous transition
graph is constructed. The minimization of this graph's high-dimensional entropy
leads to the generation of an optimal encoding tree. An innovative two-layer
skill-based learning mechanism is introduced to compute the common path entropy
of each state transition as its identified probability, thereby obviating the
requirement for expert knowledge. Moreover, SIDM can be flexibly incorporated
into various single-agent and multi-agent RL algorithms, enhancing their
performance. Finally, extensive evaluations on challenging benchmarks
demonstrate that, compared with SOTA baselines, our framework significantly and
consistently improves the policy's quality, stability, and efficiency up to
32.70
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要