Reinforcement Learning with Elastic Time Steps
CoRR(2024)
摘要
Traditional Reinforcement Learning (RL) algorithms are usually applied in
robotics to learn controllers that act with a fixed control rate. Given the
discrete nature of RL algorithms, they are oblivious to the effects of the
choice of control rate: finding the correct control rate can be difficult and
mistakes often result in excessive use of computing resources or even lack of
convergence.
We propose Soft Elastic Actor-Critic (SEAC), a novel off-policy actor-critic
algorithm to address this issue. SEAC implements elastic time steps, time steps
with a known, variable duration, which allow the agent to change its control
frequency to adapt to the situation. In practice, SEAC applies control only
when necessary, minimizing computational resources and data usage.
We evaluate SEAC's capabilities in simulation in a Newtonian kinematics maze
navigation task and on a 3D racing video game, Trackmania. SEAC outperforms the
SAC baseline in terms of energy efficiency and overall time management, and
most importantly without the need to identify a control frequency for the
learned controller. SEAC demonstrated faster and more stable training speeds
than SAC, especially at control rates where SAC struggled to converge.
We also compared SEAC with a similar approach, the Continuous-Time
Continuous-Options (CTCO) model, and SEAC resulted in better task performance.
These findings highlight the potential of SEAC for practical, real-world RL
applications in robotics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要