Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
CoRR(2024)
摘要
Sequential decision-making is desired to align with human intents and exhibit
versatility across various tasks. Previous methods formulate it as a
conditional generation process, utilizing return-conditioned diffusion models
to directly model trajectory distributions. Nevertheless, the
return-conditioned paradigm relies on pre-defined reward functions, facing
challenges when applied in multi-task settings characterized by varying reward
functions (versatility) and showing limited controllability concerning human
preferences (alignment). In this work, we adopt multi-task preferences as a
unified condition for both single- and multi-task decision-making, and propose
preference representations aligned with preference labels. The learned
representations are used to guide the conditional generation process of
diffusion models, and we introduce an auxiliary objective to maximize the
mutual information between representations and corresponding generated
trajectories, improving alignment between trajectories and preferences.
Extensive experiments in D4RL and Meta-World demonstrate that our method
presents favorable performance in single- and multi-task scenarios, and
exhibits superior alignment with preferences.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要