RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance
arxiv(2024)
摘要
Although diffusion models can generate high-quality human images, their
applications are limited by the instability in generating hands with correct
structures. Some previous works mitigate the problem by considering hand
structure yet struggle to maintain style consistency between refined malformed
hands and other image regions. In this paper, we aim to solve the problem of
inconsistency regarding hand structure and style. We propose a conditional
diffusion-based framework RHanDS to refine the hand region with the help of
decoupled structure and style guidance. Specifically, the structure guidance is
the hand mesh reconstructed from the malformed hand, serving to correct the
hand structure. The style guidance is a hand image, e.g., the malformed hand
itself, and is employed to furnish the style reference for hand refining. In
order to suppress the structure leakage when referencing hand style and
effectively utilize hand data to improve the capability of the model, we build
a multi-style hand dataset and introduce a twostage training strategy. In the
first stage, we use paired hand images for training to generate hands with the
same style as the reference. In the second stage, various hand images generated
based on the human mesh are used for training to enable the model to gain
control over the hand structure. We evaluate our method and counterparts on the
test dataset of the proposed multi-style hand dataset. The experimental results
show that RHanDS can effectively refine hands structure- and style- correctly
compared with previous methods. The codes and datasets will be available soon.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要