Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases
arxiv(2023)
Abstract
Motion understanding aims to establish a reliable mapping between motion and
action semantics, while it is a challenging many-to-many problem. An abstract
action semantic (i.e., walk forwards) could be conveyed by perceptually diverse
motions (walking with arms up or swinging). In contrast, a motion could carry
different semantics w.r.t. its context and intention. This makes an elegant
mapping between them difficult. Previous attempts adopted direct-mapping
paradigms with limited reliability. Also, current automatic metrics fail to
provide reliable assessments of the consistency between motions and action
semantics. We identify the source of these problems as the significant gap
between the two modalities. To alleviate this gap, we propose Kinematic Phrases
(KP) that take the objective kinematic facts of human motion with proper
abstraction, interpretability, and generality. Based on KP, we can unify a
motion knowledge base and build a motion understanding system. Meanwhile, KP
can be automatically converted from motions to text descriptions with no
subjective bias, inspiring Kinematic Prompt Generation (KPG) as a novel
white-box motion generation benchmark. In extensive experiments, our approach
shows superiority over other methods. Our project is available at
https://foruck.github.io/KP/.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined