From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding
arxiv(2023)
摘要
Action understanding has attracted long-term attention. It can be formed as
the mapping from the physical space to the semantic space. Typically,
researchers built datasets according to idiosyncratic choices to define classes
and push the envelope of benchmarks respectively. Datasets are incompatible
with each other like "Isolated Islands" due to semantic gaps and various class
granularities, e.g., do housework in dataset A and wash plate in dataset B. We
argue that we need a more principled semantic space to concentrate the
community efforts and use all datasets together to pursue generalizable action
learning. To this end, we design a structured action semantic space given verb
taxonomy hierarchy and covering massive actions. By aligning the classes of
previous datasets to our semantic space, we gather (image/video/skeleton/MoCap)
datasets into a unified database in a unified label system, i.e., bridging
"isolated islands" into a "Pangea". Accordingly, we propose a novel model
mapping from the physical space to semantic space to fully use Pangea. In
extensive experiments, our new system shows significant superiority, especially
in transfer learning. Our code and data will be made public at
https://mvig-rhos.com/pangea.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要