SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation
IJCAI 2023(2023)
摘要
As an important and challenging problem in computer vision, PAnoramic
Semantic Segmentation (PASS) gives complete scene perception based on an
ultra-wide angle of view. Usually, prevalent PASS methods with 2D panoramic
image input focus on solving image distortions but lack consideration of the 3D
properties of original 360^∘ data. Therefore, their performance will
drop a lot when inputting panoramic images with the 3D disturbance. To be more
robust to 3D disturbance, we propose our Spherical Geometry-Aware Transformer
for PAnoramic Semantic Segmentation (SGAT4PASS), considering 3D spherical
geometry knowledge. Specifically, a spherical geometry-aware framework is
proposed for PASS. It includes three modules, i.e., spherical geometry-aware
image projection, spherical deformable patch embedding, and a panorama-aware
loss, which takes input images with 3D disturbance into account, adds a
spherical geometry-aware constraint on the existing deformable patch embedding,
and indicates the pixel density of original 360^∘ data, respectively.
Experimental results on Stanford2D3D Panoramic datasets show that SGAT4PASS
significantly improves performance and robustness, with approximately a 2
increase in mIoU, and when small 3D disturbances occur in the data, the
stability of our performance is improved by an order of magnitude. Our code and
supplementary material are available at
https://github.com/TencentARC/SGAT4PASS.
更多查看译文
关键词
panoramic semantic
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要