Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction
arxiv(2024)
摘要
Sim2real transfer has received increasing attention lately due to the success
of learning robotic tasks in simulation end-to-end. While there has been a lot
of progress in transferring vision-based navigation policies, the existing
sim2real strategy for audio-visual navigation performs data augmentation
empirically without measuring the acoustic gap. The sound differs from light in
that it spans across much wider frequencies and thus requires a different
solution for sim2real. We propose the first treatment of sim2real for
audio-visual navigation by disentangling it into acoustic field prediction
(AFP) and waypoint navigation. We first validate our design choice in the
SoundSpaces simulator and show improvement on the Continuous AudioGoal
navigation benchmark. We then collect real-world data to measure the spectral
difference between the simulation and the real world by training AFP models
that only take a specific frequency subband as input. We further propose a
frequency-adaptive strategy that intelligently selects the best frequency band
for prediction based on both the measured spectral difference and the energy
distribution of the received audio, which improves the performance on the real
data. Lastly, we build a real robot platform and show that the transferred
policy can successfully navigate to sounding objects. This work demonstrates
the potential of building intelligent agents that can see, hear, and act
entirely from simulation, and transferring them to the real world.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要