Realistic Real-Time Simulation of RGB and Depth Sensors for Dynamic Scenarios using Augmented Image Based Rendering

Simulation remains one of the key methods for testing and validation of robotic perception systems and it also becomes increasingly important for training visuomotor policies for autonomous driving or manipulation. Further, as perception pipelines tend to leverage increasing amounts of modalities, it appears vital to simulate additional cues such as depth maps aside from RGB images. To align simulation with real-world observations, it is key to achieve realistic renderings of these maps, which includes the capability of rendering other dynamic objects in the scene. In this work, we propose an approach to real-time simulation of photo-realistic RGB images and sensor-realistic depth maps, that can contain dynamic objects at user-defined locations. Our method employs a selection of static samples of a pre-recorded database and multimodal cues from CAD models that are fused and warped to synthesize new imagery for a target camera pose. We show the efficacy of our method on newly proposed datasets recorded in a variety of different setups.