Mind the Gap - A Benchmark for Dense Depth Prediction Beyond Lidar

The large interest in autonomous vehicles is a significant driver for computer vision research. Current deep learning approaches are capable of impressive feats, like dense full frame depth prediction from a single image. While impressive results have been achieved, it is not yet clear if they are sufficient for autonomous driving. The problem remains that existing evaluation benchmarks and metrics are not yet capable of fully addressing this issue. This work takes a step towards answering this question. Current evaluation methods are incapable of proving or refuting suitability for potentially hazardous real world situations. This is due to a) the large gaps in the currently used Lidar ground truth data, which cannot test many difficult and relevant cases and b) the global summary metrics used, which are intangible with respect to rigorous performance guarantees. In this work we provide a new benchmark based on commercially available dense light-field depth data, which closes these gaps in the evaluation. We implement domain-specific and interpretable error metrics, which allow for strict assertions over the performance of tested methods. The leaderboard for dense depth prediction is publicly available. The approach is also transferable to other depth estimation tasks. Such stringent evaluations are indispensable when testing and demonstrating performance for potentially hazardous applications like autonomous driving, and are a critical aspect for the assessment of autonomous systems by regulatory bodies as well as for public acceptance.