Sketch/Image-Based 3D Scene Retrieval: Benchmark, Algorithm, Evaluation

Sketch/Image-based 3D scene retrieval is to retrieve man-made 3D scene models given a user's hand-drawn 2D scene sketch or a 2D scene image usually captured by a camera. It is a brand new but also very challenging research topic in the field of 3D object retrieval due to the semantic gap in their representations: 3D scene models or views differ from either non-realistic 2D scene sketches or realistic 2D scene images. Due to the intuitiveness in sketching and ubiquitous availability in image capturing, this research topic has vast applications such as 3D scene reconstruction, autonomous driving cars, 3D geometry video retrieval, and 3D AR/VR entertainment. To boost this interesting and important research, we build the currently largest and most comprehensive 2D scene sketch/image-based 3D scene retrieval benchmark1, develop a convolutional neural network (CNN)-based 3D scene retrieval algorithm and finally conduct an evaluation on the benchmark.