Towards Optimal CNN Descriptors for Large-Scale Image Retrieval

Instance-level image retrieval is a long-standing and challenging problem in multimedia. Recently, fine-tuning Convolutional Neural Networks (CNNs) has become a promising direction, and a number of successful strategies based on global CNN descriptors have been proposed. However, it is difficult to make direct comparisons and draw conclusions due to different settings and/or datasets. The goal of this paper is two-fold. Firstly, we present a unified implementation of modern global-CNN-based retrieval systems, break such a system into six major components, and investigate each part individually as well as globally when considering different configurations. We conduct a systematic series of experiments on a component-by-component basis and find an optimal solution in designing such a system. Secondly, we introduce a novel joint loss function with learnable parameter for fine-tuning for retrieval tasks and show, with extensive experiments, significant improvement over previous works. On the new and challenging large-scale Google-Landmarks-Dataset, we set a baseline for future research and comparisons, while on traditional retrieval benchmarks such as Oxford5k and Paris6k, as well as their recent revised versions ROxford5k and RParis6k, we achieve state-of-the-art performance under all three (Easy, Medium, and Hard) evaluation protocals by a large margin compared to competing methods.