Instruction-guided object detection

This paper aims at instruction-guided object detection, i.e., predicting the objects associated with the implementation of a specific instruction for intelligent robots. It is not practical to solve this problem by picking out the instruction-related objects from the detection results of a general object detector due to massive annotation cost and low adaptability of training data. To address these challenges, we introduce a flexible dataset that can well adapt to the variation of the instruction set and only annotates instruction-related object samples. We then propose to amend the current detection paradigm by incorporating semantic instruction description effectively. Specifically, the relationship between an instruction and related objects is modeled by the and-or graph and is further fused into a unified neural network for solving the object detection problem constrained by instructions. Our algorithm encodes the and-or graph representation to attend related objects sensitive local features and then infer instruction-related object location. Extensive evaluation on the newly constructed dataset verifies the effectiveness of our approach.