Visual Concept Conjunction Learning with Recurrent Neural Networks

Learning the conjunction of multiple visual concepts shows practical significance in various real world applications (e.g. multi-attribute image retrieval and visual relationship detection). In this paper, we propose Concept Conjunction Recurrent Neural Network (C2RNN) to tackle this problem. With our model, visual concepts involved in a conjunction are mapped into the hidden units and combined in a recurrent way to generate the representation of the concept conjunction, which is then used to compute a concept conjunction classifier as the output. We also present an order invariant version of the proposed method based on attention mechanism to learn the tasks without pre-defined concept order. To tackle concept conjunction learning from multiple semantic domains, we introduce a multiplicative framework to learn the joint representation. Experimental results on multi-attribute image retrieval and visual relationship detection show that our method achieves significantly better performance than other related methods on various datasets.