Multimodal Object Classification Using Bidirectional Gated Recurrent Unit Networks.

In the field of artificial intelligence, we have observed a strong trend towards introducing external resources to aid pattern recognition tasks. In this work, we utilize external meta-data in the form of web text, i.e. natural language, to aid image classification. We adopt pre-trained convolutional neural network (CNN) to learn the mid-level representation of images, meanwhile we model the text as sequential data using bidirectional gated recurrent unit (BGRU) encoders. To address the heterogeneity of the two different representations, we employ a neural network to learn a shared multimodal representation. Experimental results show that the multimodal CNN-BGRU model achieves remarkable classification performance on the large-scale Pascal VOC-2007 and VOC-2012 datasets.