Improving Image Captioning with Language Modeling Regularizations

Inspired by the recent work in language modeling, we investigate the effects of a set of regularization techniques on the performance of a recurrent neural network based image captioning model. Using these techniques, we achieve 13 Bleu-4 points improvements over using no regularizations. We show that our model does not suffer from loss-evaluation mismatch and also connect the model performance to dataset properties by running experiments on MSCOCO dataset. Further, we propose a human in the loop image captioning system as an alternative way to improve the model performance. Using only the first two tokens of a reference sentence of an image, we improve Bleu-4 score of our best model by 57 points with this hybrid system.