Don't Mention the Shoe! A Learning to Rank Approach to Content Selection for Image Description Generation.

We tackle the sub-task of content selectionas part of the broader challenge of automaticallygenerating image descriptions.More specifically, we explore how decisionscan be made to select what objectinstances should be mentioned in an imagedescription, given an image and labelledbounding boxes. We propose castingthe content selection problem as alearning to rank problem, where object instancesthat are most likely to be mentionedby humans when describing an imageare ranked higher than those that areless likely to be mentioned. Several featuresare explored: those derived frombounding box localisations, from conceptlabels, and from image regions. Objectinstances are then selected based on theranked list, where we investigate severalmethods for choosing a stopping criterionas the ‘cut-off’ point for objects in theranked list. Our best-performing methodachieves state-of-the-art performance onthe ImageCLEF2015 sentence generationchallenge.