Style-adaptive Photo Aesthetic Rating via Convolutional Neural Networks and Multi-task Learning

Photo aesthetic rating aims at automatically and precisely evaluating the pictorial aesthetic score. Recently, great progresses has been achieved in this area due to the fabulous ability of convolutional neural networks (CNNs). Existing deep approaches try to train CNNs from a large set of photos with score annotations. However, the considerable cost in collecting score annotations limits the generalization of these approaches to other types of media. To combat this limitation, we propose a novel photo aesthetic rating architecture. Our method comprises three modules: a CNN based feature extractor, a style classifier, and a group of style-specific aesthetic prediction models. First, we propose to train a CNN in the standard binary aesthetic classification task and use it to extract aesthetic-aware features, because binary labels are easy to access. Afterwards, we use a support vector machine (SVM) to formulate the style classifier. Besides, we explored a multi-task learning (MTL) approach to jointly learn the style-specific rating models, which will further improve the generalization ability. Additionally, to address the imbalance in the distribution of styles and ratings, we adopted the strategy of data augmentation and selective sampling. Finally, we estimate the rating of an input photo by combining the estimates of the style-specific rating models according to the outputs of the style classifier. Experiments conducted on the AVA database show that the proposed method is considerably comparative to state-of-the-art approaches.