Weakly Supervised Reinforced Multi-Operator Image Retargeting

AbstractImage retargeting aims to adjust the resolution and aspect ratio to an arbitrary size while preserving important content of the image. Usually multi-operator image retargeting demonstrates better generalization than single operator scheme due to heterogeneous characteristics of different regions in the image. Most existing multi-operator retargeting methods search the optimal operator at each step with exponential complexity and with the possibility of falling into local optimum. Therefore, in order to produce better results with lower computational costs, we formulate the multi-operator retargeting as a Markov decision-making process and apply Reinforcement Learning (RL) to achieve global optimum. Instead of using traditional image-level measures, we design a high-level semantic and aesthetic reward function to better match human visual perception. With the priori in reward, we further propose a weakly supervised Semantics and Aesthetics aware Multi-operator Image Retargeting (SAMIR) framework. Particularly, the semantic part of the reward helps to constrain the severe deformations that may occur during retargeting process, while the aesthetic part guarantees the sensory quality, which can effectively measure the perceptual effects of different operators on various image content. The operator of each step is learned in an end-to-end manner. In addition, retargeting can be performed in arbitrary target size, step size, and direction. The experiment results on both representative aesthetic datasets and retargeting datasets consistently show that our model outperforms the state-of-the-art methods.