Spatial Rate Allocation for Learning-based Video Coding.

This paper presents a method that enables arbitrary end-to-end Learning-based image/video codecs to apply spatial rate allocation. At the frame-level, the forward pass of the underlying encoder network is followed by a latent refinement step, in which a customized loss function is minimized. This loss function takes as input an arbitrary pixel-wise map that defines the interest of each pixel and computes a weighted distortion with respect to the given interest map. Back-propagation of the customized loss function using the gradient descent gives a refined version of the frame latent in which the quality of regions of interest (ROI) is improved at the cost of quality of regions of disinterest. The proposed method is implemented on top of an existing end-to-end LVC, called AIVC 1 https://githuh.com/Orange-OpenSource/AIVC, using salience-based interest maps. Experiments show that the proposed method can effectively improve the quality of regions of interest frames. Notably, BD-BR performance using Weighted PSNR (WPSNR) shows an improvement of up to 21% by the proposed method.