The latest video coding standard, Versatile Video Coding (VVC), supports multilayer coding and has been actively studied to deliver a variety of videos, such as multi-resolution (2K and 4K) videos, efficiently. To evaluate the compression performance of multilayer coding, a rate-distortion optimization theory-based framework was recently proposed. Using this framework, the optimal bitrate allocation (OBA) between the base layer (BL) and enhancement layer (EL) can be derived. The OBA is defined as the combination of BL and EL bitrates that maximizes the sum of Peak Signal-to-Noise Ratio (PSNR) values of low-resolution (2K) and high-resoltion (4K) videos. However, the PSNR does not always correlate with subjective quality. Moreover, the OBA varies depending on the video content features, indicating the need to estimate the OBA individually using a method that is tailored to the features. Therefore, we investigated the OBA using the Video Multimethod Assessment Fusion (VMAF) as a video quality metric and then developed a method to estimate the video-adaptive OBA. In this study, the OBA was derived analytically based on the rate-distortion curves obtained from multilayer coding experiments, and the proposed method estimated the OBA using video features as parameters. The proposed method could estimate the OBA with high correlations and small errors, with a Pearson linear correlation coefficient of 0.94, Spearman rank order correlation coefficient of 0.90, and root mean square error of 0.04.
The Versatile Video Coding (VVC) standard, the state-of-the-art technology in video compression, introduces affine prediction, a key innovation in inter-frame prediction that utilizes multiple Motion Vectors (MVs) to represent complex motion patterns. To determine the optimal set of MVs, affine prediction employs the Affine Motion Estimation (AME), which combines the Gradient-Based Iterative Algorithm (GBIA) and the Block Matching Algorithm (BMA). Despite its effectiveness, the high computational cost of the AME, particularly in the GBIA, poses significant challenges for its use in real scenarios. In this context, this paper proposes a fast and hardware-friendly AME approach that considers motion information from macro-level structures to infer the motion of smaller blocks. Experimental results demonstrate a 99.54% reduction in GBIA iterations, leading to a 71.08% decrease in AME runtime, with only a 0.462% increase in BD-Rate. The proposed approach also reduces memory communication in the GBIA by up to 91.66%, contributing to a more efficient and low-power hardware design.
In video encoding, saliency-based rate control improves coding efficiency without compromising perceptual quality by allocating more bits to visually important regions. While object detection can guide bit allocation through rectangular bounding boxes, segmentation masks offer a more accurate delineation of object boundaries. This paper seeks to reduce the significant computational burden of frame-by-frame instance segmentation by proposing a lightweight segmentation mask tracking scheme, in which motion vectors (MVs) from the video encoder are used to predict per-vertex displacements. Altogether, we propose two neural network designs for segmentation mask tracking: (1) a base tracker optimized for tracking accuracy; and (2) a lite tracker that balances accuracy and computational complexity. Our experimental results show that the base tracker attains 70-88% of the accuracy of frame-by-frame instance segmentation but achieves a 48x speedup and reduces computational complexity to 0.03% on CPU. For the lite tracker, the corresponding figures are 49x, 0.01%, and 67-88%. Despite tradeoffs in tracking accuracy, reducing complexity to a fraction makes our solution a viable option for practical applications.
The rapid deployment of modern video coding standards underscores the need for hardware implementations that enable real-time coding, support interoperability across codecs, and are openly accessible to community. This paper presents the first known high-level synthesis (HLS) implementation of an accurate full-search fractional motion estimation (FME). The proposed FME core is released as open-source and is compatible with High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) standards. It implements 1) an accurate multiplierless constant multiplication (MCM) unit that performs quarter-pixel interpolation over 9 x 9 pixels at a time; 2) a transform-exempted sum of absolute transformed differences (TE-SATD) unit that is optimized for area and speed; and 3) fixed-point Lagrangian optimizations for rate-distortion optimization (RDO). On an Intel Arria 10 FPGA, the FME core consumes 122 kALUTs and operates at up to 210 MHz. Our profiling results show that a single FME core can support practical HEVC and VVC encoding of 2160p video at 30-120 fps, depending on the video content and encoder preset.
Image compression for humans and machines (ICHM) requires balancing machine vision performance with human visual quality. Recent methods introduce text-assisted compression to improve perceptual quality, but they often lack explicit optimization for downstream machine vision tasks. To address this, we propose a novel dual-layer codec that jointly considers semantic accuracy and visual fidelity. The base layer is optimized for object detection and encodes semantic-aware features, while the enhancement layer refines visual quality by leveraging both the base-layer output and text guidance during encoding—without requiring text at decoding. Experiments on COCO2017 demonstrate that our method achieves superior rate-accuracy trade-offs for both object detection and image reconstruction compared to prior ICHM methods.
