The latest video coding standard, Versatile Video Coding (VVC), supports multilayer coding and has been actively studied to deliver a variety of videos, such as multi-resolution (2K and 4K) videos, efficiently. To evaluate the compression performance of multilayer coding, a rate-distortion optimization theory-based framework was recently proposed. Using this framework, the optimal bitrate allocation (OBA) between the base layer (BL) and enhancement layer (EL) can be derived. The OBA is defined as the combination of BL and EL bitrates that maximizes the sum of Peak Signal-to-Noise Ratio (PSNR) values of low-resolution (2K) and high-resoltion (4K) videos. However, the PSNR does not always correlate with subjective quality. Moreover, the OBA varies depending on the video content features, indicating the need to estimate the OBA individually using a method that is tailored to the features. Therefore, we investigated the OBA using the Video Multimethod Assessment Fusion (VMAF) as a video quality metric and then developed a method to estimate the video-adaptive OBA. In this study, the OBA was derived analytically based on the rate-distortion curves obtained from multilayer coding experiments, and the proposed method estimated the OBA using video features as parameters. The proposed method could estimate the OBA with high correlations and small errors, with a Pearson linear correlation coefficient of 0.94, Spearman rank order correlation coefficient of 0.90, and root mean square error of 0.04.
The Versatile Video Coding (VVC) standard, the state-of-the-art technology in video compression, introduces affine prediction, a key innovation in inter-frame prediction that utilizes multiple Motion Vectors (MVs) to represent complex motion patterns. To determine the optimal set of MVs, affine prediction employs the Affine Motion Estimation (AME), which combines the Gradient-Based Iterative Algorithm (GBIA) and the Block Matching Algorithm (BMA). Despite its effectiveness, the high computational cost of the AME, particularly in the GBIA, poses significant challenges for its use in real scenarios. In this context, this paper proposes a fast and hardware-friendly AME approach that considers motion information from macro-level structures to infer the motion of smaller blocks. Experimental results demonstrate a 99.54% reduction in GBIA iterations, leading to a 71.08% decrease in AME runtime, with only a 0.462% increase in BD-Rate. The proposed approach also reduces memory communication in the GBIA by up to 91.66%, contributing to a more efficient and low-power hardware design.
In video encoding, saliency-based rate control improves coding efficiency without compromising perceptual quality by allocating more bits to visually important regions. While object detection can guide bit allocation through rectangular bounding boxes, segmentation masks offer a more accurate delineation of object boundaries. This paper seeks to reduce the significant computational burden of frame-by-frame instance segmentation by proposing a lightweight segmentation mask tracking scheme, in which motion vectors (MVs) from the video encoder are used to predict per-vertex displacements. Altogether, we propose two neural network designs for segmentation mask tracking: (1) a base tracker optimized for tracking accuracy; and (2) a lite tracker that balances accuracy and computational complexity. Our experimental results show that the base tracker attains 70-88% of the accuracy of frame-by-frame instance segmentation but achieves a 48x speedup and reduces computational complexity to 0.03% on CPU. For the lite tracker, the corresponding figures are 49x, 0.01%, and 67-88%. Despite tradeoffs in tracking accuracy, reducing complexity to a fraction makes our solution a viable option for practical applications.
The rapid deployment of modern video coding standards underscores the need for hardware implementations that enable real-time coding, support interoperability across codecs, and are openly accessible to community. This paper presents the first known high-level synthesis (HLS) implementation of an accurate full-search fractional motion estimation (FME). The proposed FME core is released as open-source and is compatible with High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) standards. It implements 1) an accurate multiplierless constant multiplication (MCM) unit that performs quarter-pixel interpolation over 9 x 9 pixels at a time; 2) a transform-exempted sum of absolute transformed differences (TE-SATD) unit that is optimized for area and speed; and 3) fixed-point Lagrangian optimizations for rate-distortion optimization (RDO). On an Intel Arria 10 FPGA, the FME core consumes 122 kALUTs and operates at up to 210 MHz. Our profiling results show that a single FME core can support practical HEVC and VVC encoding of 2160p video at 30-120 fps, depending on the video content and encoder preset.
Image compression for humans and machines (ICHM) requires balancing machine vision performance with human visual quality. Recent methods introduce text-assisted compression to improve perceptual quality, but they often lack explicit optimization for downstream machine vision tasks. To address this, we propose a novel dual-layer codec that jointly considers semantic accuracy and visual fidelity. The base layer is optimized for object detection and encodes semantic-aware features, while the enhancement layer refines visual quality by leveraging both the base-layer output and text guidance during encoding—without requiring text at decoding. Experiments on COCO2017 demonstrate that our method achieves superior rate-accuracy trade-offs for both object detection and image reconstruction compared to prior ICHM methods.
Text-to-image generation has rapidly progressed with autoregressive, diffusion, and distillation-based models, enabling high-fidelity and semantically relevant image synthesis from natural language prompts. Despite their success, these models still suffer from text-image misalignment, limited detail, and outputs that may violate human commonsense or aesthetic preferences. Existing human-feedback-based fine-tuning approaches partially alleviate these issues but primarily focus on earlier diffusion models (e.g., Stable Diffusion v1.4, v2.0, SDXL) and often exhibit over-optimization or under-optimization, leaving their effectiveness on state-of-the-art models unclear. In this work, we advance preference alignment for modern text-to-image models. We utilize EvalMi-50K, a large-scale dataset with human preference scores, to train our image quality assessment (IQA) model and use the predicted scores from the IQA model to reward the generation model. Building upon the dataset and IQA model, we propose Multi-Generative-Model Guided Diffusion Preference Optimization(MGM-DPO), which leverages diverse model outputs and a stable optimization strategy to align advanced diffusion models. Applied to Stable Diffusion 3.5 Large, MGM-DPO significantly improves fidelity, prompt alignment, and human preference alignment across multiple benchmarks, achieving state-of-the-art results among human-feedback-tuned models. Code is available at https://github.com/IntMeGroup/MGM-DPO.
The problem of multi-view spatial calibration remains challenging, particularly in scenarios characterized by minimal overlapping regions across sensor views, a configuration for which robust solutions are not yet fully established. To systematically address this issue, we introduce SYNCH2-Cal, an evaluation framework designed to generate synthetic scenes containing fiducial and pattern-based calibration targets, alongside configurable camera viewpoints with precisely known spatial calibration parameters. Utilizing these synthetic datasets, we systematically evaluated several established methods for multi-view spatial calibration. Additionally, we extended the framework to incorporate the generation of 3D point clouds from the synthetic scenes, thereby enabling a rigorous assessment of various state-of-the-art point cloud registration algorithms, including traditional ICP-based and learned methods. The results obtained establish a valuable benchmark for the comparative evaluation of both multi-view calibration methods and 3D point cloud registration algorithms. Furthermore, they provide a foundation to guide future improvements of existing techniques as well as the development of next-generation algorithms.
Thermal cameras are essential in applications such as surveillance, search, and rescue due to their ability to operate effectively in complete darkness and adverse weather conditions. However, the lower resolution of thermal images often limits their effectiveness. To address this, we propose multi-feature guided cross-refinement transformer (MFG-CRT), a novel RGB-guided super-resolution architecture that integrates RGB and thermal information through a Multi-Feature Guide framework, improving both resolution and detail retention in thermal images. Leveraging a Thermal Feature Extractor and Guided Upsampling Structure, MFG-CRT effectively balances high-frequency detail preservation and global context integration. Our lightweight model achieves notable performance in RGB-guided thermal super-resolution tasks, offering low inference time and low memory usage while maintaining a strong trade-off between efficiency and accuracy. Optimized for resource-constrained environments, it is suitable for on-device applications.
In the face of the exponential growth of digital data, DNA is expected to become a new storage medium. Image data makes up a large proportion of digital data. However, existing DNA data storage models are mainly designed for general files. To address this issue, we propose a novel image encoding method for DNA data storage. We employ discrete wavelet transform to decompose the image and utilize an improved exponent-mantissa representation for numerical data. Subsequently, we achieve enhanced compression performance through context-adaptive arithmetic coding. Additionally, we construct a dictionary between ternary sequences and oligonucleotides to generate nucleotide sequences that meet the specified constraints. Experiments show that our method outperforms JPEG-DNA and BioCoder in compression performance and generates higher-quality nucleotide sequences.
The latest initiative of Alliance for Open Media Video (AOM), named as AOM Video Model (AVM), is expected to introduce new coding tools to enhance compression benefits. Semi-Decoupled Partitioning (SDP) in AVM decouples the shared tree to support separate block partitioning for the luma and chroma channels from 64x64. Further, Chroma from Luma (CfL) is a chroma-only coding tool in AVM that applies collocated luma reconstructed samples in predicting chroma samples. The dependency of reconstructed luma samples in CfL can result in a delayed decoding process of chroma blocks in separate tree partitioning and introduce a worst-case latency of 4096 luma samples. In response, this study proposes a CfL constrained strategy to reduce the worst-case latency by selectively disallowing the CfL mode in a given chroma partition tree. Detailed latency analysis is also provided to confirm the reduction of worst-case latency to 2048 luma samples. The experiments are implemented on top research-v10.0.0 under Common Test Conditions (CTC) V7. The experimental results show that when the worst-case decoder latency is minimized to 2048 luma samples, the coding loss can be kept to minimal with an average loss of 0.02% for the YUV components in random access configurations with no change in encoder and decoder timings.
Our challenge is to develop efficient block-overlapping regression strategies for clean and noisy images. Block-based approaches allow massive parallel computation to achieve fast rendering - which is important for many applications. In this paper we focus on block-overlapping regression with sparse Steered-Mixture-of-Experts (SMoE) models which have shown to provide excellent results for divers applications. Block-overlapping SMoE regression provides multi-hypotheses for estimating each pixel value from clean or noisy observations - usually simple averaging of hypotheses is performed. We introduce and investigate several linear and nonlinear Filters/Networks for fusing the estimates more efficiently. On the clean Kodak image test set our best Fusion Network can improve over the "average" filter with an impressive 4.6-5.7 dB gain. For noisy observations the gain is around 1.6 dB - with denoising results in the same ball-park as BM3D.The developed strategies and results appear of more general interest beyond SMoE regression and have the potential to improve also on the plethora of other block-regression approaches such as DNN Autoencoders and kernel methods in 2D and 3D Gaussian Splatting
Moving Picture Expert Group (MPEG) launched Lenslet Video Coding (LVC) standard project in June, 2025 to investigate codec-agnostic tools for lenslet videos, recording spatial and angular light field information captured by plenoptic cameras or synthetically generated from multiview data, to ensure standard compatibility with Versatile Video Coding (VVC) while achieving much higher compression efficency. In this paper, a novel codec-agnostic coding framework for lenslet video is proposed. It integrates microimage cropping and alignment tool (MCA) in the pre-processing step. Then, edge pixel value estimation and optimization are proposed in both pre-processing and post-processing steps to signal and reconstruct the edge pixels that are multiview-rendering less effective by 60-degree directional compensation and pixel value fitting, thereby achieving a better trade-off between rendered multiview quality and lenslet quality. The proposed method achieves an average of 26.98% bitrate reduction relative to H.266/VVC, which is much higher than state-of-the-art methods, with much higher subjective quality in the reconstructed lenslet videos.
Specular reflections pose a significant challenge for object segmentation, as their sharp intensity transitions often mislead both conventional algorithms and deep learning based methods. However, as the specular reflection must lie on the surface of the object, this fact can be exploited to improve the segmentation masks. By identifying the largest region containing the reflection as the object, we derive a more accurate object mask without requiring specialized training data or model adaption. We evaluate our method on both synthetic and real world images and compare it against established and state-of-the-art techniques including Otsu thresholding, YOLO, and SAM2. Compared to the best performing baseline SAM2, our approach achieves up to 26.7% improvement in IoU, 22.3% in DSC, and 9.7% in pixel accuracy. Qualitative evaluations on real world images further confirm the robustness and generalizability of the proposed approach.
Dynamic Vision Sensors (DVS) asynchronously capture brightness changes at pixel-level precision, enabling high temporal resolution, wide dynamic range and low power consumption. To sustainably manage redundancy in event data, we introduce a novel lossless compression framework that exploits DVS-specific characteristics through two core representations: the Super Binary Map (SBM) and the Temporal Event Vector (TEV). SBM is a voxelized binary structure capturing the inherent spatio temporal sparsity and polarity of events, compressed effectively using Run-Length Encoding (RLE). TEV complements this by precisely encoding each event's timing into compact, variable-length vectors, optimized through a context-adaptive entropy coder inspired by Markov models. This spatio temporal representation substantially enhances compression efficiency. Our approach significantly outperforms conventional methods, achieving improvements up to 52 x over AVC, 32 x over HEVC, and 9 x over VVC at fine temporal scales. At ultra-fine granularity (10(-4)s or 10,000 fps), our method attains remarkable compression ratios (up to 307x), vastly exceeding traditional approaches (e.g., LZMA, Brotli, Zlib). This consistent and substantial margin highlights the method's adaptability to dynamic and static scenes, offering a scalable, domain-specific solution for efficient lossless compression of event-based vision data.
The video-based dynamic mesh coding standard is built on the concepts of mesh decimation to form a base-mesh, subdivision surfaces, displacement data, and a lifting transform. The displacement data describes how higher level of detail geometry can be reconstructed from the subdivided base-mesh. It is hierarchically predicted through each subdivision iteration and then coded using traditional 2D video codecs or arithmetically.Because the base-mesh is smaller in size, it allows for faster decoding and transfer from CPU to GPU. Applications should therefore perform the V-DMC reconstruction process on the GPU to take advantage of this.Currently, the standard specifies a step-by-step iterative reconstruction process, but this is not optimized for real-time rendering. In this paper, we describe a rendering method for V-DMC compressed mesh sequences that enables tessellation-based reconstruction of the base-mesh.
4D Gaussian Splatting (4DGS) has demonstrated advances in the dynamic scene representation. However, the time-varying attributes across frames introduce considerable storage and transmission costs, making 4DGS challenging to widely deploy. Existing compression methods struggle to obtain inter-frame residuals due to the unstructured nature of Gaussian representations, making explicit motion estimation and residual modeling inherently challenging. To address these, we propose a Training-Free 4D Gaussian Compression framework, TF4DGC, which transforms 4D Gaussian into a well-structured 2D representation, easy to estimate motion for coding, via a UV mapping. Specifically, we project 3D Gaussians onto a canonical sphere to obtain temporally consistent UV coordinates, and organize per-frame Gaussian attributes into multi-channel video sequences. This design enables the direct use of standard video codecs (e.g., AVC, HEVC) for compression, which is compatible with widespread hardware decoder support on laptops and mobile devices. Experimental results show that our method efficiently compresses both reconstructed and generated Gaussian scenarios, highlighting its general applicability. Our method offers a scalable and practical solution for 4DGS compression and facilitates real-time deployment in bandwidth constrained environments.
3D Gaussian Splatting (3DGS) has emerged as a high-quality and efficient alternative to Neural Radiance Fields (NeRF), offering distinct advantages in scene representation and suitability for multi-view 3D object detection (MV-3DOD) tasks. However, conventional 3DGS-based detection methods typically employ a reconstruction-then-detection pipeline, which is time-consuming and unsuitable for real-time applications. This approach arises from 3DGS’s original design for reconstruction tasks, which lacks network-based training. In this paper, we propose GS-3Det, an online Gaussian detection framework for MV-3DOD, achieving real-time performance through a single forward pass for scene reconstruction and detection. Specifically, we introduce a detection-aware Gaussian grid that enables directly prediction of explicit 3D Gaussians from multi-view images, enabling efficient and robust 3D scene understanding. Additionally, we propose a Dual-Path Consistency module that leverages 3D constraints to improve the accuracy of Gaussian grid representation and detection. Experiments on the ScanNet V2 dataset demonstrate that GS-3Det surpasses state-of-the-art methods by 3.5% in mAP@0.25 and 2.7% in mAP@0.5, underscoring its generalization capability and real-time performance.
Solving jigsaw puzzles, a long-standing challenge in both human cognition and artificial intelligence, has seen significant progress with modern computer vision techniques. In this paper, we introduce a diffusion-based framework for jigsaw puzzle reconstruction, leveraging denoising diffusion models to iteratively refine piece placements. Unlike prior methods that rely on anchored reference pieces and relative positioning, our approach directly regresses absolute positions, making it more flexible and generalizable. Additionally, we extend puzzle-solving beyond square pieces by incorporating polygonal partitions and employ DDIM for efficient inference. Our modular pipeline is adaptable to various puzzle formulations, and we demonstrate its effectiveness by achieving state-of-the-art performance on the JPwLEG benchmark.
Recent hardware encoders in GPUs and mobile SoCs enable real-time high-resolution video processing by constraining the supported encoding tools available in video coding standards. These constraints are also used to meet strict power, area, and memory limits of mobile platforms. This paper presents an analysis of the hardware-based High Efficiency Video Coding (HEVC) present in the high-performance NVIDIA NVENC within the RTX 4070Ti GPU, and the power-efficient encoder present in the Snapdragon 8 Gen 2 chip within the Samsung Galaxy S23+ smartphone. The analysis is performed in two perspectives: (1) the tool set constraints employed by each implementation are identified by a bitstream analysis on UHD encoded videos, and (2) a compression-efficiency evaluation of both encoders through a rate vs. distortion and Bjontegaard-Delta Rate (BD-Rate) analysis against the HEVC Test Model reference software. The results reveal the different design trade-offs between the platforms, offering valuable insights for hardware designers by highlighting the implementation choices of major industry players like NVIDIA and Qualcomm.
How to prompt a foundation model like CLIP towards a sketch expert is the question we seek to answer in this paper. Debates on the best way to prompt have been intense and divided, however converged on one particular point that of modelling prompt learning as context token optimisation. This paper scrutinises such technical route for sketch and argues the challenge is more than a stereotyped ask from context change. In particular, we pin down the problem to the dramatic cross-modality gap between sketch and the photo-centric visual world formed within CLIP. We first show through a pilot study that relocating a sketched object to a different spatial locality can significantly improve zero-shot CLIP performance on sketch. Our core contribution is then to regard spatial misalignment as the key to explaining poor sketch adaptation in CLIP prompts that a sketched object does not reside in a place as if it were part of the scene compositions of photo. Methodologically, we leverage a lightweight network that explicitly allows differentiable spatial manipulation of sketch data and design regulatory self-supervised signals to encourage proper convergence. We showcase consistent complementary power of this simple approach by building on top of 10 existing contemporary prompting methods on the sketch recognition task. For example, we outperform the strong prompting baseline CoOp by 2.57%, MaPle by 4.83% and AdaptFormer by 5.07%. Notably, the latter two beat the traditional full parameter fine-tuning (82.98%83.39% vs. 81.51%), and does so with less than 1% of the total training parameters.
