3D Multi-frame Fusion for Video Stabilization

In this paper, we present RStab, a novel framework for video stabilizationthat integrates 3D multi-frame fusion through volume rendering. Departing fromconventional methods, we introduce a 3D multi-frame perspective to generatestabilized images, addressing the challenge of full-frame generation whilepreserving structure. The core of our approach lies in Stabilized Rendering(SR), a volume rendering module, which extends beyond the image fusion byincorporating feature fusion. The core of our RStab framework lies inStabilized Rendering (SR), a volume rendering module, fusing multi-frameinformation in 3D space. Specifically, SR involves warping features and colorsfrom multiple frames by projection, fusing them into descriptors to render thestabilized image. However, the precision of warped information depends on theprojection accuracy, a factor significantly influenced by dynamic regions. Inresponse, we introduce the Adaptive Ray Range (ARR) module to integrate depthpriors, adaptively defining the sampling range for the projection process.Additionally, we propose Color Correction (CC) assisting geometric constraintswith optical flow for accurate color aggregation. Thanks to the three modules,our RStab demonstrates superior performance compared with previous stabilizersin the field of view (FOV), image quality, and video stability across variousdatasets.