High-quality and real-time frame interpolation on heterogeneous computing system

We put up forward a high-quality frame interpolation framework and its real-time implementation on heterogeneous computing system. To obtain a truthful optical flow field, the framework takes use of recent progresses in correspondence estimation including randomized search strategy and edge-preserved flow interpolation, and adapts them well into the system through a parallelized design. Firstly, dense optical flow estimation by PatchMatch algorithm is improved by the proposed propagation scheme of good matches inside divided pixel batches along with a new message passing phase between them. Then, the parallelization of this framework is two-fold: on device-level, different tasks such as flow estimation and edge detection are simultaneously executed on CPU and GPU, while on data-level, the interpolation of flow is performed in parallel for different pixels. Finally, with interleaved flow fields in forward and backward direction, multi-hypothesis based motion compensated frame interpolation will produce high-quality videos at low computation cost. Experiments on a 16-core CPU and GTX Titan 980 Ti GPU system achieve real-time frame interpolation of 1920×1080 videos at 60Hz. Objective and subjective comparisons demonstrate that our method has obtained state-of-the-art quality.