Neural Radiance Fields (NeRF) have demonstrated promising results in synthesizing novel view images from a set of unconstrained captured scenes. One important extension of NeRF is using it on non-rigid reconstruction. Although previous NeRF-based methods for dynamic scene reconstruction have presented visually appealing results, they still often show visual artifacts such as blurry or incorrect geometry of an object. One of the causes is that previous work performs reconstruction directly on the entire video sequence. The global temporal information over the video sequence introduces noise to the network, often leading to a non-optimal canonical space representation of the dynamic scene. In this paper, we present Local Temporal (LT) NeRF, a method to synthesize novel views of dynamic scenes using local temporal priors. Our novel LT module provides the local temporal priors using multi-view stereo sampling, and improves the deformation field reconstruction and hyper-space encoding. Our novel loss functions further supervise the NeRF for better optimization. We evaluate our method with dynamic scenes captured from monocular videos, outperforming the state-of-the-art.

Six degrees-of-freedom (6-DoF) video provides telepresence by enabling users to move around in the captured scene with a wide field of regard. Compared to methods requiring sophisticated camera setups, the image-based rendering method based on photogrammetry can work with images captured with any poses, which is more suitable for casual users. However, existing image-based rendering methods are based on perspective images. When used to reconstruct 6-DoF views, it often requires capturing hundreds of images, making data capture a tedious and time-consuming process. In contrast to traditional perspective images, 360° images capture the entire surrounding view in a single shot, thus, providing a faster capturing process for 6-DoF view reconstruction. This article presents a novel method to provide 6-DoF experiences over a wide area using an unstructured collection of 360° panoramas captured by a conventional 360° camera. Our method consists of 360° data capturing, novel depth estimation to produce a high-quality spherical depth panorama, and high-fidelity free-viewpoint generation. We compared our method against state-of-the-art methods, using data captured in various environments. Our method shows better visual quality and robustness in the tested scenes.

Six degrees-of-freedom (6-DoF) video provides telepresence by enabling users to move around in the captured scene with a wide field of regard. Compared to methods requiring sophisticated camera setups, the image-based rendering method based on photogrammetry can work with images captured with any poses, which is more suitable for casual users. However, existing image-based rendering methods are based on perspective images. When used to reconstruct 6-DoF views, it often requires capturing hundreds of images, making data capture a tedious and time-consuming process. In contrast to traditional perspective images, 360° images capture the entire surrounding view in a single shot, thus, providing a faster capturing process for 6-DoF view reconstruction. This article presents a novel method to provide 6-DoF experiences over a wide area using an unstructured collection of 360° panoramas captured by a conventional 360° camera. Our method consists of 360° data capturing, novel depth estimation to produce a high-quality spherical depth panorama, and high-fidelity free-viewpoint generation. We compared our method against state-of-the-art methods, using data captured in various environments. Our method shows better visual quality and robustness in the tested scenes.

Convolutional Neural Networks using RGB-D images as input have shown superior performance in recent research in the field of semantic segmentation. In RGB-D data, the depth channel encodes information from the 3D spatial domain, which has an inherent difference with the color channels. It thus needs to be treated in a special way, rather than just processed as another channel of the input signal. Under this purpose, we propose a simple but not trivial edge-aware convolutional kernel to utilize the geometric information contained in the depth channel to extract feature maps in a more effective manner. The edge-aware convolutional kernel is built upon regular convolutional kernel, thus, it can be used to restructure existing CNN models to achieve stable and effective feature extraction for RGB-D data. We compare our result with a previous method that is closely related to our to show our method can provide more effective and stable feature extraction.