The virtual viewpoint is perceived as a new technique in virtual navigation, as yet not supported due to the lack of depth information and obscure camera parameters. In this paper, a method for achieving close-up virtual view is proposed and it only uses optical flow to build parallax effects to realize pseudo 3D projection without using depth sensor. We develop a bidirectional optical flow method to obtain any virtual viewpoint by proportional interpolation of optical flow. Moreover, with the ingenious application of the optical-flow-value, we achieve clear and visual-fidelity magnified results through lens stretching in any corner, which overcomes the visual distortion and image blur through viewpoint magnification and transition in Google Street View system.

Occlusion is visible in only one frame and cannot be seen in the other frame which is a vital challenge in video stitching. Occlusion always brings ghost artifacts in the blended area. Meanwhile, the traditional image stitching approaches ignore temporal consistency and cannot avoid flicking problem. To solve these challenges, we propose a unified framework in which the stitching quality and stabilization both perform well. Specifically, we explicitly detect the potential occlusion regions to indicate blending information. Then, based on the occlusion maps, we choose a proper strip in the overlapped region as the blending area. With spatial–temporal Bayesian view synthesis, spatial ghost-like artifacts can be significantly eliminated and the output videos can be kept stable. The experimental results show the out performance of the proposed approach compared to state-of-the-art approaches.

Vessel segmentation of digital retinal images plays an important role in diagnosis of diseases such as diabetics, hypertension and retinopathy of prematurity due to these diseases impact the retina. In this paper, a novel Size-Invariant Fully Convolutional Neural Network (SIFCN) is proposed to address the automatic retinal vessel segmentation problems. The input data of the network is the patches of images and the corresponding pixel-wise labels. A consecutive convolution layers and pooling layers follow the input data, so that the network can learn the abstract features to segment retinal vessel. Our network is designed to hold the height and width of data of each layer with padding and assign pooling stride so that the spatial information maintain and up-sample is not required. Compared with the pixel-wise retinal vessel segmentation approaches, our patch-wise segmentation is much more efficient since in each cycle it can predict all the pixels of the patch. Our overlapped SIFCN approach achieves accuracy of 0.9471, with the AUC of 0.9682. And our non-overlap SIFCN is the most efficient approach among the deep learning approaches, costing only 3.68 seconds per image, and the overlapped SIFCN costs 31.17 seconds per image.

Face recognition has attracted a lot of attention in the last decades and achieved high recognition rate under controlled environment. More and more researchers now focus on face recognition in the wild, which is difficult because of the variance of pose, illumination, occlusion and so on. In this paper, we aim to solve this problem by combining image retrieval and feature weighting. By image retrieval method, we can find those face images in the gallery set which are the most similar to the probe face image. After getting similar face subset, feature weighting is then executed on this subset. This process includes two steps. In the first step, we learn a weight for each single feature in this subset by finding its nearest neighbor. In the second step, inspired by frequent item mining method we learn a weight for a group of features. In the testing process, by weighted nearest neighbor voting for both single and grouped features, we classify the probe image to the class which has the highest similarity score. We evaluate our method on AR and Pubfig83 face data sets. Experiment shows that our method has achieved state-of-the-art performance.

Autostereoscopic 3DTV is becoming an exciting media that enable us to view a 3D scene from more than one viewpoint. Meanwhile, considered as the ultimate autostereoscopic 3DTV, Free-viewpoint TV (FTV) can provide arbitrary views by freely synthesizing and changing viewpoints. Essentially, either 3DTV or FTV is based on virtual view synthesis using captured views along with corresponding depth information. In this paper, we study how virtual views can be reliably generated from multiple captured videos for 3D display. One key challenge is that the required depth information may contain depth errors, leading to uncomfortable artifacts in the synthesized view. We review the recent progress in virtual view synthesis methods where depth reliability is considered to handle synthesis artifacts and improve the quality of the virtual view. Not only for intermediate virtual view, have we also presented high-quality close-up view synthesis methods for wider navigation in 3DTV and FTV.

In this paper, we present a new view synthesis method in multiview camera configurations of Free viewpoint TV (FTV) where potential depth errors are considered. The emphasis is on the artifacts eliminating for photorealistic synthesis especially near object boundaries. In contrast to conventional techniques which ignore geometry errors, we first categorize the artifact cases and depth modes. Furthermore, this paper infers the complementarity principle of the artifacts from left and right references. This complementarity guarantees the effectiveness of our reliability-based synthesis. The reliability reasoning is crucial for artifacts reduction. The reliable and unreliable areas from different views can be correctly labeled. Then artifacts caused by unreliable pixels from one reference can be replaced by the reliable pixels from the other reference. As a final result, artifacts of novel view are demonstrated to be significantly reduced on different multiview sequences.

View synthesis using depth maps is a well-known technique for exploiting the redundancy between multi-view videos. In this paper, we deal with the bitrates of view synthesis at the decoder side of FTV that would use compressed depth maps and views. Both inherent depth estimation error and coding distortion would degrade synthesis quality. The focus is to reduce bitrates required for generating the high-quality virtual view. We employ a reliable view synthesis method which is compared with standard MPEG view synthesis software. The experimental results show that the bitrates required for synthesizing high-quality virtual view could be reduced by utilizing our enhanced view synthesis technique to improve the PSNR at medium bitrates.

View synthesis using depth maps is an important application in 3D image processing. In this paper, a novel method is proposed for the plausible view synthesis of Free-viewpoint TV (FTV), using two input images and their depth maps. The depth estimation based on stereo matching is known to be error-prone, leading to noticeable artifacts in the synthesized new views. To produce high-quality view synthesis, we introduce a probabilistic framework which constrains the reliability of each pixel of new view by Maximizing Likelihood (ML). The spatial adaptive reliability is provided by incorporating Gamma hyper-prior and the synthesis error approximation. Furthermore, we generate the virtual view by solving a Maximum a Posterior (MAP) problem using graph cuts. We compare the proposed method with other depth based view synthesis approaches on MPEG test sequences. The results show the outperformance of our method both at subjective artifacts reduction and objective PSNR improvement.

View synthesis using depth maps is a crucial application for Free-viewpoint TV (FTV). In this paper, we propose a novel reliability based view synthesis method using two references and their depth maps. The depth estimation with stereo matching is known to be error-prone, leading to noticeable artifacts in the synthesized new views. In order to provide plausible virtual views for FTV, our focus is on the error suppression for the synthesized view. We innovatively introduce the continuous reliability using error approximation by the reference cross-check. The new view interpolation algorithm is generated with the criterion of Least Sum of Squared Errors (LSSE). Furthermore, the proposed algorithm can be considered as a reliable version of the conventional linear view blending. We experimentally demonstrate the effectiveness of our framework with MPEG standard test sequences. The results show that our method outperforms state-of-the-art view interpolation methods both at eliminating artifacts and improving PSNR.

In computer graphics, 3D objects are projected onto the viewing plane in either parallel or perspective mode. As far as perspective projection is concerned, traditional rendering engines are designed to carry out only linear planar projections, as in OpenGL. But what if we expect to get certain distorted views? For instance, we see distorted scenes when we look through a rain drop on the windshield. When we play video games, we also might want to have a pair of magic lenses that let us see a deformed virtual world. Currently the nonlinear projections and implementations still remain largely unexplored in 3D graphic environments. In the first part of this thesis, several nonlinear projection models are investigated and an algorithm is presented to apply these models to perform nonlinear perspective projections and 3D view deformations in real time. Wide-angle lenses are often used to take landscape photos and they are also useful for pictures of large groups of people, but usually they introduce distortions into photos. Researches have been done to remove these distortions from the images. However, for some artistic reasons people may want to keep them. So can we simulate this effect in the computer graphics visual world? I developed a polynomial approximation model and applied the real-time algorithm introduced in the first part of the thesis to achieve the barrel distortion effect of a wide- angle lens. An experimental system based on the above algorithm and models has been developed to navigate 3D world with various deformed views.