It is a difficult challenge for humans to carry out environmental perception work at night and in low-light scenes. Depending on its extraordinary working performance in the dark, starlight camera is widely used in night driving assistance and various surveillance missions. However, the starlight camera images are lack of colorful information, which prevents users from understanding. This paper proposes a novel approach for colorizing starlight images using Generative Adversarial Network (GAN) architecture. The proposed method overcomes the time-space asynchronism of traditional heterogeneous data acquisition. We firstly introduce starlight-RGB image pairs generation. Inspired by 3D perspective transformation, we use LiDAR, camera and Inertial Measurement Unit (IMU) data to create generated visible images. We collect synchronous visible images, LiDAR points data and IMU data in the daytime and acquire LiDAR, starcam and IMU data at night. Such image pair generation method overcomes the difficulty of obtaining pairs of data and image pairs are aligned at pixel-level. As there are no reflection LiDAR points in the sky, the perspective projection images have no content in the sky areas. Based on supervised image-to-image translation GAN architecture, we use daytime RGB images as unpaired data, which is in order to restore the texture and color of the sky. We use KITTI dataset as validation, and get good experimental performance on our datasets.

针对智能交互界面中手势识别错误导致交互界面变化错误和手势不识别两个基本问题,本文设计并实现了基于手势交互的智能教学界面,该系统可以通过获取教师的手势信息与教师进行交互.主要创新点在于提出了多种手势对应同一语义的柔性映射交互算法.本文选取了14种自然交互手势,分析了对应同一语义的多种手势之间的共同特征.实验结果显示,该算法能够有效降低用户负荷.该算法已经用于一个基于手势交互的智能教学系统界面中.

An algorithm for flexible mapping (FM) from one gesture to multiple semantics in the same situational context is presented for the first time to reduce the cognitive and operating loads of an operator by using gesture commands. First, the foundation of the FM is built with the semantic-oriented difference features of behavior model (SDFBM) of the operator as implicit inputs. Second, the gesture–semantics FM algorithm and its extended process, namely, attribute classification for FM are proposed and implemented. Third, five commonly used gestures are designed to demonstrate the way a gesture is mapped to multiple semantics. Finally, several comparative experimental results are provided to demonstrate the superiority of the proposed methods to the state of the art. The main innovation of this study is that the same gesture in the same context can be mapped to several different semantics through the SDFBM feature recognition. This study provides an intelligent and natural interaction interface model for 3D platforms and key support for gesture-based implicit interaction design. The proposed algorithms are also tested or used in several applications, such as smart teaching interface and onboard vehicular systems and intelligent TV.

Gesture recognition plays a very important role in human-computer interaction, and depth based gesture recognition receives more attention because depth sensors have the advantages of capturing depth information and being robust to illumination changes. At present, gesture recognition algorithms focus on the accuracy and efficiency of recognition on general data sets, but ignore the specific needs of interactive gestures in specific scenarios, and the general gesture data sets can not meet the actual interactive needs, which also limits the application and promotion of human-computer interaction. Aiming at the above problems, this paper creates a specific hand gesture data set, which dedicated to interactive teaching of intelligent classroom teaching, and proposes a deep neural network model which integrates global and local information for gesture recognition. The experimental results demonstrate that the proposed deep model achieves 93.6% recognition rate of 17 commonly used gestures and verifies the performance in virtual geometry teaching.

The combination of Augmented Reality (AR) technology and gesture recognition has become increasingly important in the educational system. In the smart classroom application scenario, the virtual three-dimensional complex model is combined and interacted with real hands based on augmented reality technology. The main contribution of this paper is to determine the coordinates of the five-finger tip of the hand, and calculate the initial length from the center of the 3D model. When the distance between five fingertips and three-dimension mode is shorter than the initial length during the process of fingers grasp, the three-dimension mode will get smaller. Light interaction is achieved by establishing the relationship of the two spatial models with coordinates. AR is achieved under Vuforia, and the three-dimensional model gradually becomes smaller with the five fingers grasping. The proposed method is used in an Intelligent Teaching System and run at the mobile terminal. The experimental results show that the proposed method works well in ITS.

Fingertip recognition and tracking is a key problem in gesture recognition. The current fingertip locating and tracking method is complicated, or needs to be labelled artificially. In this paper, a particle filtering method based on edge feature and pixel ratio is proposed to track the target finger in a complex background. We set the region of each particle as a fixed value, and calculate the edge orientation histograms and the proportion of the body pixels of each particle area. The similarity degree of edge features is measured by the Bhattacharyya distance, and a new similarity measure is defined to measure the similarity degree of pixel ratio. These two feature similarities will be linearly combined to track the target model. And then we calculate the farthest point from the center of the contour in the predicted model and update the target model again. The results show that the method can track the target fingertip accurately and effectively in real time, under the condition of interference.

Vanishing point detection is a challenging task due to the variations in road types and its cluttered background. Currently, most existing texture-based methods detect the vanishing point using pixel-wise voting map generation, which suffers from high computational complexity and the noise votes introduced by the incorrectly estimated texture orientations. In this paper, a block wise weighted soft voting scheme is developed for good performance in complex road scenes. First, the gLoG filters are applied to estimate the texture orientation of each pixel. Then, the image is divided into blocks in a sliding fashion, and a histogram is constructed based on the texture orientation of pixels within each block to obtain the dominant orientation bin. Instead of using the texture orientation of all valid pixels within each block, only the dominant orientation bin is utilized to perform a weighted soft voting. The experimental results on the benchmark dataset show that the proposed method achieves the best performance among all, when compared with the state-of-the-art works.

In this paper we propose a gaze zone estimation method using deep learning. Compared with traditional method, our method does not need the procedure of calibration. In the proposed method, a Kinect is used to capture the video of a computer user, which is pre-processed to suppress illumination variations. After that, haar cascade classifier is adopted to detect the face region and eye region. Then, the eye region is used to estimate the gaze zone on the monitor via a trained CNN (Convolution Neural Network). Experimental results show that the proposed method has a high accuracy, which can be applied in human-computer interaction.

Since the dynamic gesture recognition algorithm is influenced by rotation,translation and scaling,and realtime gesture recognition is still a challenging issue,we propose a dynamic gesture recognition algorithm which is based on the combination of gesture binary descriptor and Hausdorff-like distance template matching.Firstly,we converted the segmented gesture image to the standardized image,then calculated the main direction of gesture in the image,and built a 2D rectangular coordinate system to extract the gesture features.Secondly,the specific dynamic gesture type can be identified according to changes in the amount of gesture pixel points from the top five frames.Next,we used gesture binary descriptor to select a part of gesture from the specific dynamic gesture type.Finally,the method of Hausdorff-like distance template matching is used to recognize the final gesture.The main innovation of this paper embodies in two aspects.Firstly,the dynamic gesture type recognition and the method of gestures binary descriptor matching proposed in this paper greatly shorten the time cost of dynamic gesture recognition.Secondly,Hausdorff-like distance method with the main direction of gesture not only has the in-variance on rotation,translation and scaling gestures,but also has a higher recognition rate on smaller gestures.Experimental results show that this algorithm can achieve real-time correct recognition of gestures in relatively stablelight conditions.The overall recognition rate can reach 95％,the recognition rate of scal-ing gestures is more than 92％ andthe recognition rate of rotation gestures is more than 87％.This algorithm has been applied in a human-computer interfacesystem based on hand gesture.

A motion-compensated frame interpolation (MCFI) algorithm for multiview video based on inter-view and intra-view correlations is proposed in this paper. First, unidirectional motion estimation (ME) is implemented to obtain forward and backward motion vector fields (MVFs). Subsequently, occlusion blocks in previous and current frames are detected. Then, an inter-view-correlation based method is adopted for occlusion handling. After that, the motion vector (MV) outliers are detected and corrected by considering spatial, color and depth information of current viewpoint video. Finally, MVs are assigned to the interpolated frame for frame reconstruction. Experimental results demonstrate that the proposed algorithm provides better performance than existing 2D and 3D video MCFI methods.

Because the interaction in the existing virtual assembly systems is featured with machine-centered explicit interaction, the machine can not effectively perceive the user’s intentions, which leads to a higher level of the user’s cognitive load and operational burden. In order to improve the intelligence, implicit human-computer interaction was introduced into the virtual assembly system, and an algorithm of user’s intention perception based on the scene task was proposed in this paper. Firstly, explicit interactive knowledge database and implicit interactive knowledge database were constructed respectively based on the scene knowledge model and the user knowledge model. Then the user’s explicit operational intentions and implicit operational intentions were deduced by expert system. Finally man-machine opposite movement algorithms were proposed in two scenarios. On the one hand, scene could move actively close to the user when human hand was moving towards the target position. On the other hand, the object could rotate actively when human hand was grasping or releasing the object. Experimental results showed that the proposed algorithm achieved human-computer cooperative interaction and reduced the user's operational burden and cognitive burden.

An efficient frame interpolation method for 3D video is proposed in this paper. First, the motion vector field (MVF) of the interpolated frame is estimated using bidirectional motion estimation method. Then, the blocks in the interpolated frame are classified into occlusion and normal blocks according to the depth information. After that, the motion vector (MV) outliers of the normal blocks are detected and corrected. Furthermore, the MVs of the occlusion blocks are post-processed using a foreground matching method, and one occlusion block is segmented into foreground and background parts, which will be interpolated discriminatively. Simulation results show that the proposed method achieves higher PSNR and SSIM values than traditional motion-compensated frame interpolation (MCFI) methods, and moreover, the visual quality of the interpolated frames using the proposed method is better.

In view interpolation, information missing often exists in initial depth map, moreover, disocclusion regions usually occur along the foreground object boundaries after 3D warping. Generally, initial depth map and warped depth map have a strong influence on the performance of view interpolation. However, most of existing view interpolation algorithms only emphasize hole filling of the warped color image. In this paper, a superpixel-based method is proposed for initial depth map enhancement and warped depth map hole filling. Firstly, the color image is segmented using simple linear iterative clustering (SLIC) algorithm, and after that, the associated depth map is segmented with the same label. Then, the depth-missing pixels are recovered by considering color and depth superpixel information jointly. Additionally, holes of the disocclusion regions in the warped depth map can also be filled efficiently via superpixel-based segmentation. Experimental results show that with the proposed method the quality of the interpolated view has been improved significantly in terms of both subjective and objective evaluations.

In this study, a Man-machine Finger-guessing game is designed based on the IntelliSense and Man-machine coordination mechanism of hand gesture. The image sequence is obtained by the Kinect and the human hand is extracted using segmentation and skin color modeling. The proposed SCDDF (Shape Context Density Distribution Feature), which combined DDF (Density Distribution Feature) algorithm and shape context recognition algorithm, is used to extract gesture identity. Gestures are finally identified by registering with templates in the pre-established gesture library. Furthermore, we proposed a new human-computer cooperative mechanism, including two points: (1) The virtual interface is used to control the ‘Midas Touch problem’. (2) The whole game is more natural and smooth. In the aspect of gesture recognition, we combined DDF algorithm and shape context recognition algorithm, and proposed the SCDDF algorithm. The new algorithm improved recognition rate by 14.3% compared with DDF algorithm.

View synthesis is an important building block in three dimension (3D) video processing and communications. Based on one or several views, view synthesis creates other views for the purpose of view prediction (for compression) or view rendering (for multiview-display). The quality of view synthesis depends on how one fills the occlusion area as well as how the pixels are created. Consequently, luminance adjustment and hole filling are two key issues in view synthesis. In this paper, two views are used to produce an arbitrary virtual synthesized view. One view is merged into another view using a local luminance adjustment method, based on local neighborhood region for the calculation of adjustment coefficient. Moreover, a maximum neighborhood spreading strength hole filling method is presented to deal with the micro texture structure when the hole is being filled. For each pixel at the hole boundary, its neighborhood pixels with the maximum spreading strength direction are selected as candidates; and among them, the pixel with the maximum spreading strength is used to fill the hole from boundary to center. If there still exist disocclusion pixels after once scan, the filling process is repeated until all hole pixels are filled. Simulation results show that the proposed method is efficient, robust and achieves high performance in subjection and objection.

The use of three-dimensional (3D) gesture input devices is important and necessary in 3D systems, but such devices face considerable challenges posed by the high dimensionality of dexterous hand motion. The objective of this study is to achieve real-time interaction in object selection and direct manipulation in 3D application systems by capturing and visualizing the interaction intentions and probing the cognitive behavior models of users. An interactive operation procedure is divided into three stages: object selection, manipulation and reset. Trajectory scene interaction (TSI) is proposed for object selection starting from a fixed position called a forward point (FP). The manipulations exerted on the selected object include grasping and translation. After these manipulations, the gesture is reset to the FP. This work offers four novel contributions. First, flexible object selection and atomic operation model (AOM)-based animations are fused to form a uniform, real-time human-computer interaction (HCI) paradigm. Second, a cognitive behavior model is proposed for recognizing and reacting to hand gestures as captured by a monocular camera. Third, an approach to capturing, expressing, and probing a user's interaction intention is presented. Fourth, a 3D real-time gesture input interface is achieved. The use of the proposed HCI interface, which offers fast speed, satisfactory accuracy and a responsive user experience, is demonstrated in virtual assembly, a game of chess, dialing a cell phone number and menu operation.

Multiple watermarking technique, embedding several watermarks in one carrier, has enabled many interesting applications. In this study, a novel multiple watermarking algorithm is proposed based on the spirit of spread transform dither modulation (STDM). It can embed multiple watermarks into the same region and the same transform domain of one image; meanwhile, the embedded watermarks can be extracted independently and blindly in the detector without any interference. Furthermore, to improve the fidelity of the watermarked image, the properties of the dither modulation quantizer and the proposed multiple watermarks embedding strategy are investigated, and two practical optimization methods are proposed. Finally, to enhance the application flexibility, an extension of the proposed algorithm is proposed which can sequentially embeds different watermarks into one image during each stage of its circulation. Compared with the pioneering multiple watermarking algorithms, the proposed one owns more flexibility in practical application and is more robust against distortion due to basic operations such as random noise, JPEG compression and volumetric scaling.

This paper proposes a novel motion compensated frame interpolation (MCFI) method based on adaptive hybrid motion estimation and compensation (AHMEC) for 3D video. In our work, we deal with the problem of ghost artifacts around the foreground object boundaries by considering motion and depth information jointly. First, the motion vector field (MVF) of the interpolated frame is estimated using block-based method. We use depth and motion information to distinguish the occlusion areas in the interpolated frame. Then, an adaptive pixel-based motion estimation (ME) method is applied to detail the MVF in the covering and uncovering areas. Simulation results show that the proposed MCFI algorithm outperforms the conventional algorithms in terms of objective and subjective performances.

Finding 3D hand models corresponding to the user's 3D hand pose in the initial frames makes the initialization for the 3D hand model much more significant in 3D human hand tracking. Blending computer interaction techniques and cognition theories, a novel initialization approach for 3D hand model is put forward in the present paper to make the initialization process more human-oriented. The proposed initialization process is primarily divided into three steps. The first step is the approximate classification of the user's poses as dominated by a computer. The second step is to adjust by freehand as dominated by the user. The third step is to modify the 3D hand model as dominated by the computer. The present study attempts to describe and shape the user's behavioral model, upon the initialization algorithm is designed and optimized. To improve the performance of the initialization algorithm, User experience, time cost, and accuracy are fused into an evaluation criterion for the optimization of the proposed algorithm. The main contributions of the present work consists of modeling the operator's cognitive behavior, attempting to answer why and how the cognitive behavioral model guides the proposed algorithm in assigning tasks for the initialization between human and computer. The experimental results demonstrate good performance by the proposed method and its potential applications. In addition, the proposed approach could provide the user with an easier, more pleasurable, and more satisfactory experience. The developed initialization system is successfully applied to several application systems.

A frame rate up-conversion (FRUC) method for 3D video (3DV) is presented in this paper. Inspired by the fact that moving foreground objects draw more attention of the viewers, in our method depth guided extended block matching (DGE-BM) is adopted to maintain the completeness of the foreground object. We first obtain the motion vector field (MVF) of the interpolated frame via block-based bi-directional motion estimation (ME). And the blocks of the interpolated frame are classified according to the depth information. Then, the boundary blocks are divided into sub-blocks, whose motion vectors (MVs) are estimated using DGE-BM. Finally, the refined MVF is applied to do motion compensation. Experimental results show that the frame interpolation quality of the proposed method achieves significantly improvement comparing with existing algorithms.