Optical motion capture systems have attracted much interest over the past years, due to their advantages with respect to non-optical counterparts. Moreover, with the technological advances on camera devices, computer graphics and computational methodologies, it becomes technically and economically feasible to consider motion capture systems made of large networks of cameras with embedded communication and processing units on board (i.e., smart cameras). In this case, the approaches relying on the classical 3D reconstruction methods would become inefficient, since their nature is intrinsically centralized. For this reason, we propose a distributed 3D reconstruction algorithm, which relies on a specific organization of cameras to remarkably speed up the scene reconstruction task. Indeed, numerical and experimental results show that the proposed computational scheme overcomes classical centralized solutions, in terms of reconstruction speed. Furthermore, the high processing speed does not compromise the estimation accuracy, since the algorithm is designed to be robust to occlusions and noise.
It is evident that the prevalence of the autism spectrum disorder (ASD) has been in a worrying situation with an average rate of 1% around the world. However, existing clinical diagnostic methods for ASD and medical conditions are far from adequate for a wide range of early screening and diagnosis, especially in remote areas. In this paper, a detailed protocol aiming to describe the clinical task - Expressing Needs with Index Finger Pointing (ENIFP) is proposed and a multi-sensor vision system is developed to record and analyze the performance of the children. Mutual gaze and gesture are considered to be the main basis to judge the performance of the children. The improved SSD algorithm is applied to locate the hands and recognize gestures. Eight subjects including 5 typically developed adults and 3 children (2 ASD and 1 non-ASD) are invited to the experiment. The result shows that the system can check the mutual gaze and recognize the gesture - index finger pointing accurately and demonstrates the proposed system has the potential to assist screen ASD.
Several computer vision and artificial intelligence projects are nowadays exploiting the manifold data distribution using, e.g., the diffusion process. This approach has produced dramatic improvements on the final performance thanks to the application of such algorithms to the kNN graph. Unfortunately, this recent technique needs a manual configuration of several parameters, thus it is not straightforward to find the best configuration for each dataset. Moreover, the brute-force approach is computationally very demanding when used to optimally set the parameters of the diffusion approach. We propose to use genetic algorithms to find the optimal setting of all the diffusion parameters with respect to retrieval performance for each different dataset. Our approach is faster than others used as references (brute-force, random-search and PSO). A comparison with these methods has been made on three public image datasets: Oxford5k, Paris6k and Oxford105k.
Single Image Super-Resolution (SISR) aims to generate a high-resolution (HR) image of a given low-resolution (LR) image. The most of existing convolutional neural network (CNN) based SISR methods usually take an assumption that a LR image is only bicubicly down sampled version of an HR image. However, the true degradation (i.e. the LR image is a bicubicly downsampled, blurred and noisy version of an HR image) of a LR image goes beyond the widely used bicubic assumption, which makes the SISR problem highly ill-posed nature of inverse problems. To address this issue, we propose a deep SISR network that works for blur kernels of different sizes, and different noise levels in an unified residual CNN-based denoiser network, which significantly improves a practical CNN-based super-resolver for real applications. Extensive experimental results on synthetic LR datasets and real images demonstrate that our proposed method not only can produce better results on more realistic degradation but also computational efficient to practical SISR applications.
The success of Deep Learning (DL) algorithms in computer vision tasks have created an on-going demand of dedicated hardware architectures that could keep up with the their required computation and memory complexities. This task is particularly challenging when embedded smart camera platforms have constrained resources such as power consumption, Processing Element (PE) and communication. This article describes a heterogeneous system embedding an FPGA and a GPU for executing CNN inference for computer vision applications. The built system addresses some challenges of embedded CNN such as task and data partitioning, and workload balancing. The selected heterogeneous platform embeds an Nvidia® Jetson TX2 for the CPU-GPU side and an Intel Altera® Cyclone10GX for the FPGA side interconnected by PCIe Gen2 with a MIPI-CSI camera for prototyping. This test environment will be used as a support for future work on a methodology for optimized model partitioning.
Super-multiview video consists in a 2D arrangement of cameras acquiring the same scene and it is a well-suited format for immersive and free navigation video services. However, the large number of acquired viewpoints calls for extremely effective compression tools. View synthesis allows to reconstruct a viewpoint using nearby cameras texture and depth information. In this work we explore the potential of recent advances in view synthesis algorithms to enhance the compression performances of super-multiview video. Towards this end we consider five methods that replace one viewpoint with a synthesized view, possibly enhanced with some side information. Our experiments suggest that, if the geometry information (i.e. depth map) is reliable, these methods have the potential to improve rate-distortion performance with respect to traditional approaches, at least for some specific content and configuration. Moreover, our results shed some light about how to further improve compression performance by integrating new view-synthesis prediction tools within a 3D video encoder.
We propose a solution to increase the privacy of people recorded with security cameras without decreasing the details stored in the videos. We strongly believe that CCTV recordings are a necessary and precious source of information to be analyzed when a crime or other unfortunate events happen; for this reason, we would like to have powerful surveillance systems that are able to hide the identity of the recorded people while allowing subsequent recovery of the data. We use face morphing algorithms in order to transform the faces in such a way that the protected video keeps the original likeness but does not leak sensitive face information. We store the transformed information in a decentralized way and we adopt smart contracts on permissioned blockchains to guarantee that in order to retrieve the data, a collection of trusted authorities must give their consent.
The implementation of algorithms based on Deep Learning at edge visual systems is currently a challenge. In addition to accuracy, the network architecture also has an impact on inference performance in terms of throughput and power consumption. This demo showcases per-layer inference performance of various convolutional neural networks running at a low-cost edge platform. Furthermore, an empirical model is applied to predict processing time and power consumption prior to actually running the networks. A comparison between the prediction from our model and the actual inference performance is displayed in real time.
Person re-identification (re-id) remains an important task that aims to retrieve a person's images from an image dataset, given a probe image. The lack of cross-view (pose variations) training data and significant intra-class (domain) variations across different cameras make re-id more challenging. To solve these issues, this work proposes a Domain and Pose Invariant Generative Adversarial Network (DPI-GAN) to generate images for both domain and pose variations capture. It is based on a CycleGAN structure in which the generator networks are conditioned on a new pose. Identity and pose discriminators networks are used to monitor the image generation process. These generated images are used for learning domain and pose invariant features to improve the performance of person re-identification.
Stereo matching consists in extracting 3D information from digital images, such as those obtained by a CCD camera. It is an important issue under several real world applications, such as positioning systems for mobile robots, augmented reality systems, etc. In previous works one of the most popular trend to address the stereo matching challenge is that compares scene information from two viewpoints (left-right) with an eppipolar geometry via correlation metrics. In regard to the correlation metrics, most previous works compute the similarity between pixels in the left image and pixels in the right image using a correlation index computed on neighborhoods of these pixels called correlation windows. Unfortunately, in order to preserve edges, small correlation windows need to be used, while, for homogeneous areas, large correlation windows are required. To address this problem, we lay down on the hypothesis that small correlation windows combined with large correlation windows should deliver accurate results under homogeneous areas while at the same time edges are preserved. To validate our hypothesis, in this paper a similarity criterion based on the grayscale homogeneity of the correlation window being processed is presented. Preliminary results are encourageous, validates our hypothesis and demonstrated the viability performance and scope of the proposed approach.
Convolutional Neural Networks have demonstrated their competence in extracting information from data, especially in the field of computer vision. Their computational complexity prompts for hardware acceleration. The challenge in the design of hardware accelerators for CNNs is providing a sustained throughput with low power consumption, for what FPGAs have captured community attention. In CNNs pooling layers are introduced to reduce model spatial dimensions. This work explores the influence of pooling layers modification in some state-of-the-art CNNs, namely AlexNet and SqueezeNet. The objective is to optimize hardware resources utilization without negative impact on inference accuracy.
Time-to-digital converters based on Nutt method are especially suitable for FPGA implementation. They are able to provide high resolution, range and linearity with low resources usage. The core of this architecture consist in a coarse counter for long range, a fine time interpolator for high resolution and real-time calibration for high linearity. This paper reviews different time interpolation and real-time calibration techniques. Moreover, a comparison of state-of-the-art FPGA-based TDCs is presented as well.
This demo displays an autonomous image acquisition and processing system that operates simultaneously with two image sensors either in the visible and the Long Wave Infrared Band (LWIR), inside the Infrared (IR) band. The entire system is controlled a Raspberry Pi board that allows to easily program image processing algorithms to process the images acquired with each sensor. It is a competitive alternative to conventional commercial closed systems with infrared cameras. The proposed imaging system can be easily adapted to different operation scenarios by adding new peripherals, sensors and full custom image processing algorithms.
In this demo, we present a method for optimal video camera positioning. The final objective is to maximize visual coverage in complex indoor environments. Starting from a predefined camera model and environmental setup, we employ a particle swarm optimizer (PSO) to determine the best configuration of the camera network to satisfy our target coverage. The target coverage objectives can vary and depend on realistic factors such as lighting and obstacles.
The research on autonomous vehicles has grown increasingly with the advent of neural networks. Dynamic obstacle detection is a fundamental step for self-driving vehicles in traffic environments. This paper presents a comparison of state-of-art object detection techniques like Faster R-CNN, YOLO and SSD with 2D image data. The algorithms for detection in driving, must be reliable, robust and should have a real time performance. The three methods are trained and tested on PASCAL VOC 2007 and 2012 datasets and both qualitative and quantitative results are presented. SSD model can be seen as a tradeoff for speed and small object detection. A novel method for object detection using 3D data (RGB and depth) is proposed. The proposed model incorporates two stage architecture modality for RGB and depth processing and later fused hierarchically. The model will be trained and tested on RGBD dataset in the future.
Camera resectioning is essential in computer vision and 3D reconstruction to estimate the position of matching pinhole cameras in 3D worlds. While the internal camera parameters are usually known or can be easily computed offline, in camera networks extrinsic parameters need to be computed each time a camera changes position, thus not allowing for smooth and dynamic network reconfiguration. In this work we propose a fully markerless, unsupervised, and automatic tool for the estimation of the extrinsic parameters of a camera network, based on 3D human mesh recovery from RGB videos. We show how it is possible to retrieve, from monocular images and with just a weak prior knowledge of the intrinsic parameters, the real-world position of the cameras in the network, together with the floor plane. Our solution also works with a single RGB camera and allows the user to dynamically add, re-position, or remove cameras from the network.
Virtual reality (VR) is an experience taking place within simulated and immersive environments. Although in recent years several virtual reality applications such as, virtual reality gaming, medical educational and military training applications have been developed; one important limitation still remains for the tracking sensor. Commercial headsets such as the Oculus Rift or HTC Vive have tracking sensors which project active signals to the user's body and limits the motion understanding. To address this problem, we propose a novel passive sensor (which consist of an FPGA-based smart camera) which computes the optical flow an estimates semantic information about the user movement inside the camera fabric. Then, using these semantic information as feedback for the virtual reality engine; accurate tracking without active signals being projected to the user's body and with the capability to implement several cameras in order to achieve a better movement understanding is possible. Preliminary results are encourageous, demonstrating the possibility of a visual-based tracking approach suitable for virtual reality applications.
In this paper, a face liveness detection benchmark is established and maintained, wherein 400 images pairs captured with binocular camera are made openly available for research purposes. This image dataset contains numbers of people with varied expressions, illumination, and background environment conditions, etc., among which 200 image pairs characterize lively human faces, and the other half are planar face pictures. The benchmark provides a platform for researchers to test stereo matching algorithms for liveness detection, where the detection performance is evaluated via a binary classification on the detection response for being a lively human or not. The feasibility of SIFT features are verified based on a comparative analysis of the classification result, and a set of optimal parameters for the classification is given which provides a reference for further research. * denotes the equal contributions.
Image recognition is an open challenge in computer vision since its early stages. The application of deep neural networks yielded significant improvements towards its solution. Despite their classification abilities, deep networks need datasets with thousands of labelled images and prohibitive computational capabilities to achieve good performance. To address some of these challenges, the CapsNet neural architecture has been recently proposed as a promising machine learning model for image classification based on the idea of capsules. A capsule is a group of neurons whose output represents the presence of features of the same entity. In this paper, we start from the CapsNet architecture to explore and analyse the interaction between the presence of features within certain, similar classes. This is achieved by means of techniques for the features interaction, working on the outputs of two independent capsule-based models. To understand the importance of the interaction between capsules, extensive experiments have been carried out on four challenging dataset. Results show that the exploitation of capsules interaction yields to performance improvements.
Low-level handcrafted features (e.g., edge and saliency) dominate the design of traditional algorithms, and endow themselves the effective capability of dealing with simple classification problems. However, such excellent properties have not been well explored in the popular deep convolutional neural networks (DCNNs). In this paper, we propose a new deep model, termed Guided Convolutional Networks (GCNs), using low-level handcrafted features to guide the training process of the DCNNs, which can be used in the following vision tasks. Furthermore, signature structure is also investigated with saliency information as a basic block to help the network to be slim. With the modulated binary convolutional way, the memory of our small network is reduced by 132 theoretically. Experiments also demonstrate GCNs have comparable results in presicion compared with state-of-the-art networks such as Wide-ResNet (WRN) while reducing the network dramatically.
