The capability of automatically detecting people and understanding their behaviours is an important functionality of intelligent video systems. The interest in behaviour understanding has effectively increased in recent years, motivated by a societal needs. This thesis is focused on the development of algorithms and solutions for different environments exploiting top-view RGB-D data. In particular, the addressed topics refer to Human Behaviour Understanding (HBU) in different research areas. The first goal is to implement people detection algorithms in order to monitor the people activities. To this aim, a thorough study of the state of the art has been conducted to identify the advantages and weakness. An initial approach, proposed in this thesis, is based on Computer Vision (CV) techniques, it regards the extraction the head of each person using depth data. Another approach is based on deep learning and is proposed to simplify the heads detection implementation in chaotic environments and in the presence of people with different heights. These solutions are validated with a specific dataset. The second goal is to extract several feature from subject and to identify possible interactions that they have with the surrounding environment. Finally, in order to demonstrate the actual contribution of algorithms for understanding the human behaviour in different environments, several use cases have been realized and tested.

The paper “Convolutional Networks for semantic Heads Segmentation using Top-View Depth Data in Crowded Environment” [1] introduces an approach to track and detect people in cases of heavy occlusions based on CNNs for semantic segmentation using top-view RGB-D visual data. The purpose is the design of a novel U-Net architecture, U-Net 3, that has been modified compared to the previous ones at the end of each layer. In order to evaluate this new architecture a comparison has been made with other networks in the literature used for semantic segmentation. The implementation is in Python code using Keras API with Tensorflow library. The input data consist of depth frames, from Asus Xtion Pro Live OpenNI recordings (.oni). The dataset used for training and testing of the networks has been manually labeled and it is freely available as well as the source code. The aforementioned networks have their stand-alone Python script implementation for training and testing. A Python script for the on-line prediction in OpenNI recordings (.oni) is also provided. Evaluation of the networks has been made with different metrics implementations (precision, recall, F1 Score, Sørensen-Dice coefficient), included in the networks scripts.

The recent advancement and development of computer electronic devices has led to the adoption of smart home sensing systems, stimulating the demand for associated products and services. Accordingly, the increasingly large amount of data calls the machine learning (ML) field for automatic recognition of human behaviour. In this work, different deep learning (DL) models that learn to classify human activities were proposed. In particular, the long short-term memory (LSTM) was applied for modelling spatio-temporal sequences acquired by smart home sensors. Experimental results performed on the Center for Advanced Studies in Adaptive Systems datasets show that the proposed LSTM-based approaches outperform existing DL and ML methods, giving superior results compared to the existing literature.

Understanding shopper behaviour is one of the keys to success for retailers. In particular, it is necessary that managers know which retail attributes are important to which shoppers and their main goal is to improve the consumer shopping experience. In this work, we present sCREEN (Consumer REtail ExperieNce), an intelligent mechatronic system for indoor navigation assistance in retail environments that minimizes the need for active tagging and does not require metrics maps. The tracking system is based on Ultra-wideband technology. The digital devices are installed in the shopping carts and baskets and sCREEN allows modelling and forecasting customer navigation in retail environments. This paper contributes the design of an intelligent mechatronic system with the use of a novel Hidden Markov Models (HMMs) for the representation of shoppers' shelf/category attraction and usual retail scenarios such as product out of stock or changes on store layout. Observations are viewed as a perceived intelligent system performance. By forecasting consumers next shelf/category attraction, the system can present the item location information to the consumer, including a walking route map to a location of the product in the retail store, and/or the number of an aisle in which the product is located. Effective and efficient design processes for mechatronic systems are a prerequisite for competitiveness in an intelligent retail environment. Experiments are performed in a real retail environment that is a German supermarket, during business hours. A dataset, with consumers trajectories, timestamps and the corresponding ground truth for training as well as evaluating the HMM, have been built and made publicly available. The results in terms of Precision, Recall and F1-score demonstrate the effectiveness and suitability of our approach, with a precision value that exceeds the 76% in all test cases.

Person re-identification is an important topic in retail, scene monitoring, human-computer interaction, people counting, ambient assisted living and many other application fields. A dataset for person re-identification TVPR (Top View Person Re-Identification) based on a number of significant features derived from both depth and color images has been previously built. This dataset uses an RGB-D camera in a top-view configuration to extract anthropometric features for the recognition of people in view of the camera, reducing the problem of occlusions while being privacy preserving. In this paper, we introduce a machine learning method for person re-identification using the TVPR dataset. In particular, we propose the combination of multiple k-nearest neighbor classifiers based on different distance functions and feature subsets derived from depth and color images. Moreover, the neighborhood component feature selection is used to learn the depth features' weighting vector by minimizing the leave-one-out regularized training error. The classification process is performed by selecting the first passage under the camera for training and using the others as the testing set. Experimental results show that the proposed methodology outperforms standard supervised classifiers widely used for the re-identification task. This improvement encourages the application of this approach in the retail context in order to improve retail analytics, customer service and shopping space management.

Qualitative assessment of general movements in preterm infants is widely used in clinical practice. It can enable early detection of neurological dysfunctions and consequent neuromotor impairments in high risk infants. However, the outcome of these assessments is not standardized and it is influenced by examiner's subjective interpretation. For this reason, there is an increasing interest in the use of automated movement recognition technologies being applied in this field. In this work, we use a video-based system for preterm infant's movements assessment to provide a 3D motion analysis method able to extract some important indicators from the sequence of depth images collected by using an RGB-D sensor placed over the infant lying on the crib. The advantage of the proposed method is that it is objective, contactless, non-invasive, easy to install, affordable and suitable to be used in an indoor environment with poor lighting, as might be rooms in the Neonatal Intensive Care Unit, where these infants are taken into care. Experimental results show that the proposed method is able to derive from statistical analysis of depth data some key performance indicators, each of which describes different characteristics of the infant's spontaneous movements. Preliminary tests are conducted in the experimental phase on a preterm infant hospitalized in a women's and children's hospital. The project can be used to investigate the relationship between the characteristics of spontaneous movements and the presence of pathologies as cerebral palsy or other minor neurological dysfunctions.

Customer experience depends not only on the aspects which retailers can easily control, but also on emotional factors that are unpredictable. In this paper, a Multi-Task Multi-Kernel learning approach is proposed to recognise positive users' emotion in a retail scenario. The overall system is composed by the Ultra-Wide Band (UWB) tracking system and a consumer smartwatch device. Data gathered from sensors are combined in a multi-kernel scenario to estimate shoppers emotion (i.e., valence and arousal) which is strictly correlated to different shoppers feelings. Results in term of accuracy and macro-F1 score prove the effectiveness and the suitability of the proposed approach.

The information of the number of passengers getting in/off a vehicle is very important for public bus transport companies. In fact, the operators need to estimate the number of travellers using their vehicles for marketing purposes, for evaluating transit service capacities and allocating the proper number of buses for each connection-line. The goal of this work is to provide a system for counting and monitoring passengers, both adults and children, at the entrance of bus. This system is mainly based on an RGB-D sensor, located over each bus door, and image processing and understanding software. The RGB image could be affected by a high luminescence sensibility, whereas depth data allow a greater reliability and accuracy in people counting. The correctness and effectiveness of our method has been confirmed by experiments conducted in a real scenario. Furthermore, this approach has the advantage of being computationally inexpensive and flexible enough to obtain, in real time, statistical measures on the amount of people present in the bus, with the use of an Analytical Processing System (a separate process) that accesses the data stored in the database and extracts statistical data and knowledge about the bus passengers.

This paper presents a literature review on the use of RGB-D camera for people detection and tracking. Our aim is to use this state-of-the-art report to demonstrate the potential of top-view configuration for people detection and tracking applications in several sub-domains, to outline key limitations and to indicate areas of technology, where solutions for remaining challenges may be found. The survey examines the success of RGB-D cameras because of their affordability and for the additional rough depth information coupled with visual images that provide. These cameras in configuration top-view have already been successfully applied in the several fields to univocally identify people and to analyse behaviours and interactions. From this report, it emerges that detecting and tracking people can be a valuable source of information for many fields and purposes.

Automated recognition of Activities of Daily Living allows to identify possible health problems and apply corrective strategies in Ambient Assisted Living (AAL). Activities of Daily Living analysis can provide very useful information for elder care and long-term care services. This paper presents an automated RGB-D video analysis system that recognises human ADLs activities, related to classical daily actions. The main goal is to predict the probability of an analysed subject action. Thus, abnormal behaviour can be detected. The activity detection and recognition is performed using an affordable RGB-D camera. Human activities, despite their unstructured nature, tend to have a natural hierarchical structure; for instance, generally making a coffee involves a three-step process of turning on the coffee machine, putting sugar in cup and opening the fridge for milk. Action sequence recognition is then handled using a discriminative Hidden Markov Model (HMM). RADiaL, a dataset with RGB-D images and 3D position of each person for training as well as evaluating the HMM, has been built and made publicly available.

Nowadays, home devices with network capabilities are widely used. The technology integration offers new and exciting opportunities to increase the device connectivity within a home for many proposals of home automation. In this paper, it has been developed a framework that allows to quickly develop new hardware and software complex systems, rapidly integrate new classes of devices in existing systems and control and centralize the data. The preliminary results obtained are already consistent and demonstrate its suitability and its effectiveness.

Video analytics, involves a variety of techniques to monitor, analyse, and extract meaningful information from video streams. In this light, person re-identification is an important topic in scene monitoring, human computer interaction, retail, people counting, ambient assisted living and many other computer vision research. The existing datasets are not suitable for activity monitoring and human behaviour analysis. For this reason we build a novel dataset for person re-identification that uses an RGB-D camera in a top-view configuration. This setup choice is primarily due to the reduction of occlusions and it has also the advantage of being privacy preserving, because faces are not recorded by the camera. The use of an RGB-D camera allows to extract anthropometric features for the recognition of people passing under the camera. The paper describes in details the collection and construction modalities of the dataset TVPR. This is composed by 100 people and for each video frame nine depth and colour features are computed and provided together with key descriptive statistics.

In the context of retailing, the monitoring of consumer behaviours is particularly important for supporting vendors in their management and marketing decisions. Many studies have been carried out about various aspects and consequences of different behaviours. However, only recently the potential of computing systems is being used for automated data collection and processing. In this work, we present a novel pervasive system able to automatically monitor consumer behaviour in front of shelves in an intelligent retail environment (IRE). Data collected are stored into a cloud server for data analysis and insights, ready to be used by a Decision Support System (DSS).

The development of reliable and precise indoor localization systems would considerably improve the ability to investigate shopper movements and behavior inside retail environments. Previous approaches used either computer vision technologies or the analysis of signals emitted by communication devices (beacons). While computer vision approaches provide higher level of accuracy, beacons cover a wider operational area. In this paper, we propose a sensor fusion approach between active radio beacons and RGB-D cameras. This system, used in an intelligent retail environment where cameras are already installed for other purposes, allows an affordable environment set-up and a low operational costs for customer indoor localization and tracking. We adopted a Kalman filter to fuse localization data from radio signals emitted by beacons are used to track users' mobile devices and RGB-D cameras used to refine position estimations. By combing coarse localization datasets from active beacons and RGB-D data from sparse cameras, we demonstrate that the indoor position estimation is strongly enhanced. The aim of this general framework is to provide retailers with useful information by analyzing consumer activities inside the store. To prove the robustness of our approach, several tests were conducted into a real indoor showroom by analyzing real customers behavior with encouraging results. (C) 2016 Elsevier B.V. All rights reserved.

In this study, we present a medical imaging method to estimate the preterm infant respiratory rate with a non invasive embedded RGB-D sensor. The respiratory rate is derived by measuring morphological chest wall movements with a depth sensor. The performances of our method are evaluated by comparing the values of respiratory rate measurements obtained using our method with those resulting from a standard reference device, used as a benchmark. Experimental results showed that the proposed method can correctly measure the respiratory rate in preterm infants and activates an alarm signal when respiratory rate values go out of the physiological range or when the infant remains stationary for a long time. As future development, the project can be used as multi purpose non invasive test on movements and their symmetry or on the infants' ability to correctly inflate.

Medical devices, generally, have been unique units, so that a typical hospital room in an intensive care ward hosts a big number of stand-alone devices, each one with its own user interface. Nowadays, these devices have some connection mechanisms, but the goal is to go beyond the simple connectivity that permits only the reading of the measured value on a computer monitor, to move towards a more complex device integration that offers the possibility to stream device data directly into patient electronic records, to create a database for further statistical investigations and to integrate the information derived from multiple medical devices into a single display. In this paper, we propose a method to connect and integrate into a cloud-based infrastructure the bilirubinometer, which is a typical Neonatal Intensive Care Unit biomedical device that measures the amount of bilirubin in infants' blood, which is symptomatic of neonatal jaundice. After data extraction process from the instrument, the database is normalised according to a record format defined in XML language. Then, through Web Services of importation, data is transferred to the cloud database. It enables online sharing of encrypted healthcare data, reported in a standard language to allow a complete interoperability as well as a total security of shared data. Our proposed infrastructure allows the communication between different medical devices of a Neonatal Intensive Care Unit and permits the automation of the process of device data collection, transmission, storage, processing and availability for medical staff.

The aim of this work is to define a fall detection video system for indoor environments based on a RGB-D sensor and a low power and low cost embedded system that processes the sensor data in order to provide a description of human activities in the field of the Ambient Assisted Living. The RGB image is affected by a high luminescence sensibility, so the depth data have the aim to improve the human activity recognition. The system is usable in a sufficiently small room and it requires a RGB-D sensor located in the center of the ceiling and an embedded system connected on a computer network. The embedded system controls the RGB-D sensor and, in the mean time, classifies the images using computer vision algorithms based on the depth map. “Water Filling” algorithm or “Multi-Level Segmentation” algorithm are used to detect person. For each person, the system detects the position with respect to the room, estimating also the human posture. Among the features extracted we enumerate the height, the head size and the distance between the head and the shoulders. The system tracks a person through the frames starting from the first identification. Further, group interactions are monitored and analyzed. The posture detection algorithm takes into account the distance between the person head and the floor during the time. During the experimental phase, conducted in many domestic scenarios, the effectiveness of the proposed solution has been proved, that is fast, accurate and ables to provide a fall map in-home fall risk assessment.

The success of pervasive smart environments lies in the capacity to involve visitors to interact with them. It is essential for retail stores. In this paper we describe the setting-up of a low cost system for the indoor localization and customer interaction, developed with a complex infrastructure of wireless embedded sensors. The creation of a responsive store allows customers to connect the real world to their smart devices and will overcome the lack of ubiquity in public spaces; furthermore, from in-venue analytics and proximity sensor it is possible to customize the user experience. First of all we describe the whole sensor network. We go in deep into the active beacon technology adopted for this study. Then, thanks to the analytics, we present a data evaluation with the aim of determining the best sensor arrangement, according to several user tests. Beside the strong enhancement of human interaction, the results of our essay demonstrate how embedded localization systems could be a useful source for data collection beside the strong enhancement of human interaction. This paper is focused to help retailers and insiders for many purposes such as products development or improvement, segmentation strategies and human behaviour analyses into such stores where the embedded computing augment the environment.

The aim of this paper is to present an integrated system consisted of a RGB-D camera and a software able to monitor shoppers in intelligent retail environments. We propose an innovative low cost smart system that can understand the shoppers’ behavior and, in particular, their interactions with the products in the shelves, with the aim to develop an automatic RGB-D technique for video analysis. The system of cameras detects the presence of people and univocally identifies them. Through the depth frames, the system detects the interactions of the shoppers with the products on the shelf and determines if a product is picked up or if the product is taken and then put back and finally, if there is not contact with the products. The system is low cost and easy to install, and experimental results demonstrated that its performances are satisfactory also in real environments.