This paper presents a comprehensive analysis of the energy consumption characteristics of a Silicon (Si)-based Reconfigurable IoT (RIoT) node developed in the initial phase of the SUPERIOT project, focusing on key operating states, including Bluetooth Low Energy (BLE) communication, Narrow-Band Visible Light Communication (NBVLC), sensing, and E-ink display. Extensive measurements were conducted to establish a detailed energy profile, which serves as a benchmark for evaluating the effectiveness of subsequent optimizations and future node iterations. To minimize the energy consumption, multiple optimizations were implemented at both the software and hardware levels, achieving a reduction of over 60 software modifications alone. Further improvements were realized by optimizing the E-ink display driving waveform and implementing a very low-power mode for non-communication activities. Based on the measured data, three measurement-based energy consumption models were developed to characterize the energy behavior of the node under: (i) normal, unoptimized operation, (ii) low-power, software-optimized operation, and (iii) very low-power, hardware-optimized operation. These models, validated with new measurement data, achieved an accuracy exceeding 97 predicting energy consumption in diverse configurations.

Road scene parsing is a crucial capability for self-driving vehicles and intelligent road inspection systems. Recent research has increasingly focused on enhancing driving safety and comfort by improving the detection of both drivable areas and road defects. This article reviews state-of-the-art networks developed over the past decade for both general-purpose semantic segmentation and specialized road scene parsing tasks. It also includes extensive experimental comparisons of these networks across five public datasets. Additionally, we explore the key challenges and emerging trends in the field, aiming to guide researchers toward developing next-generation models for more effective and reliable road scene parsing.

This paper presents a novel approach for multimodal data fusion based on the Vector-Quantized Variational Autoencoder (VQVAE) architecture. The proposed method is simple yet effective in achieving excellent reconstruction performance on paired MNIST-SVHN data and WiFi spectrogram data. Additionally, the multimodal VQVAE model is extended to the 5G communication scenario, where an end-to-end Channel State Information (CSI) feedback system is implemented to compress data transmitted between the base-station (eNodeB) and User Equipment (UE), without significant loss of performance. The proposed model learns a discriminative compressed feature space for various types of input data (CSI, spectrograms, natural images, etc), making it a suitable solution for applications with limited computational resources.

With the ongoing evolution of autonomous driving technology, road environment perception systems have become a significant focus of research. However, there is currently a paucity of comprehensive survey articles that provide a systematic overview of state-of-the-art (SoTA) computer vision techniques and vibration methods for road defect detection, particularly with regards to deep learning methods. This chapter aims to fill this gap by describing the sensing technologies for vision-based and vibration-based road environment data acquisition, summarizing several public datasets for pothole and crack detection, and providing a comprehensive review of SoTA road defect detection algorithms. Additionally, this chapter also discusses the core competencies of autonomous vehicle software systems, such as planning and control. Finally, we offer a glimpse into the future of autonomous driving systems, envisioning the integration of both vision and motion sensors. We believe that this survey will act as a helpful guide for advancing road defect detection technology, providing strategic advice and practical guidance to those involved in developing such systems.

A new method for multimodal sensor fusion is introduced. The technique relies on a two-stage process. In the first stage, a multimodal generative model is constructed from unlabelled training data. In the second stage, the generative model serves as a reconstruction prior and the search manifold for the sensor fusion tasks. The method also handles cases where observations are accessed only via subsampling i.e. compressed sensing. We demonstrate the effectiveness and excellent performance on a range of multimodal fusion experiments such as multisensory classification, denoising, and recovery from subsampled observations.

The task of semantic segmentation involves labeling each pixel in an image with its corresponding object class, which is achieved by clustering regions belonging to the same category using artificial intelligence. This is an important step from image processing to image analysis and has numerous applications in areas such as automatic driving, image enhancement, and 3D map reconstruction. With the emergence of deep learning, several sophisticated and efficient algorithms have been developed for this task. This chapter aims to review these methods, starting with a discussion of state-of-the-art semantic segmentation methods for both single modality and data fusion, emphasizing their contributions and significance in the field. Additionally, an overview of commonly used datasets is provided to assist researchers in selecting the appropriate dataset for their needs and goals. A comprehensive summary of evaluation metrics used to assess semantic segmentation results, along with corresponding benchmarks for a number of classic datasets, is also presented. Finally, practical applications of semantic segmentation in autonomous driving are explored, and conclusions are drawn on the current state of the art.

Stereo matching is the process of generating dense correspondences in stereo images in order to create a disparity map for depth perception. Stereo matching is different from flow estimation task due to stereo rectification, which ensures that correspondences are always co-linear in a pair of stereo images. Stereo vision has become increasingly popular in mobile devices, such as autonomous cars and unmanned aerial vehicles, thanks to recent advances in full-feature embedded microcomputers. However, due to limited computing resources, there is a growing need for stereo matching algorithms that strike a balance between disparity estimation accuracy and efficiency. Challenges in this field include the lack of disparity ground truth, domain adaptation, and intractable areas such as occlusions. This chapter covers the fundamentals of stereopsis, including the perspective camera model and epipolar geometry, and reviews the most advanced stereo matching algorithms. It also explores disparity confidence measures, disparity estimation evaluation metrics, and publicly available datasets and benchmarks, before summarizing the outstanding challenges in this field.

Human Activity Recognition (HAR) training data is often privacy-sensitive or held by non-cooperative entities. Federated Learning (FL) addresses such concerns by training ML models on edge clients. This work studies the impact of privacy in federated HAR at a user, environment, and sensor level. We show that the performance of FL for HAR depends on the assumed privacy level of the FL system and primarily upon the colocation of data from different sensors. By avoiding data sharing and assuming privacy at the human or environment level, as prior works have done, the accuracy decreases by 5-7%. However, extending this to the modality level and strictly separating sensor data between multiple clients may decrease the accuracy by 19-42%. As this form of privacy is necessary for the ethical utilisation of passive sensing methods in HAR, we implement a system where clients mutually train both a general FL model and a group-level one per modality. Our evaluation shows that this method leads to only a 7-13% decrease in accuracy, making it possible to build HAR systems with diverse hardware.

The pervasiveness of Wi-Fi signals provides significant opportunities for human sensing and activity recognition in fields such as healthcare. The sensors most commonly used for passive Wi-Fi sensing are based on passive Wi-Fi radar (PWR) and channel state information (CSI) data, however current systems do not effectively exploit the information acquired through multiple sensors to recognise the different activities. In this paper, we explore new properties of the Transformer architecture for multimodal sensor fusion. We study different signal processing techniques to extract multiple image-based features from PWR and CSI data such as spectrograms, scalograms and Markov transition field (MTF). We first propose the Fusion Transformer, an attention-based model for multimodal and multi-sensor fusion. Experimental results show that our Fusion Transformer approach can achieve competitive results compared to a ResNet architecture but with much fewer resources. To further improve our model, we propose a simple and effective framework for multimodal and multi-sensor self-supervised learning (SSL). The self-supervised Fusion Transformer outperforms the baselines, achieving a F1-score of 95.9%. Finally, we show how this approach significantly outperforms the others when trained with as little as 1% (2 minutes) of labelled training data to 20% (40 minutes) of labelled training data.

This paper presents a comprehensive dataset intended to evaluate passive Human Activity Recognition (HAR) and localization techniques with measurements obtained from synchronized Radio-Frequency (RF) devices and vision-based sensors. The dataset consists of RF data including Channel State Information (CSI) extracted from a WiFi Network Interface Card (NIC), Passive WiFi Radar (PWR) built upon a Software Defined Radio (SDR) platform, and Ultra-Wideband (UWB) signals acquired via commercial off-the-shelf hardware. It also consists of vision/Infra-red based data acquired from Kinect sensors. Approximately 8 hours of annotated measurements are provided, which are collected across two rooms from 6 participants performing 6 daily activities. This dataset can be exploited to advance WiFi and vision-based HAR, for example, using pattern recognition, skeletal representation, deep learning algorithms or other novel approaches to accurately recognize human activities. Furthermore, it can potentially be used to passively track a human in an indoor environment. Such datasets are key tools required for the development of new algorithms and methods in the context of smart homes, elderly care, and surveillance applications.

Recently, governments have been resorting to cutting-edge artificial intelligence technologies to facilitate the digital transformation of smart cities. Remarkable progress has been made to strengthen smart city governance and sustainability, especially in road condition assessment. Road data acquisition and defect detection, two major processes of intelligent road condition assessment, play an important role in ensuring road maintainability while providing maximum traffic security and driving comfort. Traditional manual visual inspection is inefficient and lacks objectivity. Therefore, intelligent road condition assessment systems developed based on data-driven techniques have received increasing attention. This chapter presents the state-of-the-art intelligent road condition assessment systems, the existing challenges, and future development trends.

Nowadays, an increasing amount of attention is being devoted towards passive and non-intrusive sensing methods. The prime example is healthcare applications, where on-body sensors are not always an option or in other applications which require the detection and tracking of unauthorized (non-cooperative) targets within a given environment. Therefore, in this paper we present a dataset consisting of measurements obtained from Radio-Frequency (RF) devices. Essentially, the dataset consists of Ultra-Wideband (UWB) data in the form of Channel Impulse Response (CIR), acquired via a Commercial Off-the-Shelf (COTS) UWB equipment. Approximately 1.6 hours of annotated measurements are provided, which are collected in a residential environment. This dataset can be used to passively track a target's location in an indoor environment. Additionally, it can also be used to advance UWB-based Human Activity Recognition (HAR) since three basic human activities were recorded, namely, sitting, standing and walking. We anticipate that such datasets may be utilized to develop novel algorithms and methodologies for healthcare, smart homes and security applications.

Road damage is an inconvenience and a safety hazard, severely affecting vehicle condition, driving comfort, and traffic safety. The traditional manual visual road inspection process is pricey, dangerous, exhausting, and cumbersome. Also, manual road inspection results are qualitative and subjective, as they depend entirely on the inspector's personal experience. Therefore, there is an ever-increasing need for automated road inspection systems. This chapter first compares the five most common road damage types. Then, 2-D/3-D road imaging systems are discussed. Finally, state-of-the-art machine vision and intelligence-based road damage detection algorithms are introduced.

The use of Wi-Fi signals for human sensing has gained significant interest over the past decade. Such techniques provide affordable and reliable solutions for healthcare-focused events such as vital sign detection, prevention of falls and long-term monitoring of chronic diseases, among others. Currently, there are two major approaches for Wi-Fi sensing: (1) passive Wi-Fi radar (PWR) which uses well established techniques from bistatic radar, and channel state information (CSI) based wireless sensing (SENS) which exploits human-induced variations in the communication channel between a pair of transmitter and receiver. However, there has not been a comprehensive study to understand and compare the differences in terms of effectiveness and limitations in real-world deployment. In this paper, we present the fundamentals of the two systems with associated methodologies and signal processing. A thorough measurement campaign was carried out to evaluate the human activity detection performance of both systems. Experimental results show that SENS system provides better detection performance in a line-of-sight (LoS) condition, whereas PWR system performs better in a non-LoS (NLoS) setting. Furthermore, based on our findings, we recommend that future Wi-Fi sensing applications should leverage the advantages from both PWR and SENS systems.

Collision-free space detection is a critical component of autonomous vehicle perception. The state-of-the-art algorithms are typically based on supervised deep learning. Their performance is dependent on the quality and amount of labeled training data. It remains an open challenge to train deep convolutional neural networks (DCNNs) using only a small quantity of training samples. Therefore, in this article, we mainly explore an effective training data augmentation approach that can be employed to improve the overall DCNN performance, when additional images captured from different views are available. Due to the fact that the pixels in collision-free space (generally regarded as a planar surface) between two images, captured from different views, can be associated using a homography matrix, the target image can be transformed into the reference view. This provides a simple but effective way to generate training data from additional multiview images. Extensive experimental results, conducted with six state-of-the-art semantic segmentation DCNNs on three datasets, validate the effectiveness of the proposed method for enhancing collision-free space detection performance. When validated on the KITTI road benchmark, our approach provides the best results, compared with other state-of-the-art stereo vision-based collision-free space detection approaches.

Road defects can severely affect the safety of road users and vehicle conditions. Over the past decade, due to the limited amount of labeled training data, machine vision-based road defect detection approaches have been mainly used, while machine/deep learning-based methods were merely discussed. With the recent development of artificial intelligence, convolutional neural network (CNN)-based road defect detection systems for automated road condition assessment have become an active sphere of study. In this regard, this paper presents a comprehensive road defect detection system based on computer stereo vision, non-linear regression, and CNN. A dense disparity image is first estimated from a pair of stereo road images using an efficient stereo matching algorithm. The estimated disparity image is then transformed to better identify road defects by minimizing a global energy function w.r.t. road disparity projection model coefficients and stereo rig roll angle, using the non-linear regression approach. Finally, three popular semantic segmentation CNNs are trained using the transformed disparity images. Extensive experiments are conducted to demonstrate the performance of our proposed road defect detection approach. The achieved pixel-level accuracy and intersection over union (IoU) are 98.37% and 67.65%, respectively.

Urban digital twin (UDT) technologies offer new opportunities for intelligent road inspection (IRI). This paper first reviews the state-of-the-art algorithms used in the two key components of UDT-based IRI systems: (1) multi-temporal, multi-dimension, multi-score, and heterogeneous road data acquisition, and (2) road distress detection. This paper then summarizes the UDTIRI competition, organized in conjunction with IEEE Bigdata 2022. More details on our competition are available at sites.google.com/view/udtiri-workshop/bigdata-2022.

Road crack detection is a crucial civil infrastructure inspection task. Road crack detection is generally performed by either certified inspectors or structural engineers. Nevertheless, this process is time-consuming and subjective. Deep convolutional neural networks (DCNNs) have demonstrated compelling results for image classification, but there are currently no comprehensive comparisons among them, in regard to road crack detection. Therefore, in this paper, we conduct extensive experiments to compare 30 state-of-the-art (SoTA) DCNNs for road crack detection: Each DCNN is trained on a training set; The best performing models are selected on the validation set; Their performance is further quantified on a test set with respect to six evaluation metrics: precision, recall, accuracy, F-score, area under receiver operating characteristic (AUROC), and runtime. The experimental results suggest that road crack detection is a relatively easy image classification task. All the SoTA DCNNs perform similarly. The DCNNs evaluated in this study also achieved very similar performance when only a small amount of training data is available. Furthermore, PNASNet achieved the best trade-off between speed and accuracy, and thus, it is more practical to be used for real-time and robust road crack detection. Moreover, it was found that the best DCNN models did not generalize well when tested on new unseen data sets consisting of images not specifically related to road cracks.

In this paper, we introduce a novel suspect-and-investigate framework, which can be easily embedded in a drone for automated parking violation detection (PVD). Our proposed framework consists of: 1) SwiftFlow, an efficient and accurate convolutional neural network (CNN) for unsupervised optical flow estimation; 2) Flow-RCNN, a flow-guided CNN for car detection and classification; and 3) an illegally parked car (IPC) candidate investigation module developed based on visual SLAM. The proposed framework was successfully embedded in a drone from ATG Robotics. The experimental results demonstrate that, firstly, our proposed SwiftFlow outperforms all other state-of-the-art unsupervised optical flow estimation approaches in terms of both speed and accuracy; secondly, IPC candidates can be effectively and efficiently detected by our proposed Flow-RCNN, with a better performance than our baseline network, Faster-RCNN; finally, the actual IPCs can be successfully verified by our investigation module after drone re-localization.

Human Activity Recognition (HAR) is becoming increasingly important in smart homes and healthcare applications such as assisted-living and remote health monitoring. In this paper, we use Ultra-Wideband (UWB) and commodity WiFi systems for the passive sensing of human activities. These systems are based on a receiver-only radar network that detects reflections of ambient Radio-Frequency (RF) signals from humans in the form of Channel Impulse Response (CIR) and Channel State Information (CSI). An experiment was performed whereby the transmitter and receiver were separated by a fixed distance in a Line-of-Sight (LoS) setting. Five activities were performed in between them, namely, sitting, standing, lying down, standing from the floor and walking. We use the high-resolution CIRs provided by the UWB modules as features in machine and deep learning algorithms for classifying the activities. Experimental results show that a classification performance with an F1-score as high as 95.53% is achieved using processed UWB CIR data as features. Furthermore, we analysed the classification performance in the same physical layout using CSI data extracted from a dedicated WiFi Network Interface Card (NIC). In this case, maximum F1-scores of 92.24% and 80.89% are obtained when amplitude CSI data and spectrograms are used as features, respectively.