Machine learning-based embedded systems for safety-critical applications, such as aerospace and autonomous driving, must be robust to perturbations caused by soft errors. As transistor geometries shrink and voltages decrease, modern electronic devices become more susceptible to background radiation, increasing the concern about failures produced by soft errors. The resilience of deep neural networks (DNNs) to these errors depends not only on target device technology but also on model structure and the numerical representation and arithmetic precision of their parameters. Compression techniques like pruning and quantization, used to reduce memory footprint and computational complexity, alter both model structure and representation, affecting soft error robustness. In this regard, although often overlooked, the choice of activation functions (AFs) impacts not only accuracy and trainability but also compressibility and error resilience. This paper explores the use of bounded AFs to enhance robustness against parameter perturbations, while evaluating their effects on model accuracy, compressibility, and computational load with a technology-agnostic approach. We focus on encoder-decoder convolutional models developed for semantic segmentation of hyperspectral images with application to autonomous driving systems. Experiments are conducted on an AMD-Xilinx's KV260 SoM.

The use of hyperspectral imaging (HSI) for autonomous navigation is a promising field of research that aims to improve the accuracy and robustness of detection, tracking, and scene understanding systems based on vision sensors. The combination of advanced computer algorithms, such as deep neural networks (DNNs), and small-size snapshot HSI cameras allows to strengthen the reliability of those vision systems. Using HSI, some intrinsic limitations of greyscale and RGB imaging in depicting physical properties of targets related to the spectral reflectance of materials (metamerism) are overcome. Despite the promising results of many published HSI-based computer vision developments, the strict requirements of safety-critical applications such as autonomous driving systems (ADS) regarding latency, resource consumption, and security are prompting the migration of machine learning (ML)-based solutions to edge platforms. This involves a thorough software/hardware co-design scheme to distribute and optimize the tasks efficiently among the limited resources of computing platforms. With respect to inference, the over-parameterized nature of DNNs poses significant computational challenges for real-time on-the-edge deployment. In addition, the intensive data preprocessing required by HSI, which is frequently overlooked, must be carefully managed in terms of memory arrangement and inter-task communication to enable an efficient integrated pipeline design on a system on chip (SoC). This work presents a set of optimization techniques for the practical co-design of a DNN-based HSI segmentation processor deployed on a field programmable gate array (FPGA)-based SoC targeted at ADS, including key optimizations such as functional software/hardware task distribution, hardware-aware preprocessing, ML model compression, and a complete pipelined deployment. Applied compression techniques significantly reduce the complexity of the designed DNN to 24.34% of the original operations and to 1.02% of the original number of parameters, achieving a 2.86x speed-up in the inference task without noticeable degradation of the segmentation accuracy.

The use of hyperspectral imaging (HSI) in autonomous driving (AD), while promising, faces many challenges related to the specifics and requirements of this application domain. On the one hand, non-controlled and variable lighting conditions, the wide depth-of-field ranges, and dynamic scenes with fast-moving objects. On the other hand, the requirements for real-time operation and the limited computational resources of embedded platforms. The combination of these factors determines both the criteria for selecting appropriate HSI technologies and the development of custom vision algorithms that leverage the spectral and spatial information obtained from the sensors. In this article, we analyse several techniques explored in the research of HSI-based vision systems with application to AD, using as an example results obtained from experiments using data from the most recent version of the HSI-Drive dataset.

Integrating hyperspectral imagery (HSI) with deep neural networks (DNNs) can strengthen the accuracy of intelligent vision systems by combining spectral and spatial information, which is useful for tasks like semantic segmentation in autonomous driving. To advance research in such safety-critical systems, determining the precise contribution of spectral information to complex DNNs' output is needed. To address this, several saliency methods, such as class activation maps (CAM), have been proposed primarily for image classification. However, recent studies have raised concerns regarding their reliability. In this paper, we address their limitations and propose an alternative approach by leveraging the data provided by activations and weights from relevant DNN layers to better capture the relationship between input features and predictions. The study aims to assess the superior performance of HSI compared to 3-channel and single-channel DNNs. We also address the influence of spectral signature normalization for enhancing DNN robustness in real-world driving conditions.

Machine learning-based embedded systems employed in safety-critical applications such as aerospace and autonomous driving need to be robust against perturbations produced by soft errors. Soft errors are an increasing concern in modern digital processors since smaller transistor geometries and lower voltages give electronic devices a higher sensitivity to background radiation. The resilience of deep neural network (DNN) models to perturbations in their parameters is determined, to a large extent, by the structure of the model itself, and also by the selected numerical representation and used arithmetic precision. When compression techniques such as model pruning and model quantization are applied to reduce memory footprint and computational complexity for deployment, both model structure and numerical representation are modified and thus, soft error robustness also changes. In this sense, although the choice of activation functions (AFs) in DNN models is frequently ignored, it conditions not only their accuracy and trainability, but also compressibility rates and numerical robustness. This paper investigates the suitability of using bounded AFs to improve model robustness against DNN parameter perturbations, assessing at the same time the impact of this choice on deployment in terms of model accuracy, compressibility, and computational burden. In particular, we analyze encoder-decoder fully convolutional models aimed at performing semantic segmentation tasks on hyperspectral images for scene understanding in autonomous driving. Deployment characterization is performed experimentally on an AMD-Xilinx’s KV260 SoM.

As the deployment of artificial intelligence (AI) algorithms at edge devices becomes increasingly prevalent, enhancing the robustness and reliability of autonomous AI-based perception and decision systems is becoming as relevant as precision and performance, especially in applications areas considered safety-critical such as autonomous driving and aerospace. This paper delves into the robustness assessment in embedded Deep Neural Networks (DNNs), particularly focusing on the impact of parameter perturbations produced by single event upsets (SEUs) on convolutional neural networks (CNN) for image semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit sensitivity of various encoder-decoder models to soft errors, this study thoroughly investigates the vulnerability of segmentation DNNs to SEUs and evaluates the consequences of techniques like model pruning and parameter quantization on the robustness of compressed models aimed at embedded implementations. The findings offer valuable insights into the mechanisms underlying SEU-induced failures that allow for evaluating the robustness of DNNs once trained in advance. Moreover, based on the collected data, we propose a set of practical lightweight error mitigation techniques with no memory or computational cost suitable for resource-constrained deployments. The code used to perform the fault injection (FI) campaign is available at https://github.com/jonGuti13/TensorFI2, while the code to implement proposed techniques is available at https://github.com/jonGuti13/parameterProtection.

The article discusses the use of low cost System-On-Module (SOM) platforms for the implementation of efficient hyperspectral imaging (HSI) processors for application in autonomous driving. The work addresses the challenges of shaping and deploying multiple layer fully convolutional networks (FCN) for low-latency, on-board image semantic segmentation using resource- and power-constrained processing devices. The paper describes in detail the steps followed to redesign and customize a successfully trained HSI segmentation lightweight FCN that was previously tested on a high-end heterogeneous multiprocessing system-on-chip (MPSoC) to accommodate it to the constraints imposed by a low-cost SOM. This SOM features a lower-end but much cheaper MPSoC suitable for the deployment of automatic driving systems (ADS). In particular the article reports the data- and hardware-specific quantization techniques utilized to fit the FCN into a commercial fixed-point programmable AI coprocessor IP, and proposes a full customized post-training quantization scheme to reduce computation and storage costs without compromising segmentation accuracy.

We present the updated version of the HSI-Drive dataset aimed at developing automated driving systems (ADS) using hyperspectral imaging (HSI). The v2.0 version includes new annotated images from videos recorded during winter and fall in real driving scenarios. Added to the spring and summer images included in the previous v1.1 version, the new dataset contains 752 images covering the four seasons. In this paper, we show the improvements achieved over previously published results obtained on the v1.1 dataset, showcasing the enhanced performance of models trained on the new v2.0 dataset. We also show the progress made in comprehensive scene understanding by experimenting with more capable image segmentation models. These models include new segmentation categories aimed at the identification of essential road safety objects such as the presence of vehicles and road signs, as well as highly vulnerable groups like pedestrians and cyclists. In addition, we provide evidence of the performance and robustness of the models when applied to segmenting HSI video sequences captured in various environments and conditions. Finally, for a correct assessment of the results described in this work, the constraints imposed by the processing platforms that can sensibly be deployed in vehicles for ADS must be taken into account. Thus, and although implementation details are out of the scope of this paper, we focus our research on the development of computationally efficient, lightweight ML models that can eventually operate at high throughput rates. The dataset and some examples of segmented videos are available in https://ipaccess.ehu.eus/HSI-Drive/.

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient color constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. In order to be realistic from an engineering viewpoint, this research is focused on the development of a feasible HSI segmentation system for ADAS, which implies considering implementation constraints and latency specifications throughout the algorithmic development process. For this reason, it is of particular importance to include the study of the raw image preprocessing stage into the data processing pipeline. Accordingly, this paper describes the development and deployment of a complete machine learning-based HSI segmentation system for ADAS, including the characterization of its performance on different embedded computing platforms, including a single board computer, an embedded GPU SoC and a programmable system on chip (PSoC) with embedded FPGA. We verify the superiority of the FPGA-PSoC over the GPU-SoC in terms of energy consumption and, particularly, processing latency, and demonstrate that it is feasible to achieve segmentation speeds within the range of ADAS industry specifications using standard development tools.

We propose a new hybrid quantum algorithm based on the classical Ant Colony Optimization algorithm to produce approximate solutions for NP-hard problems, in particular optimization problems. First, we discuss some previously proposed Quantum Ant Colony Optimization algorithms, and based on them, we develop an improved algorithm that can be truly implemented on near-term quantum computers. Our iterative algorithm codifies only the information about the pheromones and the exploration parameter in the quantum state, while subrogating the calculation of the numerical result to a classical computer. A new guided exploration strategy is used in order to take advantage of the quantum computation power and generate new possible solutions as a superposition of states. This approach is specially useful to solve constrained optimization problems, where we can implement efficiently the exploration of new paths without having to check the correspondence of a path to a solution before the measurement of the state. As an example of a NP-hard problem, we choose to solve the Quadratic Assignment Problem. The benchmarks made by simulating the noiseless quantum circuit and the experiments made on IBM quantum computers show the validity of the algorithm.

Advanced Driver Assistance Systems (ADAS) are designed with the main purpose of increasing the safety and comfort of vehicle occupants. Most of current computer vision-based ADAS perform detection and tracking tasks quite successfully under regular conditions, but are not completely reliable, particularly under adverse weather and changing lighting conditions, neither in complex situations with many overlapping objects. In this work we explore the use of hyperspectral imaging (HSI) in ADAS on the assumption that the distinct near infrared (NIR) spectral reflectances of different materials can help to better separate the objects in a driving scene. In particular, this paper describes some experimental results of the application of fully convolutional networks (FCN) to the image segmentation of HSI for ADAS applications. More specifically, our aim is to investigate to what extent the spatial features codified by convolutional filters can be helpful to improve the performance of HSI segmentation systems. With that aim, we use the HSI-Drive v1.1 dataset, which provides a set of labelled images recorded in real driving conditions with a small-size snapshot NIR-HSI camera. Finally, we analyze the implementability of such a HSI segmentation system by prototyping the developed FCN model together with the necessary hyperspectral cube preprocessing stage and characterizing its performance on an MPSoC.

Shuttling protocols designed by shortcut-to-adiabaticity techniques may suffer from perturbations and imperfect implementations. We study the motional excitation of a single ion shuttled in harmonic traps with time-dependent, 'systematic' oscillatory perturbations around the nominal parameters. These elementary perturbations could form any other by superposition. Robust shuttling strategies are proposed and compared, and optimizations are performed. This article is part of the theme issue 'Shortcuts to adiabaticity: theoretical, experimental and interdisciplinary perspectives'.

Driver Assistance Systems (DAS) have been progressively incorporated into commercial vehicles in recent years. All these systems are paving the way for the forthcoming autonomous vehicle which will become a reality in the near future. Existing systems are based on numerous electronic systems with advanced skills, high performances, and high degrees of adaptability and intelligence. As is to be expected, these cutting-edge features require, in most cases, the use of powerful computing platforms. However, the deployment of such platforms is not an easy task, since they have to be integrated in the vehicle where there exist important restrictions regarding size, power consumption and cost. In this sense, every smart proposal aimed at reducing the complexity of these systems without degrading performance, is always a valuable contribution in the field. In this work, we propose a methodology to reduce the dimensionality of a driver distraction recognition system. The methodology is based on a multi-objective genetic algorithm that looks for the minimum set of useful features collected during the driving task and also for the simplest recognition system. The recognition algorithm is an Extreme Learning Machine (ELM) whose simplicity and fast learning procedure make it especially suitable to be used by a Genetic Algorithm which needs to evaluate thousands of candidate solutions. The proposed methodology has been tested with a real-world database collected from different drivers performing an itinerary with an instrumented car. The results obtained validate the proposal as a method to reduce the complexity of a driver distraction recognition system.

We present a structured dataset for the research and development of automated driving systems (ADS) supported by hyperspectral imaging (HSI). The dataset contains per-pixel manually annotated images selected from videos recorded in real driving conditions that have been organized according to four environment parameters: season, daytime, road type, and weather conditions. The aim is to provide high data diversity and facilitate the automatic generation of data subsets for the evaluation of machine learning (ML) techniques applied to the research of ADS in different driving scenarios and environmental conditions. The video sequences have been captured with a small-size 25-band VNIR (Visible-NearlnfraRed) snapshot hyperspectral camera mounted on a driving automobile. The current selection of classes for image annotation is aimed to provide reliable data for the spectral analysis of the items in the scenes; it is thus based on material surface reflectance patterns (spectral signatures). It is foreseen that future versions of the dataset will also incorporate alternative dense semantic labeling of the annotated images. The first version of the dataset, named HSI-Drive v1.0, is publicly available for download 3 3 http://ipaccess.ehu.eus/HSI-Drive.

Quantum machine learning emerges from the symbiosis of quantum mechanics and machine learning. In particular, the latter gets displayed in quantum sciences as: (i) the use of classical machine learning as a tool applied to quantum physics problems, (ii) or the use of quantum resources such as superposition, entanglement, or quantum optimization protocols to enhance the performance of classification and regression tasks compare to their classical counterparts. This paper reviews examples in these two scenarios. On the one hand, a classical neural network is applied to design a new quantum sensing protocol. On the other hand, the design of a quantum neural network based on the dynamics of a quantum perceptron with the application of shortcuts to adiabaticity gives rise to a short operation time and robust performance. These examples demonstrate the mutual reinforcement of both neural networks and quantum physics.

The current health situation with the use of masks complicates the analysis of gaze and head direction in driver monitoring systems based on facial detection since landmarks are not working properly. Due to this issue, the need to solve occlusion problems using an alternative method to the current ones has increased. On the other hand, the deployment of these systems inside the vehicles must be carried out in the least intrusive way possible for the driver. This article presents an approach for driver distraction analysis based on the driver’s eyes without using landmarks applying Deep Learning methods, and the study of different parameters such as detection speed for the deployment of the best accuracy-speed method in an embedded platform. Different state-of-the-art and open source neural networks have been used and tuned to address our current problem. On the other hand, as is well known, training these models requires an enormous amount of data. In the case of gaze, there are very few data sets dedicated specifically to it. UnityEyes software has been used to create the training and test datasets for the system since it creates the necessary amount of data needed by the models easily.

Artificial neural networks (NNs) bridge input data into output results by approximately encoding the function that relates them. This is achieved after training the network with a collection of known inputs and results leading to an adjustment of the neuron connections and biases. In the context of quantum detection schemes, NNs find a natural playground. In particular, in the presence of a target (e.g. an electromagnetic field), a quantum sensor delivers a response, i.e. the input data, which can be subsequently processed by a NN that outputs the target features. In this work we demonstrate that adequately trained NNs enable to characterize a target with (i) minimal knowledge of the underlying physical model (ii) in regimes where the quantum sensor presents complex responses and (iii) under a significant shot noise due to a reduced number of measurements. We exemplify the method with a development for 171Yb+ atomic sensors. However, our protocol is general, thus applicable to arbitrary quantum detection scenarios.

Drivers provide a wide range of focus characteristics that can evaluate their attention level and analyze their behavioral states while driving. This information is critical for the development of new automated driving functionalities that support and assist the driver according to his/her state, ensuring safety for them and other users on the road. In this sense, this paper proposes a Driver Monitoring System (DMS) based on image processing and Convolutional Neural Networks (CNN), that analyzes two important driver distraction aspects: inattention of the road and drowsiness. Our approach makes use of CNN models for detecting the gaze and the head direction, which involves training datasets with different pre-defined labels. Additionally, the system is complemented with the drowsiness level measurement, using face features to detect the time that the eyes are closed or opened, and the blinking rate. Crossing the inference results of these models, the system can provide an accurate estimation of driver attention level. The different parts of the presented DMS have been trained in a Hardware-in-the-loop driving simulator with an eye fish camera. It has been tested as a real-time application recording driver with different characteristics.

Traffic Sign Recognition (TSR) is an important application that must be incorporated in autonomous vehicles. However, machine learning methods, used normally for TSR, demand high computational resources, which is in conflict with a system that is to be incorporated into a vehicle where size, cost, power consumption and real-time response are important requirements. In this paper, we propose a TSR system based on a Reduced Kernel Extreme Learning machine (RK-ELM) which is efficiently implemented in a Graphic Processing Unit (GPU). On the one hand, the inherent simplicity of ELM-based models makes possible the recognition process to be realized in a very fast and direct way. On the other hand, the computations involved in RK-ELM can be easily implemented in a GPUs so the recognition process is clearly boosted. Experiments carried out with a commonly used dataset benchmark validate our proposal.

Automotive ride comfort has become an important research topic in recent years due to the increasing level of automation in currently produced cars. These premises also apply to manned cars. In this work, a hybrid hardware/software extreme learning machine for improved ride comfort in automobiles is proposed. This system is based on a single-chip implementation able to provide real-time information about the level of ride comfort by classifying driving data into several comfort classes. To develop this system, unsupervised hierarchical clustering analysis (HCA) and supervised extreme learning machine (ELM) have been used jointly, to enhance the overall performance of the entire system, reaching classification success rates of up to 95%. This approach has been implemented on a Xilinx Zynq-7000 programmable system-on-chip. This chip is able to process data in real time and to identify the comfort class, achieving low latency marks and high operational frequencies due to its DSP-based implementation. These performance and accuracy marks, together with its low power consumption make this development suitable for novel practical implementations in current production cars.