The need for deterministic timing and high-accuracy synchronization in the distributed deployment of ultralow-latency sensors, such as SPAD arrays, is challenging the technical boundaries of current communication systems. In this work, we describe the first steps toward the design of a System-on-Chip (SoC) that synchronously captures events from monophotonic sensors and communicates with other networked nodes using low-latency deterministic streaming over high-speed Ethernet. With this aim, we have developed a «custom EtherBone master core» capable of assembling a functional EtherBone frame that enables, through its exchange, remote Wishbone operations between external modules inside a White Rabbit (WR) network. Compared to the standard version, the new core design minimizes both latency and silicon area by eliminating all features of the default EtherBone module that are not necessary in our application. Experimental results obtained in a real, point-topoint, synchronized network setup based on the Cute-WR-A7 hardware verify the core's correct operation.

Machine learning-based embedded systems for safety-critical applications, such as aerospace and autonomous driving, must be robust to perturbations caused by soft errors. As transistor geometries shrink and voltages decrease, modern electronic devices become more susceptible to background radiation, increasing the concern about failures produced by soft errors. The resilience of deep neural networks (DNNs) to these errors depends not only on target device technology but also on model structure and the numerical representation and arithmetic precision of their parameters. Compression techniques like pruning and quantization, used to reduce memory footprint and computational complexity, alter both model structure and representation, affecting soft error robustness. In this regard, although often overlooked, the choice of activation functions (AFs) impacts not only accuracy and trainability but also compressibility and error resilience. This paper explores the use of bounded AFs to enhance robustness against parameter perturbations, while evaluating their effects on model accuracy, compressibility, and computational load with a technology-agnostic approach. We focus on encoder-decoder convolutional models developed for semantic segmentation of hyperspectral images with application to autonomous driving systems. Experiments are conducted on an AMD-Xilinx's KV260 SoM.

The use of hyperspectral imaging (HSI) for autonomous navigation is a promising field of research that aims to improve the accuracy and robustness of detection, tracking, and scene understanding systems based on vision sensors. The combination of advanced computer algorithms, such as deep neural networks (DNNs), and small-size snapshot HSI cameras allows to strengthen the reliability of those vision systems. Using HSI, some intrinsic limitations of greyscale and RGB imaging in depicting physical properties of targets related to the spectral reflectance of materials (metamerism) are overcome. Despite the promising results of many published HSI-based computer vision developments, the strict requirements of safety-critical applications such as autonomous driving systems (ADS) regarding latency, resource consumption, and security are prompting the migration of machine learning (ML)-based solutions to edge platforms. This involves a thorough software/hardware co-design scheme to distribute and optimize the tasks efficiently among the limited resources of computing platforms. With respect to inference, the over-parameterized nature of DNNs poses significant computational challenges for real-time on-the-edge deployment. In addition, the intensive data preprocessing required by HSI, which is frequently overlooked, must be carefully managed in terms of memory arrangement and inter-task communication to enable an efficient integrated pipeline design on a system on chip (SoC). This work presents a set of optimization techniques for the practical co-design of a DNN-based HSI segmentation processor deployed on a field programmable gate array (FPGA)-based SoC targeted at ADS, including key optimizations such as functional software/hardware task distribution, hardware-aware preprocessing, ML model compression, and a complete pipelined deployment. Applied compression techniques significantly reduce the complexity of the designed DNN to 24.34% of the original operations and to 1.02% of the original number of parameters, achieving a 2.86x speed-up in the inference task without noticeable degradation of the segmentation accuracy.

The use of hyperspectral imaging (HSI) in autonomous driving (AD), while promising, faces many challenges related to the specifics and requirements of this application domain. On the one hand, non-controlled and variable lighting conditions, the wide depth-of-field ranges, and dynamic scenes with fast-moving objects. On the other hand, the requirements for real-time operation and the limited computational resources of embedded platforms. The combination of these factors determines both the criteria for selecting appropriate HSI technologies and the development of custom vision algorithms that leverage the spectral and spatial information obtained from the sensors. In this article, we analyse several techniques explored in the research of HSI-based vision systems with application to AD, using as an example results obtained from experiments using data from the most recent version of the HSI-Drive dataset.

This paper explores the application of artificial intelligence on edge devices to enhance security in critical infrastructures, with a specific focus on the use case of a battery-powered mobile system for fire detection in tunnels. The study leverages the YOLOv5 convolutional neural network (CNN) for real-time detection, focusing on a comparative analysis across three low-power platforms, NXP i.MX93, Xilinx Kria KV260, and NVIDIA Jetson Orin Nano, evaluating their performance in terms of detection accuracy (mAP), inference time, and energy consumption. The paper also presents a methodology for implementing neural networks on various platforms, aiming to provide a scalable approach to edge artificial intelligence (AI) deployment. The findings offer valuable insights into the trade-offs between computational efficiency and power consumption, guiding the selection of edge computing solutions in security-critical applications.

Integrating hyperspectral imagery (HSI) with deep neural networks (DNNs) can strengthen the accuracy of intelligent vision systems by combining spectral and spatial information, which is useful for tasks like semantic segmentation in autonomous driving. To advance research in such safety-critical systems, determining the precise contribution of spectral information to complex DNNs' output is needed. To address this, several saliency methods, such as class activation maps (CAM), have been proposed primarily for image classification. However, recent studies have raised concerns regarding their reliability. In this paper, we address their limitations and propose an alternative approach by leveraging the data provided by activations and weights from relevant DNN layers to better capture the relationship between input features and predictions. The study aims to assess the superior performance of HSI compared to 3-channel and single-channel DNNs. We also address the influence of spectral signature normalization for enhancing DNN robustness in real-world driving conditions.

Machine learning-based embedded systems employed in safety-critical applications such as aerospace and autonomous driving need to be robust against perturbations produced by soft errors. Soft errors are an increasing concern in modern digital processors since smaller transistor geometries and lower voltages give electronic devices a higher sensitivity to background radiation. The resilience of deep neural network (DNN) models to perturbations in their parameters is determined, to a large extent, by the structure of the model itself, and also by the selected numerical representation and used arithmetic precision. When compression techniques such as model pruning and model quantization are applied to reduce memory footprint and computational complexity for deployment, both model structure and numerical representation are modified and thus, soft error robustness also changes. In this sense, although the choice of activation functions (AFs) in DNN models is frequently ignored, it conditions not only their accuracy and trainability, but also compressibility rates and numerical robustness. This paper investigates the suitability of using bounded AFs to improve model robustness against DNN parameter perturbations, assessing at the same time the impact of this choice on deployment in terms of model accuracy, compressibility, and computational burden. In particular, we analyze encoder-decoder fully convolutional models aimed at performing semantic segmentation tasks on hyperspectral images for scene understanding in autonomous driving. Deployment characterization is performed experimentally on an AMD-Xilinx’s KV260 SoM.

Performing AI inference ubiquitously requires energy-efficient, small footprint and highly reliable processing devices. Heterogeneous processing architectures combining customized CPUs with domain specific coprocessors can provide a good trade-off between computational efficiency and application flexibility for edge AI deployments while shortening development times compared to full custom application-specific processor designs. Following the impulse for the European sovereignty in the microelectronics field, in this work we propose the use of a RISCV based open-source hardware platform and Free/Libre and/or Open Source (FLOS) Electronic Design Automation (EDA) tools to evaluate the performance of different coprocessor integration options in a System-on-Chip (SoC) prototyped on FPGA. We tested four integration options (XBUS, Stream, CFS and CFU) to obtain precise data that will allow making the correct design decisions for the future development of integrated devices for high-performance AI at the edge.

This paper provides an overview of the main research activities carried out by Spanish organizations in areas related to neuromorphic technologies, spanning physical, materials, circuitry, and architectural levels. It also discusses the potential of these technologies to create competitive advantages for the Spanish industry, and to foster new applications and business opportunities via deep-tech startups - especially related to novel neuromorphic sensing modalities (e.g., Dynamic Vision Sensors).

As the deployment of artificial intelligence (AI) algorithms at edge devices becomes increasingly prevalent, enhancing the robustness and reliability of autonomous AI-based perception and decision systems is becoming as relevant as precision and performance, especially in applications areas considered safety-critical such as autonomous driving and aerospace. This paper delves into the robustness assessment in embedded Deep Neural Networks (DNNs), particularly focusing on the impact of parameter perturbations produced by single event upsets (SEUs) on convolutional neural networks (CNN) for image semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit sensitivity of various encoder-decoder models to soft errors, this study thoroughly investigates the vulnerability of segmentation DNNs to SEUs and evaluates the consequences of techniques like model pruning and parameter quantization on the robustness of compressed models aimed at embedded implementations. The findings offer valuable insights into the mechanisms underlying SEU-induced failures that allow for evaluating the robustness of DNNs once trained in advance. Moreover, based on the collected data, we propose a set of practical lightweight error mitigation techniques with no memory or computational cost suitable for resource-constrained deployments. The code used to perform the fault injection (FI) campaign is available at https://github.com/jonGuti13/TensorFI2, while the code to implement proposed techniques is available at https://github.com/jonGuti13/parameterProtection.

The article discusses the use of low cost System-On-Module (SOM) platforms for the implementation of efficient hyperspectral imaging (HSI) processors for application in autonomous driving. The work addresses the challenges of shaping and deploying multiple layer fully convolutional networks (FCN) for low-latency, on-board image semantic segmentation using resource- and power-constrained processing devices. The paper describes in detail the steps followed to redesign and customize a successfully trained HSI segmentation lightweight FCN that was previously tested on a high-end heterogeneous multiprocessing system-on-chip (MPSoC) to accommodate it to the constraints imposed by a low-cost SOM. This SOM features a lower-end but much cheaper MPSoC suitable for the deployment of automatic driving systems (ADS). In particular the article reports the data- and hardware-specific quantization techniques utilized to fit the FCN into a commercial fixed-point programmable AI coprocessor IP, and proposes a full customized post-training quantization scheme to reduce computation and storage costs without compromising segmentation accuracy.

We present the updated version of the HSI-Drive dataset aimed at developing automated driving systems (ADS) using hyperspectral imaging (HSI). The v2.0 version includes new annotated images from videos recorded during winter and fall in real driving scenarios. Added to the spring and summer images included in the previous v1.1 version, the new dataset contains 752 images covering the four seasons. In this paper, we show the improvements achieved over previously published results obtained on the v1.1 dataset, showcasing the enhanced performance of models trained on the new v2.0 dataset. We also show the progress made in comprehensive scene understanding by experimenting with more capable image segmentation models. These models include new segmentation categories aimed at the identification of essential road safety objects such as the presence of vehicles and road signs, as well as highly vulnerable groups like pedestrians and cyclists. In addition, we provide evidence of the performance and robustness of the models when applied to segmenting HSI video sequences captured in various environments and conditions. Finally, for a correct assessment of the results described in this work, the constraints imposed by the processing platforms that can sensibly be deployed in vehicles for ADS must be taken into account. Thus, and although implementation details are out of the scope of this paper, we focus our research on the development of computationally efficient, lightweight ML models that can eventually operate at high throughput rates. The dataset and some examples of segmented videos are available in https://ipaccess.ehu.eus/HSI-Drive/.

SiliconBurmuin is aimed at creating a multi-disciplinary neurocomputing community in the Basque Country, bringing together technology and scientific research centres and industry companies. This community will: (1) identify key biological structures and mechanisms that play a major role in vision across species, and (2) transform this knowledge into novel mathematical formalisms, neuromorphic designs and algorithms to solve industry challenges and enable new experiments of interest in neuroscience and clinical research. To achieve the latter objective in a time-effective manner, SiliconBurmuin will draw strong connections with the ongoing Horizon Europe Nim-bleAI project, with which it shares coordination. This is expected to allow reinforcement of ideas, knowledge and technology via a common prototyping platform where to implement IP from both projects. In addition to describing the research objectives and direction of SiliconBurmuin, this paper posits that co-coordination and co-funding of aligned projects at EU and regional levels might well be a catalyst for raising regional self-awareness of own potential and develop it to help fulfill global challenges, such as semiconductor sovereignty.

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient color constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. In order to be realistic from an engineering viewpoint, this research is focused on the development of a feasible HSI segmentation system for ADAS, which implies considering implementation constraints and latency specifications throughout the algorithmic development process. For this reason, it is of particular importance to include the study of the raw image preprocessing stage into the data processing pipeline. Accordingly, this paper describes the development and deployment of a complete machine learning-based HSI segmentation system for ADAS, including the characterization of its performance on different embedded computing platforms, including a single board computer, an embedded GPU SoC and a programmable system on chip (PSoC) with embedded FPGA. We verify the superiority of the FPGA-PSoC over the GPU-SoC in terms of energy consumption and, particularly, processing latency, and demonstrate that it is feasible to achieve segmentation speeds within the range of ADAS industry specifications using standard development tools.

Advanced Driver Assistance Systems (ADAS) are designed with the main purpose of increasing the safety and comfort of vehicle occupants. Most of current computer vision-based ADAS perform detection and tracking tasks quite successfully under regular conditions, but are not completely reliable, particularly under adverse weather and changing lighting conditions, neither in complex situations with many overlapping objects. In this work we explore the use of hyperspectral imaging (HSI) in ADAS on the assumption that the distinct near infrared (NIR) spectral reflectances of different materials can help to better separate the objects in a driving scene. In particular, this paper describes some experimental results of the application of fully convolutional networks (FCN) to the image segmentation of HSI for ADAS applications. More specifically, our aim is to investigate to what extent the spatial features codified by convolutional filters can be helpful to improve the performance of HSI segmentation systems. With that aim, we use the HSI-Drive v1.1 dataset, which provides a set of labelled images recorded in real driving conditions with a small-size snapshot NIR-HSI camera. Finally, we analyze the implementability of such a HSI segmentation system by prototyping the developed FCN model together with the necessary hyperspectral cube preprocessing stage and characterizing its performance on an MPSoC.

Driver Assistance Systems (DAS) have been progressively incorporated into commercial vehicles in recent years. All these systems are paving the way for the forthcoming autonomous vehicle which will become a reality in the near future. Existing systems are based on numerous electronic systems with advanced skills, high performances, and high degrees of adaptability and intelligence. As is to be expected, these cutting-edge features require, in most cases, the use of powerful computing platforms. However, the deployment of such platforms is not an easy task, since they have to be integrated in the vehicle where there exist important restrictions regarding size, power consumption and cost. In this sense, every smart proposal aimed at reducing the complexity of these systems without degrading performance, is always a valuable contribution in the field. In this work, we propose a methodology to reduce the dimensionality of a driver distraction recognition system. The methodology is based on a multi-objective genetic algorithm that looks for the minimum set of useful features collected during the driving task and also for the simplest recognition system. The recognition algorithm is an Extreme Learning Machine (ELM) whose simplicity and fast learning procedure make it especially suitable to be used by a Genetic Algorithm which needs to evaluate thousands of candidate solutions. The proposed methodology has been tested with a real-world database collected from different drivers performing an itinerary with an instrumented car. The results obtained validate the proposal as a method to reduce the complexity of a driver distraction recognition system.

We present a structured dataset for the research and development of automated driving systems (ADS) supported by hyperspectral imaging (HSI). The dataset contains per-pixel manually annotated images selected from videos recorded in real driving conditions that have been organized according to four environment parameters: season, daytime, road type, and weather conditions. The aim is to provide high data diversity and facilitate the automatic generation of data subsets for the evaluation of machine learning (ML) techniques applied to the research of ADS in different driving scenarios and environmental conditions. The video sequences have been captured with a small-size 25-band VNIR (Visible-NearlnfraRed) snapshot hyperspectral camera mounted on a driving automobile. The current selection of classes for image annotation is aimed to provide reliable data for the spectral analysis of the items in the scenes; it is thus based on material surface reflectance patterns (spectral signatures). It is foreseen that future versions of the dataset will also incorporate alternative dense semantic labeling of the annotated images. The first version of the dataset, named HSI-Drive v1.0, is publicly available for download 3 3 http://ipaccess.ehu.eus/HSI-Drive.

The use of reduced-precision formats is a valuable strategy to improve performance and reduce costs in embedded computing. In case of embedded model predictive control (MPC), utilizing reduced-precision numbers to speed-up underlying optimization algorithms can help to extend the application scope of MPC. In this paper we show how the improved spectral properties of linear systems inside interior point-proximal method of multipliers (IP-PMM) combined with the application of online regularization and instability correction mechanisms, can prevent embedded MPC controllers from failure when reduced-precision arithmetic units are used. Thus, the proposed approach can also contribute to designing efficient domain-specific processors for embedded MPC using custom floating-point formats. To our knowledge this is the first time an IP-PMM algorithm is applied to solve quadratic programming (QP) problems in MPC.

Designing embedded model predictive controllers (MPCs) for systems with fast sampling rates is a tedious and multidisciplinary process. The reason is that from the high-level description of the control problem to the final implementation on a computational platform, the MPC design space covers multiple abstraction levels in which critical design decisions have to be made. Therefore, currently, the identification of the best design criteria can not be automated. Instead, in this paper we take an step towards the development of a computer-aided design (CAD) tool that enables the semi-automatic implementation of application-specific embedded MPCs on field-programmable gate arrays (FPGAs) for its application to control-engineering problems with fast sampling rates.

This paper presents a system-level co-simulation and co-verification workflow to ease the transition from a software-only procedure, executed in a General Purpose processor, to the integration of a custom hardware accelerator developed in a Hardware Description Language (HDL). We propose a tool which enables Dynamic Binary Modification to decouple the development of the hardware accelerator from the software-only application to be accelerated. It provides support for rapid iterative exploration and functional verification of hardware designs while keeping the unmodified software application as a reference. DBHI is able to instrument an application and inject compiled hardware. It allows progressive migration from application source code, to non-synthesizable HDL, and to synthesizable HDL. At the same time, it preserves cycle-accurate/bit-accurate results, and provides run-time visibility of the internal data buffers for debugging purposes. Foreign architecture emulation overhead during development is avoided, and early integration with peripherals in the target System-on-Chip is possible. The proposed design flow was evaluated on executions of hardware simulations on x86-64 and Arm. DBHI was developed from existing off-the-shelf tools, and we evaluated it on multiple architectures, however, the technique is not tied to any specific architecture.