Convolutional Neural Networks (CNNs) have become an essential tool in the domain of vision processing. However, dedicated accelerators are needed for energy-efficient execution of these networks, especially for embedded devices with tight energy constraints. Integrating multiple of these accelerators via chiplets promises a way to scale up the performance of these emerging systems by partitioning a neural network across multiple accelerators. This approach enables the execution of different layers on an accelerator with the best-suited dataflow. However, partitioning a neural network is a non-trivial task, especially when different accelerator architectures must be considered. In this paper, we propose our framework DSEParted, which automates the co-design of network partitioning and hardware architecture optimization. It employs a hierarchical optimization approach to gradually reduce the number of design candidates until an optimal system configuration is found for a partitioned computation of a neural network. We demonstrate that our framework can design a system that, for GoogLeNet, reduces latency by 22.5% when optimizing for latency. In addition, when optimizing for energy, it reduces the system area by 7.9%, with no impact on latency or energy compared to a baseline system. Further, we show that our partitioning-aware pruning strategy can reduce the EDP of the system by up to 49.7% in the case of ResNeXt-50, compared to a strategy that only optimizes the accelerators individually. Through the provided information, designers receive feedback on the efficiency of the full system at an early development stage. Our work is available open source(1).

Deep Neural Networks (DNNs) deployed in resource-constrained embedded systems, like automotive platforms, often face severe “memory wall” problems and energy-consuming data movements. To address this challenge, existing work has proposed Processing-In-Memory (PIM) architectures based on emerging non-volatile memory, which perform analog-domain computations inside memory, offering great potentials to solve the “memory wall” problem. However, PIM-based accelerators suffer from circuit noise, impacting DNN accuracy. Therefore, building a heterogeneous multi-chiplet architecture based on energy-efficient analog PIM and high-precision digital computing units has become a promising DNN acceleration solution that achieves both high energy efficiency and high accuracy. In this paper, we present an automated framework to explore layer-wise DNN inference partitioning in hybrid analog-digital multi-chiplet systems. After performing a topological ordering, it analyzes the robustness of each layer to constrain mapping decisions across available analog PIM chiplets and digital chiplets. By considering several functional and performance metrics, the framework identifies Pareto-optimal partitioning schemes. Exten-sive experimental results on a variety of DNN models show a reduction in latency of up to 52% with a loss in accuracy of less than 1%.

The ZuSE-KI-Mobil (ZuKIMo) research project presents a heterogeneous system-on-chip (SoC) designed for use in a variety of automotive and industrial edge applications. Implemented using GlobalFoundries (GF) 22-nm FD-SOI technology, the SoC features a modular architecture with a configurable, bit-serial, mixed-precision neural processing unit (NPU) core. This core can be adapted to different use cases, comes with a compact instruction set, and improves the performance of dilated convolutions. A hardware-accelerated, tunable image signal processor (ISP) hyperparameter pipeline reduces tuning time and increases detection confidence for AI tasks. The system also incorporates a selective, per-layer fault-tolerance mechanism and supports rapid prototyping via an Apache TVM-driven compiler flow and cycle-accurate simulation. The adaptable hardware generation process is designed with future chiplet-based scaling in mind, providing a flexible foundation for upcoming heterogeneous SoC designs.

In recent years, camera-based systems have increasingly been integrated into safety-critical systems. One key component for such a camera-based system is the image signal processor, which transforms raw sensor data captured in a Bayer pattern into RGB images. These images are then evaluated by Deep Neural Networks (DNNs), which are used for tasks such as image classification or object detection. However, image signal processors are also subjected to hardware faults. To achieve the required dependability level in safety-critical operation, it is necessary for the preprocessing pipeline in the image signal processor (ISP) to run without errors. Although there is much research on safety mechanisms for DNN accelerators, the fault tolerance of DNNs regarding faults originating from the ISP pipeline has not been thoroughly investigated. Therefore, we study how susceptible DNNs are to false detections due to hardware errors in the image signal processor and present a novel safety mechanism based on pixel histograms to prevent the evaluation of faulty image frames. With our novel safety mechanism, we achieve a diagnostic coverage of 96.8% while adding minimal hardware and operation overhead.

In high-performance data acquisition systems, realtime interpretation of captured signals is crucial due to limited offline storage and processing capabilities. Large scientific applications, such as particle detectors or massive radar testbeds, often capture inherently sparse data for which Point Cloud Networks (PCNs) have shown promising performance in online event processing tasks. In this work, we propose a novel approach for deploying point-based network layers on FPGAs in high-throughput, low-latency environments. We provide a programming model accompanied by an FPGA overlay which exploits a user-defined degree of spatial parallelism. By leveraging our dynamic dataflow architecture, we are able to guarantee time-variant real-time requirements for known point cloud sizes. In two case studies, we evaluate the inference latency of state-of-theart PCN layers. Based on the input point cloud size, our results demonstrate between 5.17x and 26.39x average latency improvements respectively, compared to our baseline implementations without spatial parallelism. Therefore, our proposed concept enables dynamic event interpretation for future scientific data acquisition systems.

Integrating neural network inference in systems with variable execution contexts proves difficult due to the difficulty in predicting runtime and energy consumption of embedded accelerators. Additionally, out of distribution inputs, adversarial attacks or latency requirements and available power budget can lead to the necessity of ending inference early. This comes at the cost of a lower confidence in the inference result. In case of out of distribution samples, classification diverging from the original class set might be desired. Incorporating early exits into existing neural networks for dynamically shortening the inference proves beneficial in situations where the original model structure or parameters cannot be altered, when training data is limited or to integrate neural network models into systems otherwise unsuitable due to high execution times. To allow the adaptation of existing models, we introduce ICE TEA, a tool implementing the insertion of custom early exits to existing neural networks to integrate them in a time- or energy constraint, or anomaly-prone context. We evaluate our framework using GoogleNet and ResNeXt-101 and were able to detect anomalies with an accuracy up to 89.1 %. Per introduced early exit, we achieve an accuracy of up to 68.3 % per exit, with an equivalence of 75.9 % to the original model.

The ZuSE-KI-Mobil (ZuKIMo) project, a nationally funded initiative, focuses on creating an advanced ecosystem optimized for AI-driven applications in automotive, drone, and industrial domains. At the heart of this effort is a state-of-the-art System-on-Chip (SoC), successfully taped out using 22 nm FDX technology, integrating a novel AI accelerator tailored to specific use case requirements, along with proof-of-concept demonstrators that validate the platform's real-world application potential. Key aspects include the customized compiler flow, the hardware generation process of the novel AI accelerator, and the acceleration of different applications using the ZuKIMo platform. Examples of these applications are 3D object detection and disengagement prediction in autonomous driving. The paper provides an overview of the ZuKIMo ecosystem, highlighting its contributions to AI performance, energy efficiency, and safety in heterogeneous AI hardware platforms.

The increasing computational complexity of AI workloads has led to the introduction of numerous accelerator architectures. However, these designs often neglect the software tooling necessary to generate optimized mappings. Additionally, future compute architectures are expected to become more heterogeneous, resulting in additional challenges regarding mapping of tasks to compute units. In this Ph.D. project, a novel methodology to generate compilers with an optimization pipeline for custom hardware accelerators is presented. The foundation of the work is a new Architecture Description Language (ADL) used to capture architectural details necessary for the compilation process. Based on this, simulators on both core and system level are generated that assists in the design of the underlying hardware. Finally, to assist with compilation, a flow to automatically register the new hardware design to a retargetable compiler will be designed. For the optimization pipeline, we focus on two specific aspects: Tensorization to map instruction sequences to the accelerator, and automated scheduling to optimize the execution time and energy.

Future automotive architectures are expected to transition from a network-centric to a domain-centered architecture featuring central compute units. Powerful domain controllers or smart sensors alleviate the load on these central units and communication systems. These controllers execute tasks with varying criticalities on heterogeneous multicore processors, and are ideally capable of dynamically balancing the computing load between the central unit and sensors. Here, Artificial Intelligence (AI) capabilities play a crucial role, as it is in high demand for such an automotive architecture. However, AI still requires specialized accelerators to improve their computation performance. Task-oriented distributed computing with criticalities up to ASIL-D necessitates the development and utilization of specialized methodologies, such as safety, through the isolation and abstraction of low-level hardware concepts. Meanwhile, online monitoring and diagnostics become vital features to detect errors during operation. The EMDRIVE architecture includes methods, components, and strategies to enhance the performance, safety, and security of such distributed computing platforms. The nationally funded EMDRIVE project connects its twelve partners from academia and industry and is currently in its intermediate stage.

In recent years, Deep Neural Networks (DNNs) have become common in the embedded domain for a wide variety of tasks. However, due to strict area and energy requirements, dedicated accelerators for these networks are necessary. Different architectures to fulfill this role are available, but an accelerator targeting the ultra-low area design point is still missing. In this work, we present an architecture focusing on minimizing resource utilization and energy requirements. We propose a small-scale vector processor that can be integrated into a RISC-V core. Unlike traditional, feature-rich vector instruction sets, we instead determine the most crucial instructions for DNN workloads through hardware-software co-design. To evaluate our implementation, we integrate our vector extension into an open-source RISC-V System-on-Chip, the NeoRV32, and measure the hardware overhead and achieved acceleration of operations commonly found in neural networks. Our results show an acceleration of up to 11.9x on convolutional workloads over the baseline architecture, while we demonstrate that our implementation introduces significantly less hardware overhead than previous works. We provide our work open-source to use for further exploration of low area design of VPUs 1 . 1 https://github.com/itiv-kit/rvve

Neural networks achieve high accuracy in tasks like image recognition or segmentation. However, their application in safety-critical domains is limited due to their black-box nature and vulnerability to specific types of attacks. To mitigate this, methods detecting out-of-distribution or adversarial attacks in parallel to the network inference were introduced. These methods are hard to compare because they were developed for different use cases, datasets, and networks. To fill this gap, we introduce EFFECT, an end-to-end framework to evaluate and compare new methods for anomaly detection, without the need for retraining and by using traces of intermediate inference results. The presented workflow works with every preexisting neural network architecture and evaluates the considered anomaly detection methods in terms of accuracy and computational complexity. We demonstrate EFFECT's capabilities, by creating new detectors for ShuffleNet and MobileNetV2 for anomaly detection as well as fault origin detection. EFFECT allows us to design an anomaly detector, based on the Mahalanobis distance as well as CNN based detectors. For both use cases, we achieve accuracies of over 85 %, classifying inferences as normal or abnormal, and thus beating existing methods.

In the era of electric vehicles, the reliability of power electronics has become a crucial part as the industry evolves. Changes in electrical parameters caused by aging and degradation can lead to performance deterioration and eventually the total failure (end-of-life) of power electronic components, whereas the lifetime of power transistors depends to a large extent on the temperature fluctuations. Therefore, it is desired to extend the lifetime by minimizing the temperature swings without affecting the vehicle dynamics. In this paper, we propose the LETSCOPE (Lifecycle Extensions through Software-Defined Predictive Control of Power Electronics) framework for the lifetime extension with a neural-network-based intelligent management system. The driving behavior in the near future is estimated based on information fusion, which enables the predictive optimization of the temperature fluctuations. Besides, to perform real-time on-board management, an integrated SoC with hardware accelerator is developed for an efficient execution under resource constraints. Proof-of-concept experiments on a benchmark route show that the LETSCOPE framework can improve the expected lifetime of power electronics by up to 87% without vehicle dynamics degradation. Moreover, the monitoring of critical parameters provides an early warning of total failure, which contributes to the overall system reliability.

Applications such as autonomous driving or assistive robotics heavily rely on the usage of Deep Neural Networks. In particular, Convolutional Neural Networks (CNNs) provide precise and reliable results in image processing tasks like camera-based object detection or semantic segmentation.However, to achieve even better results, CNNs are becoming more and more complex. Deploying these networks in distributed embedded systems thereby imposes new challenges, due to additional constraints regarding performance and energy consumption in the near-sensor compute platforms, i.e. the sensor nodes. Processing all data in the central node, however, is disadvantageous since raw data of camera consumes large bandwidth and running CNN inference of multiple tasks requires certain performance. Moreover, sending raw data over the interconnect is not advisable for privacy reasons. Hence, offloading CNN workload to the sensor nodes in the system can lead to reduced traffic on the link and a higher level of data security. However, due to the limited hardware-resources on the sensor nodes, partitioning CNNs has to be done carefully to meet overall latency requirements and energy constraints. Therefore, we present CNNParted, an open-source framework for efficient, hardware-aware CNN inference partitioning targeting embedded AI applications. It automatically searches for potential partitioning points in the CNN to find a beneficial workload distribution between sensor nodes and a central edge node. Thereby, CNNParted not only analyzes the CNN architecture but also takes hardware components, such as dedicated hardware accelerators and memories, into consideration to evaluate inference partitioning regarding latency and energy consumption.Exemplary, we apply CNNParted to three commonly used feed forward CNNs in embedded systems. Thereby, the framework first searches for several potential partitioning points and then evaluates the latter regarding inference latency and energy consumption. Based on the results, beneficial partitioning points can be identified depending on the system constraints. Using the framework, we are able to find and evaluate 10 potential partitioning points for FCN ResNet-50, 13 partitioning points for GoogLeNet, and 8 partitioning points for SqueezeNet V1.1 within 520 s, 330 s, and 140 s, respectively, on an AMD EPYC 7702P running 8 concurrent threads. For GoogLeNet, we determine two partitioning points that provide a good trade-off between required bandwidth, latency and energy consumption. We also provide insights into further interesting findings that can be derived from the evaluation results.

Applications in safety-critical areas like automotive or aerospace require functions to handle system failures. A commonly used technique on different system levels is checkpointing, which enables a rollback in such an event. However, at register level, power failures cause CMOS flip-flops to lose data, requiring a complete reboot instead of a fast rollback.In this paper, we propose the use of hybrid volatile/non-volatile flip-flops to prevent data loss on power shortages. To reduce the area overhead introduced by those, we present our automated toolchain based on open-source tools to find potential state-holding flip-flops in a netlist and implement those as hybrid volatile/non-volatile for checkpointing. This approach not only reduces area overhead and improves resiliency, but also enables aggressive power-gating strategies to increase energy efficiency.Exemplary, we test our toolchain for three RISC-V based processors and verify our results in simulation. Thereby, we can show that our proposed approach is able to determine all state-holding flip-flops in each architecture.

Since their breakthrough, complexity of Deep Neural Networks (DNNs) is rising steadily. As a result, accelerators for DNNs are now used in many domains. However, designing and configuring an accelerator that meets the requirements of a given application perfectly is a challenging task. In this paper, we therefore present our approach to support the accelerator design process. With an analytical model of a systolic array we can estimate performance, energy consumption and area for each design option. To determine these metrics, usually a cycle accurate simulation is performed, which is a time-consuming task. Hence, the design space has to be restricted heavily. Analytical modelling, however, allows for fast evaluation of a design using a mathematical abstraction of the accelerator. For DNNs, this works especially well since the dataflow and memory accesses have high regularity. To show the correctness of our model, we perform an exemplary realization with the state-of-the-art systolic array generator Gemmini and compare it with a cycle accurate simulation and state-of-the-art modelling tools, showing less than 1% deviation. We also conducted a design space exploration, showing the analytical model's capabilities to support an accelerator design. In a case study on ResNet-34, we can demonstrate that our model and DSE tool reduces the time to find the best-fitting solution by four or two orders of magnitude compared to a cycle-accurate simulation or state-of-the-art modelling tools, respectively.

With deep neural networks (DNNs) gaining popularity for tasks like object detection and image segmentation in domains like autonomous driving and smart agriculture, DNN inference on edge devices is increasingly necessary. While inference on edge devices can provide real time capabilities, these devices might be time and energy constraint. Offloading execution to the cloud can provide a possibility to save energy and execution time on the edge device, when a stable network connection is present. Contrary to that, failing to communicate with cloud servers during operation is detrimental to offloaded inference.In this work, we introduce a scheduling algorithm capable of leveraging the strength of cloud offloading in reducing execution time and power consumption on the edge device, while simultaneously being able to recover in the event of a network failure and continue inference seamlessly. Only raising the worst case execution time by the length of one branching interval, our method provides the ability to greatly reduce energy consumption and execution time. For FCN-ResNet50 inference on an edge accelerator, we were able to lower the overall execution time from 142.25 ms to 12.23 ms and the energy consumption on the edge device from 105.02 mJ to 5.46 mJ, while recovering from connectivity failures twice with continued inference. Comparable improvements can be achieved for other use cases using GoogleNet or SqueezeNet.