Camera-based monitoring shows potential for continuous respiration rate measurement in clinical settings. The main cause of errors in camera-based respiration rate measurement using optical flow is non-breathing motion, which disturbs the measurement of the breathing motion. To prevent this, uninformative video frames containing non-breathing motion should be detected and removed from consideration for respiration rate extraction. Currently, signal quality metrics or global motion detection are used to detect non-breathing motion, but these methods have limitations when applied in clinical settings. Therefore, in this research, we propose a new informativeness metric for optical flow based respiration rate monitoring. This metric exploits the residual error of the optical flow fit for determining the presence of non-breathing motion on a per breath basis. This new metric is evaluated on a clinical dataset that contains RGB videos of 25 ICU patients. We find that a 3 breaths/min agreement of 98.7% and a mean absolute error of 0.21 breaths/min is achieved across all patients. These results show that using our new informativeness metric, we can achieve highly accurate camera-based respiration rate monitoring without relying on assumptions on signal quality or waveform morphology.

Processing weak analog signals from biomedical, environmental, and IoT sensors in a low-power event-driven manner is a major challenge, as conventional synchronous digitization wastes energy and fails to capture the fine temporal dynamics of slow and sparse natural signals. Neuromorphic computing has emerged as a promising paradigm for edge sensing; however, most state-of-the-art neuromorphic chips lack dedicated analog-to-spike interfaces, omit crucial signal conditioning such as gain control, and offer only limited flexibility, restricting their use in real-world sensing tasks. To overcome these limitations, we present AURA, a reconfigurable mixed-signal asynchronous spiking neural network (SNN) system for ultra-low-power sensing and computing. AURA integrates analog soma and synapse with on-chip analog front-end that directly encode sensory signals into spikes, reducing conversion overhead and enhancing temporal fidelity. The processor supports inter(intra)-chip communication based on four-phase handshake protocol and Address-Event Representation (AER). Online learning is enabled via reconfigurable connectivity and synaptic weight through external PC. Designed in IHP 130nm CMOS process, AURA achieves ∼ 0.4pJ per spike (operated within 500 Hz (incl. integ.)) in simulations.

Convolutional Spiking Neural Networks (SNN) offer the potential for highly energy-efficient vision processing by exploiting sparse, event-driven computation. However, existing SNN accelerators underutilize the inherent parallelism of convolutional layers and lack the flexibility to accommodate varying memory demands and input sparsity across layers. This paper presents Mega, a digital architecture for convolutional SNNs that addresses these limitations through three key contributions: (1) highly parallel acceleration of 3 × 3 convolutions, (2) a unified data memory for spikes, neuron states, and weights, and (3) efficient spike map processing with low-overhead spike detection. Fabricated in GlobalFoundries 22 nm FDSOI technology, Mega achieves an energy efficiency of 0.375 pJ/SOP, improving the state of the art by 4×.

Heterogeneous architectures became a compelling choice for edge processors executing complex DNN workloads, as they provide an ideal blend of openness, customization, energy-efficient heterogeneity, and scalable performance. Compiler optimization for DNNs on heterogeneous System-on-Chip (SoC) architectures however, must navigate complex hardware-software co-design, data movement minimization, aggressive parallelism exploitation, and advanced static/dynamic code transformations to deliver high performance and energy efficiency.This paper presents a novel compiler ecosystem for highly heterogeneous SoCs with multiple back-end targets, spanning from typical CPUs, to programmable RISC-V clusters and up to dedicated and reconfigurable accelerators. It puts together static analysis, optimization, and scheduling infrastructure to overcome the limitations of current state-of-the-art tools for heterogeneous edge AI processors. Our compilation pipeline introduces several innovative features: (1) an automatic end-to-end flow for RISC-V-based platforms, (2) efficient data layout remapping (reducing memory footprint by 35% on average) and recognition of complex ternary reductions for auto-vectorization, (3) code layout adaptation for hardware simplification, (4) a novel MLIR-based RISC-V backend supporting optimized matrix-multiplication micro-kernels that reach 90% of peak performance, (5) periodic scheduling capabilities for layer-fused CNNs, and (6) automated mapping and scheduling onto heterogeneous CGRA templates for advanced parallel kernel execution, delivering 33% higher energy efficiency than the scalar implementation and up to 3.6× higher performance. These advances enable hardware-aware compilation that reduces manual optimization effort, lowers energy consumption through memory and computation optimization, and minimizes memory footprint and data transfers.

Bio-inspired sensors like Dynamic Vision Sensors (DVS) and silicon cochleas are often combined with Spiking Neural Networks (SNNs), enabling efficient, event-driven processing similar to biological sensory systems. To realize the low-power constraints of the edge, the SNN should run on a hardware architecture that can exploit the sparse nature of the spikes. In this paper, we introduce LOKI, a digital architecture for Fully-Connected (FC) SNNs. By using Multi-Cycle Clock-Gated (MCCG) SRAMs, LOKI can operate at 0.59 V, while running at a clock frequency of 667 MHz. At full throughput, LOKI only consumes 0.266 pJ/SOP. We evaluate LOKI on both the Neuromorphic MNIST (N-MNIST) and the Keyword Spotting k(KWS) tasks, achieving 98.0 % accuracy at 119.8 nJ/inference and 93.0 % accuracy at 546.5 nJ/inference respectively.

Spiking Neural Networks (SNNs) provide an efficient framework for processing dynamic spatio-temporal signals and for investigating the learning principles underlying biological neural systems. A key challenge in training SNNs is to solve both spatial and temporal credit assignment. The dominant approach for training SNNs is Backpropagation Through Time (BPTT) with surrogate gradients. However, BPTT is in stark contrast with the spatial and temporal locality observed in biological neural systems and leads to high computational and memory demands, limiting efficient training strategies and on-device learning. Although existing local learning rules achieve local temporal credit assignment by leveraging eligibility traces, they fail to address the spatial credit assignment without resorting to auxiliary layer-wise matrices, which increase memory overhead and hinder scalability, especially on embedded devices. In this work, we propose Traces Propagation (TP), a forward-only, memory-efficient, scalable, and fully local learning rule that combines eligibility traces with a layer-wise contrastive loss without requiring auxiliary layer-wise matrices. TP outperforms other fully local learning rules on NMNIST and SHD datasets. On more complex datasets such as DVS-GESTURE and DVS-CIFAR10, TP showcases competitive performance and scales effectively to deeper SNN architectures such as VGG-9, while providing favorable memory scaling compared to prior fully local scalable rules, for datasets with a significant number of classes. Finally, we show that TP is well suited for practical fine-tuning tasks, such as keyword spotting on the Google Speech Commands dataset, thus paving the way for efficient learning at the edge.

We provide new insights into the performance of camera-based heart and respiration rate extraction and evaluate its usability for replacing spot checks conducted in the general ward. A study was performed comprising of 36 ICU patients recorded for a total time of 699 h. The 5 beats/minute agreement between camera and ECG-based heart rate measurements was 81.5%, with a coverage of 81.9%, where the largest gap between measurements was 239 min. The challenges encountered in heart rate monitoring were limited visibility of the patient’s face and irregular heart rates, which led to poor agreement between camera- and ECG-based heart rate measurements. To prevent non-breathing motion from causing error in respiration rate extraction, we developed a metric which was used to detect non-breathing motion. The 3 breaths/minute agreement between the camera- and contact-based respiration rate measurements was 91.1%, with a coverage of 59.1%, where the largest gap between measurements was 114 min. Encountered challenges were the morphology of the respiration signal and irregular breathing. While a few challenges need to be overcome, the results show promise for the usability of camera-based heart and respiration rate monitoring as a replacement for spot checks of these vital parameters conducted in the general ward.

This work evaluates a forward-only learning algorithm on the MNIST dataset with hardware-in-the-loop training of a 4f optical correlator, achieving 87.6% accuracy with O(n2) complexity, compared to backpropagation, which achieves 88.8% accuracy with O(n2 log n) complexity.

Digital Twins (DTs) are virtual representations of physical systems or Physical Twins (PTs) that are used for various data-driven applications in manufacturing industry such as predictive maintenance, diagnostics and condition monitoring. The data in DTs is collected through virtual sensors, which augment the physical sensors by providing additional data that cannot be directly observed. In high-throughput production systems like semiconductor manufacturing equipment, highspeed and high-precision motion stages control the equipment's movement. However, the use of DTs in these systems is limited due to their real-time requirements. To enable real-time applications, the DT must be synchronized with its physical counterpart to ensure timely data from virtual sensors. The synchronization mechanism should be non-blocking to prevent any impact on the throughput of physical systems. In this paper, we propose a synchronization mechanism for DTs in high-speed high-precision motion control systems. The mechanism involves sharing PT states with DT over a network and compensating for network delays. The synchronization mechanism is validated in a framework comprising an industrial motion stage system and its digital twin. The validation is done for different synchronization delay scenarios, demonstrating its effectiveness of proposed approach in eliminating synchronization delays without blocking the PT operation. The proposed mechanism enables real-time virtual sensing ensuring data timeliness with high accuracy.

Many autonomous systems are increasingly adopting Neural Networks (NNs) based perception in vision-in-the-loop (VIL) control systems. In many industrial applications, the features (shape, size and texture) of the object of interest varies, which imposes robustness requirements on the perception algorithm. Further, performance of the VIL system imposes strict latency requirements. Using NNs in VIL system poses two challenges. First, the NN models should be lightweight resulting in a low closed-loop latency. Second, availability of representative training data for ensuring robustness of the lightweight NN models. Collecting such training data is expensive and often, infeasible in many industrial systems. In this work we propose an approach for training the NNs used for VIL applications using digital twins (DT). The DT is used for automatically generating and labelling training data representing various features like object shapes and directional lighting. Starting from a lightweight NN base model, our proposed approach fine-tunes or retrains the model using DT-generated training data achieving desired performance and robustness on a different target operating condition. The approach is validated considering a VIL semiconductor motion stage system with square and rectangular dies of dimension of (0.5cm × 0.5cm) and (0.5cm × +1cm) respectively. The VIL system limits the positioning error in the range of 2% compared to 12% positioning error with no vision feedback.

Spiking Neural Networks (SNNs) are event-driven bio-inspired neural networks. Recent research has trained SNN models with accuracy on par with Artificial Neural Networks (ANNs) on computer vision tasks. Due to their sparse, event-based computation, SNNs are particularly promising for energy-efficient processing, especially in event-based vision applications. However, neurons have internal states which evolve over time and keeping track of them can be costly. Hence, efficiently deploying them, especially on memory-constrained edge devices, requires careful mapping of their computation across both spatial and temporal dimensions. To address this issue, we introduce STEMS, Spatial-Temporal Mapping for SNNs. STEMS supports inter-layer mapping exploration, as well as loop tiling optimizations. By applying STEMS inter-layer exploration, we show up to 12x reduction in external memory traffic and up-to 5x reduction in energy consumption. Finally, we show that neuron states may not be needed in early SNN layers. By optimizing neuron states in one of our benchmarks, we reduced neuron states by 20x and improved energy performance by 1.4x saving without sacrificing accuracy.

Stereo-based depth estimation is becoming more and more important in many applications like self-driving vehicles, earth observation, cartography, robotics and so on. Modern approaches to depth estimation employ artificial intelligence techniques, particularly convolutional neural networks (CNNs). However, stereo-based depth estimation networks involve dual processing paths for left and right input images, which merge at intermediate layers, posing challenges for efficient deployment on modern hardware accelerators. Specifically, modern depth-first and layer-fused execution strategies, which are commonly used to reduce I/O communication and on-chip memory demands, are not readily compatible with such non-linear network structures. To address this limitation, we propose a flexible multi-core hardware architecture tailored for stereo-based depth estimation CNNs. The architecture supports layer-fused execution while efficiently managing dual-path computation and its fusion, enabling improved resource utilization. Experimental results demonstrate a latency reduction of up to 24% compared to state-of-the-art depth-first implementations that do not incorporate stereo-specific optimizations.

The high-speed, high-precision positioning of objects is a critical component in various industrial manufacturing processes. The semiconductor die packaging, for instance, requires the precise pickup and placement of semiconductor dies on substrates. This is done by coupling the silicon wafer which contains thousands of semiconductor dies, with a motion control platform equipped with linear motor and encoder. The motion controller relies on linear motor and encoder to accurately position the silicon wafer at reference positions, which are determined through the relative positions of the dies on the wafer. However, the challenge arises when neighboring dies get misaligned during the pickup process, making it impossible to read the position of the die through encoder. This paper addresses the challenge of precise alignment in high-speed, micro-scale manufacturing environments, where traditional methods struggle due to the disconnect between the point-of-interest (dies) and point-of-control (motor/silicon wafer). To overcome these challenges, we propose a Deep Neural Network (DNN) based perception that allows for robust sensing of die positions. We also propose a fusion mechanism to incorporate this vision feedback with the encoder to accurately detect the misalignment and compensate for it before periodic pickups of the dies. We use a software-in-the-loop validation framework to demonstrate that our proposed method could successfully eliminate the misalignment before the pickup in the range under consideration.

Kernel fusion is a crucial optimization technique for GPU applications, particularly deep neural networks, where it involves combining multiple consecutive kernels into a single larger kernel. This approach aims to enhance performance by reducing the need for slow off-chip memory accesses. Instead, intermediate results between successive kernels are stored in faster on-chip memory like shared memory. This strategy has the potential to not only boost performance, but also reduce energy consumption. Typically, GPU kernels fall into two categories: tensor operations and element operations. In deep learning, fusing a tensor operation kernel with an element operation kernel that follows it, such as combining convolution with ReLU, is a common practice to achieve improved performance. While combining two tensor kernels in a single GPU kernel has shown benefits in certain applications, it is not a straightforward task. The advantages and limitations of this approach remain unclear, prompting several questions: 1) What advantages does tensor kernel fusion offer on GPGPUs? 2) What limitations does it have and why is it not widely adopted? 3) In what practical scenarios is tensor kernel fusion beneficial? To address these questions, we conducted both analytical and experimental studies on Nvidia Tensor Core GPUs, using the CUTLASS kernel library with extensions. Our experimental findings revealed that for tall and narrow matrix multiplications, employing a 1D tiling strategy outperforms the commonly used 2D tiling strategy. By comparing tensor kernel fusions with a 1D tiling baseline, we demonstrated significant performance gains for tall and narrow matrix multiplications with fusion. However, we observe that these benefits diminish as the matrix sizes increase in width.

Bayesian inference is a probabilistic approach to the problem of drawing conclusions from observed data. Its main challenge is computational, which the Bayesian community tends to address through approximation techniques. However, these techniques come with their own set of challenges, including approximation errors, the difficulty of assessing these errors, and the inherited NP-hardness of the inference problem. Concurrently, in an effort to keep up with Moore’s law, the computer engineering community has developed an increasing number of programming techniques for today’s heterogeneous hardware. These techniques aim to optimize the execution schedule, which refers to the order and mapping of computations on the available execution units. In this work, we advocate for the utilization of these techniques to avoid a common pitfall known as premature approximation. Notably, these techniques have the potential to significantly enhance performance, thereby reducing the need for approximation and thus mitigating the challenges that accompany it. We first demonstrate how optimization of the storage strategy, i.e. when and where intermediate results are stored, allows for a trade-off between runtime and peak memory usage. We then investigate various techniques that aim to automatically generate efficient execution schedules. Finally, we focus on a specific, runtime-efficient execution schedule identified through design space exploration and compare its performance with that of two established solvers for probabilistic inference. The results show that our optimized implementation achieves speedups ranging from \(2\times\) to \(316\times\) for the UAI 2014 Promedus benchmark problems compared to the reference solvers. The ideas and methods presented in our study are examined within the framework of exact inference for discrete random variables. However, they are effectively applicable to scenarios involving continuous variables.

Probabilistic inference is a fundamental task in modern machine learning. Recent advances in tensor network (TN) contraction algorithms have enabled the development of better exact inference methods. However, many common inference tasks in probabilistic graphical models (PGMs) still lack corresponding TN-based adaptations. In this paper, we advance the connection between PGMs and TNs by formulating and implementing tensor-based solutions for the following inference tasks: (A) computing the partition function, (B) computing the marginal probability of sets of variables in the model, (C) determining the most likely assignment to a set of variables, (D) the same as (C) but after having marginalized a different set of variables, and (E) generating samples from a learned probability distribution using a generalized method. Our study is motivated by recent technical advances in the fields of quantum circuit simulation, quantum many-body physics, and statistical physics. Through an experimental evaluation, we demonstrate that the integration of these quantum technologies with a series of algorithms introduced in this study significantly improves the performance efficiency of existing methods for solving probabilistic inference tasks.

Within the realm of probabilistic graphical models, message-passing algorithms offer a powerful framework for efficient inference. When dealing with discrete variables, these algorithms essentially amount to the addition and multiplication of multidimensional arrays with labeled dimensions, known as factors. The complexity of these algorithms is dictated by the highest-dimensional factor appearing across all computations, a metric known as the induced tree width. Although state-of-the-art methods aimed at minimizing this metric have expanded the feasi-bility of exact inference, many real-world problems continue to be intractable. In this paper, we introduce a novel method for adding and multiplying factors that results in a substantial improvement in the inference performance, especially for increasingly complex models. Our approach aligns well with existing state-of-the-art methods designed to minimize the induced tree width, thereby further expanding the tractability spectrum of exact inference for more complex models. To demonstrate the efficacy of our method, we conduct a comparative evaluation against two other open-source libraries for probabilistic inference. Our approach exhibits an average speedup of 23 times for the UAI 2014 benchmark set. For the 10 most complex problems, the average speedup increases to 64 times, demonstrating its scalability.

Accurate object positioning is critical in many industrial manufacturing applications. The execution time and precision of the object positioning task have a significant impact on the overall performance and throughput, especially in cost-sensitive industries such as semiconductor manufacturing. In addition, the object positioning algorithm must adapt to changes in object size, features, and environmental conditions in real-time. While traditional sensors struggle to cope with dynamic conditions, vision-based perception is more adaptable and robust. Vision-based perception can capture and analyze visual information by using cameras and image processing algorithms, providing a robust way to locate objects in dynamic environments. However, classical perception algorithms based on vision cannot handle objects with different characteristics, and modern object detectors that rely on deep neural networks struggle to adapt to image sizes, resulting in unnecessary computations. To address these challenges, this paper proposes an approach for designing a branched multi-input deep neural network (DNN) that considers variations in input image sizes to adapt the input branches. In essence, the proposed DNN reduces the computation time for images with lower dimensions. To validate the proposed approach, an IC dataset is created that represents the variations in object sizes as seen in semiconductor manufacturing machines. Depending on the choice of input branches, the average inference time is reduced by over 30% with a slight gain in detection accuracy.

Memristor-based in-memory neuromorphic computing systems promise a highly efficient implementation of vector-matrix multiplications, commonly used in artificial neural networks (ANNs). However, the immature fabrication process of memristors and circuit level limitations, i.e., stuck-at-fault (SAF), IR-drop, and device-to-device (D2D) variation, degrade the reliability of these platforms and thus impede their wide deployment. In this paper, we present ReMeCo, a redundancy-based reliability improvement framework. It addresses the non-idealities while constraining the induced overhead. It achieves this by performing a sensitivity analysis on ANN. With the acquired insight, ReMeCo avoids the redundant calculation of least sensitive neurons and layers. ReMeCo uses a heuristic approach to find the balance between recovered accuracy and imposed overhead. ReMeCo further decreases hardware redundancy by exploiting the bit-slicing technique. In addition, the framework employs the ensemble averaging method at the output of every ANN layer to incorporate the redundant neurons. The efficacy of the ReMeCo is assessed using two well-known ANN models, i.e., LeNet, and AlexNet, running the MNIST and CIFAR10 datasets. Our results show 98.5% accuracy recovery with roughly 4% redundancy which is more than 20× lower than the state-of-the-art.