Most connected component labelling and analysis algorithms are based on some variant of Union-Find. In this paper, it is shown the Find operation is unnecessary for single-pass algorithms, leading to the Union-Retire approach where the focus is on connectivity rather than labelling. The computational complexity of the resulting algorithm is linear with the number of pixels within an image. It is shown that the resulting algorithm requires significantly less memory than existing algorithms, making it ideal for embedded image processing.

Recently automated frameworks have been proposed, mapping neural networks from a high-level description onto embedded devices, most of them in an end-to-end manner. This paper aims to give an overview of their main characteristics and achievements. A special focus is lying on internal predictions during design space exploration (DSE) regarding hardware targets (performance, area or power consumption), enabling fast exploration of the individually defined search spaces, especially in early design stages. Additionally, recent research results that are not part of such frameworks, but present novel estimation techniques are also covered by this work.

A key issue in system design is the lack of communication between hardware, software and domain expert. Recent research work shows progress in automatic HW/SW co-design flows of neural accelerators that seems to make this kind of communication obsolete. Most real-world systems, however, are a composition of multiple processing units, communication networks and memories. A HW/SW co-design process of (reconfigurable) neural accelerators, therefore, is an important sub-problem towards a common co-design methodology. The ultimate challenge is to define the constraints for the design space exploration on system level - a task which requires deep knowledge and understanding of hardware architectures, mapping of workloads onto hardware and the application domain, e.g. artificial intelligence. For most projects, these skills are distributed among several people or even different teams which is one of the major reasons why there is no established end-to-end development methodology for digital systems. This position paper discusses possibilities how to establish such a methodology for systems that include (reconfigurable) dedicated accelerators and outlines the central role that languages and tools play in the process.

Gigantic rates of data production in the era of Big Data, Internet of Thing (IoT), and Smart Cyber Physical Systems (CPS) pose incessantly escalating demands for massive data processing, storage, and transmission while continuously interacting with the physical world using edge sensors and actuators. For IoT systems, there is now a strong trend to move the intelligence from the cloud to the edge or the extreme edge (known as TinyML). Yet, this shift to edge AI systems requires to design powerful machine learning systems under very strict resource constraints. This poses a difficult design task that needs to take the complete system stack from machine learning algorithm, to model optimization and compression, to software implementation, to hardware platform and ML accelerator design into account. This paper discusses the open research challenges to achieve such a holistic Design Space Exploration for a HW/SW Co-design for Edge AI Systems and discusses the current state with three currently developed flows: one design flow for systems with tightly-coupled accelerator architectures based on RISC-V, one approach using loosely-coupled, application-specific accelerators as well as one framework that integrates software and hardware optimization techniques to built efficient Deep Neural Network (DNN) systems.

Advances in deep neural networks (DNNs) and the availability of massive real-world data have enabled superhuman levels of accuracy on many AI tasks and ushered the explosive growth of AI workloads across the spectrum of computing devices. However, their superior accuracy comes at a high computational cost, which necessitates approaches beyond traditional computing paradigms to improve their operational efficiency. Leveraging the application-level insight of error resilience, we demonstrate how approximate computing (AxC) can significantly boost the efficiency of AI platforms and play a pivotal role in the broader adoption of AI-based applications and services. To this end, we present RaPiD, a multi-tera operations per second (TOPS) AI hardware accelerator core (fabricated at 14-nm technology) that we built from the ground-up using AxC techniques across the stack including algorithms, architecture, programmability, and hardware. We highlight the workload-guided systematic explorations of AxC techniques for AI, including custom number representations, quantization/pruning methodologies, mixed-precision architecture design, instruction sets, and compiler technologies with quality programmability, employed in the RaPiD accelerator.

Single-pass connected components analysis (CCA) algorithms suffer from a time overhead to resolve labels at the end of each image row. This work demonstrates how this overhead can be eliminated by replacing the conventional raster scan by a zig-zag scan. This enables chains of labels to be correctly resolved while processing the next image row. The effect is faster processing in the worst case with no end of row overheads. CCA hardware architectures using the novel algorithm proposed in this paper are, therefore, able to process images at higher throughput than other state-of-the-art methods while reducing the hardware requirements. The latency introduced by the conversion from raster scan to zig-zag scan is compensated for by a new method of detecting object completion, which enables the feature vector for completed connected components to be output at the earliest possible opportunity.

In this paper, a memory-efficient architecture for single-pass connected components analysis suited for high-throughput embedded image processing systems is proposed which achieves a speedup by partitioning the image into slices. Although global data dependencies of image segments spanning several image slices exist, a temporal and spatial local algorithm is proposed, together with a suited FPGA hardware architecture processing pixel data at low latency. The low latency of the proposed architecture allows reuse of labels associated with the image objects. This reduces the amount of memory by a factor of more than 5 in the considered implementations which is a significant contribution since memory is a critical resource in embedded image processing on FPGAs. Therefore, a significantly higher bandwidth of pixel data can be processed with this architecture compared to the state-of-the-art architectures using the same amount of hardware resources.

End-to-end performance estimation and measurement of deep neural network (DNN) systems become more important with increasing complexity of DNN systems consisting of hardware and software components. The methodology proposed in this paper aims at a reduced turn-around time for evaluating different design choices of hardware and software components of DNN systems. This reduction is achieved by moving the performance estimation from the implementation phase to the concept phase by employing virtual hardware models instead of gathering measurement results from physical prototypes. Deep learning compilers introduce hardware-specific transformations and are, therefore, considered a part of the design flow of virtual system models to extract end-to-end performance estimations. To validate the run-time accuracy of the proposed methodology, a system processing the DilatedVGG DNN is realized both as virtual system model and as hardware implementation. The results show that up to 92% accuracy can be reached in predicting the processing time of the DNN inference.

Union-find algorithms form the basis of managing sets of equivalent labels within most connected components labelling algorithms. The new class of single-pass connected components analysis (CCA) algorithms (where a feature vector of each component is extracted during processing) are analysed and compared within this context. Such algorithms have been developed for stream processing, using customised hardware architectures. Many of these use an improved union-find algorithm requiring only a single lookup for its find operation. This paper analyses this optimisation and formally proves that the resulting single lookup connected components algorithm associates each pixel with its correct component when extracting the components’ feature vectors. Analysis of the algorithm led to a new double lookup algorithm that reduces the total number of memory accesses and is a step towards unifying pixel-based methods and run-based methods. State-of-the-art CCA algorithms are compared in terms of the number of memory accesses, which is a limiting factor for hardware-based acceleration, with key implementation trade-offs identified between hardware resources and worst-case processing speed.

A multi-TOPS AI core is presented for acceleration of deep learning training and inference in systems from edge devices to data centers. With a programmable architecture and custom ISA, this engine achieves >90% sustained utilization across the range of neural network topologies by employing a dataflow architecture and an on-chip scratchpad hierarchy. Compute precision is optimized at 16b floating point (fp 16) for high model accuracy in training and inference as well as 1b/2b (bi-nary/ternary) integer for aggressive inference performance. At 1.5 GHz, the AI core prototype achieves 1.5 TFLOPS fp 16, 12 TOPS ternary, or 24 TOPS binary peak performance in 14nm CMOS.

The combination of growth in compute capabilities and availability of large datasets has led to a re-birth of deep learning. Deep Neural Networks (DNNs) have become state-of-the-art in a variety of machine learning tasks spanning domains across vision, speech, and machine translation. Deep Learning (DL) achieves high accuracy in these tasks at the expense of 100s of ExaOps of computation; posing significant challenges to efficient large-scale deployment in both resource-constrained environments and data centers. One of the key enablers to improve operational efficiency of DNNs is the observation that when extracting deep insight from vast quantities of structured and unstructured data the exactness imposed by traditional computing is not required. Relaxing the "exactness" constraint enables exploiting opportunities for approximate computing across all layers of the system stack. In this talk we present a multi-TOPS AI core [3] for acceleration of deep learning training and inference in systems from edge devices to data centers. We demonstrate that to derive high sustained utilization and energy efficiency from the AI core requires ground-up re-thinking to exploit approximate computing across the stack including algorithms, architecture, programmability, and hardware. Model accuracy is the fundamental measure of deep learning quality. The compute engine precision in our AI core is carefully calibrated to realize significant reduction in area and power while not compromising numerical accuracy. Our research at the DL algorithms/applications-level [2] shows that it is possible to carefully tune the precision of both weights and activations to as low as 2-bits for inference and was used to guide the choices of compute precision supported in the architecture and hardware for both training and inference. Similarly, distributed DL training's scalability is impacted by the communication overhead to exchange gradients and weights after each mini-batch. Our research on gradient compression [1] shows by selectively sending gradients larger than a threshold, and by further choosing the threshold based on the importance of the gradient we achieve achieve compression ratio of 40X for convolutional layers, and up to 200X for fully-connected layers of the network without losing model accuracy. These results guide the choice of interconnection network topology exploration for a system of accelerators built using the AI core. Overall, our work shows how the benefits from exploiting approximation using algorithm/application's robustness to tolerate reduced precision, and compressed data communication can be combined effectively with the architecture and hardware of the accelerator designed to support these reduced-precision computation and compressed data communication. Our results demonstate improved end-to-end efficiency of the DL accelerator across different metrics such as high sustained TOPs, high TOPs/watt and TOPs/mm2 catering to different operating environments for both training and inference.

This letter presents a multi-TOPS AI accelerator core for deep learning training and inference. With a programmable architecture and custom ISA, this engine achieves >90% sustained utilization across the range of neural network topologies by employing a dataflow architecture to provide high throughput and an on-chip scratchpad hierarchy to meet the bandwidth demands of the compute units. A custom 16b floating point (fp16) representation with 1 sign bit, 6 exponent bits, and 9 mantissa bits has also been developed for high model accuracy in training and inference as well as 1b/2b (binary/ternary) integer for aggressive inference performance. At 1.5 GHz, the AI core prototype achieves 1.5 TFLOPS fp16, 12 TOPS ternary, or 24 TOPS binary peak performance in 14-nm CMOS.

A resource-efficient hardware architecture for connected component analysis (CCA) of streamed video data is presented, which reduces the required hardware resources, especially for larger image widths. On-chip memory requirements increase with image width and dominate the resources of state-of-the-art CCA single-pass hardware architectures. A reduction in on-chip memory resources is essential to meet the ever increasing image sizes of high-definition (HD) and ultra-HD standards. The proposed architecture is resource efficient due to several innovations. An improved label recycling scheme detects the last pixel of an image object in the video stream only a few clock cycles after its occurrence, allowing the reuse of a label in the following image row. The coordinated application of these techniques leads to significant memory savings of more than two orders in magnitude compared with classical two-pass connected component labeling architectures. Compared with the most memory-efficient state-of-the-art single-pass CCA hardware architecture, 42% or more of on-chip memory resources are saved depending on the features extracted. Based on these savings, it is possible to realize an architecture processing video streams of larger images sizes, or to use a smaller and more energy-efficient field-programmable gate array device, or to increase the functionality of already existing image processing pipelines in reconfigurable computing and embedded systems.

In this paper, an adaptive architecture for dynamic management and allocation of on-chip FPGA Block Random Access Memory (BRAM) resources is presented. This facilitates the dynamic sharing of valuable and scarce on-chip memory among several processing elements (PEs), according to their dynamic run-time memory requirements. Different real-time applications are becoming increasingly dynamic which leads to unexpected and variable memory footprints, and static allocation of the worst-case memory requirements would result in costly overheads and inefficient memory utilization. The proposed scalable BRAM memory management architecture adaptively manages these dynamic memory requirements and balances the buffer memory over several PEs to reduce the total memory required, compared to the worst-case memory footprint for all PEs. The run-time adaptive system allocates BRAM to each PE sufficiently fast enough as required and utilized. In a case study, a significant improvement in BRAM utilization with limited overhead has been achieved due to the adaptive memory management architecture. The proposed system supports different BRAM types and configurations, and automated dynamic allocation and deallocation of BRAM resources, and is therefore well suited for the dynamic memory footprints of FPGA-based reconfigurable architectures.

High-speed imaging is one of the key methods in a wide range of applications to understand basic physical processes. In traditional high-speed imaging a certain number of images is taken and stored in a high-speed memory directly attached to the image sensor of the imaging device. However the size of this memory limits the number of frames or the image size which can be capture. In this proposal a real-time processing of image data on a reconfigurable logic device, which is directly connected to the image sensor is proposed. This architecture is able to process a very high-speed pixel stream without the need of storing full images at all and is able to extract object feature in real-time. The proposed FPGA image processing architecture uses a combination of two recently proposed single pass algorithm for the task of feature extraction and is able to process up to several 1000 frames per second.

Calculation of mean, variance and standard deviation are often required for segmentation or feature extraction. In image processing, often an integer approximation is adequate. Conventional methods require division and square root operations, which are expensive to realize in hardware in terms of both the amount of required resources and latency. A new class of iterative algorithms is developed based on integer arithmetic. An implementation of the algorithms as a hardware architecture for a Field-Programmable Gate Array (FPGA) is compared with architectures using the conventional approach, which shows a significantly reduced latency while using less hardware resources.

A memory efficient architecture for single-pass connected components analysis suited for high throughput embedded image processing systems is proposed which achieves a high throughput by partitioning the image into several vertical slices processed in parallel. The low latency of the architecture allows reuse of labels associated with the image objects. This reduces the amount of memory by a factor of more than 5 compared to previous work. This is significant, since memory is a critical resource in embedded image processing on FPGAs.