Recent advances in neural network design have emphasized attention-based Transformer models, which deliver state-of-the-art accuracy across applications in natural language processing and computer vision. However, the computational demands of Transformers pose significant challenges for deployment in latency-sensitive scenarios and resource-constrained devices, resulting from Transformers' quadratic scaling to sequence length in attention mechanism and extensive data movement. Many works have tried to alleviate these limitations while failing to give comprehensive solutions that consider the heterogeneous characteristics within Transformers. In this paper, we propose RECONFORMER, a system-on-chip that accelerates Transformers through multi-level reconfigurability. At component level, RECONFORMER introduces reconfigurable processing elements by incorporating range-based approximations for non-linear functions, resulting in 21% reduced logic usage per accelerator. At the system level, RECONFORMER has a run-time coarse-grained reconfigurability, which provides adaptive parallelism strategies and dynamic cache-coherence mode to address diverse kernel requirements. As a result, RECONFORMER achieves up to 5.54x speedup in multi-head attention and 3.61x in end-to-end performance improvement over a static system. RECONFORMER attains remarkable performance compared to other platforms, resulting in 13.03x and 5.30x higher efficiency than edge and server CPUs, respectively. Even compared to an edge GPU, RECONFORMER achieves 1.79x improvement in efficiency, highlighting our system as a compelling alternative for edge deployment.
Developing FPGA applications, either using RTL or HLS, requires knowledge about the platform on which the application will be deployed. The availability of resources changes according to each FPGA model. To deal with this, the components of applications may be implemented using different primitives. For example, Floating-Point operations can be implemented using Flip-Flops and Look-up Tables, or with the addition of DSP units, the same happens for other components, such as FIFOs. During development, the implementation of the components is often chosen and used for the rest of the design. Using Integer Programming, this work explores how different floating-point implementations can be mixed to achieve performance optimization in a stencil application. Furthermore, it discusses how the same idea can be used for other components and applications.
Field-Programmable Gate Array (FPGA) routing is computationally expensive, taking hours or days with no guarantee of success. While prior work has used machine learning (ML) to guide placement and routing or predict routing outcomes, the process remains challenging to model precisely. This demo presents a new machine learning-based tool that addresses this problem.
The prefabricated resources in a field-programmable gate array (FPGA) can pose challenges resulting in unexpectedly long routing times. This sometimes leads to FPGA engineers prematurely terminating a viable routing run because they mistakenly believe the lengthy runtime indicates an unroutable design. In other cases, the FPGA engineer may wait long periods of time on a run that is doomed to never converge to a solution. As this leads to time wasted on runs which never reach design closure in either case, it is ideal to instead have an ML model decide whether or not to terminate routing. In this work, we introduce data-driven machine learning (ML) techniques for predicting both FPGA design routability and routing runtime continuously during routing.
The recent edge computing such as the Internet of Things needs to handle a large amount of data streams such as generated from sensory and video devices. To enhance the application performance, one of the major techniques is to reduce the data amount and accelerates the communication performance. We proposed a reconfigurable lossless data compression technique called Adaptive Stream-based Entropy (ASE) coding that handles such continuous data stream at high data rate. Our demonstration shows energy impact for a system that has a data sink in memory devices by reducing the number of the I/O accesses applying ASE coding. We implemented a system with an FPGA that handled video data stream with buffering compressed video frames to DDR3 DRAM and also displays the original video stream on a monitor. The system improves energy consumption of the memory devices without performance degradation.
Power-efficient mid-range FPGAs enhance computational capabilities, reduce system size and weight, and extend battery life in power-sensitive applications such as autonomous robots and AI-enabled devices. This paper presents architectural enhancements in Lattice Avant mid-range FPGAs, manufactured using 16 nm FinFET process technology. We explore the benefits of Four-Input Lookup Tables (LUT4s) organized into Programmable Function Unit (PFU) clusters of 12 LUT4s, along with associated registers and other logic. PFUs are coupled with optimized routing to construct Programmable Logic Cells (PLCs), which are then arrayed to form the FPGA fabric. These enhancements result in reduced area, lower static power, and decreased dynamic power consumption compared to other LUT and PFU size choices. Our analysis, which combines published benchmarks, internal analysis of LUT and PLC areas, as well as internal benchmarks, indicates that, on average, users will have about 30 % LUT4s leftover in a LUT4-based PLC array if that array is constructed in the same die area as a LUT6-based PLC array that was large enough to fit the original design. Alternatively, an average design can fit into a LUT4 PLC array occupying approximately 70 % of the die area required by a LUT6-based PLC array, thereby reducing system size and power consumption. Additionally, the slice clock in the Avant architecture, which serves both registers in the slice and includes a per-slice clock turnoff feature, reduces total register clock toggles by 43 % compared to a slice that has 12 registers. The reason is that a partially used smaller slice, which has fewer total registers, results in fewer unneeded registers toggling compared to a partially used larger slice. This reduction in unnecessary toggles, in turn, decreases the clock dynamic power in the final stage of the clock distribution.
Developing multi-FPGA applications is a complex task. It involves using several tools, to deal with data movement, FPGA usage, and application execution. OMPC-F is an OpenMPbased framework that abstracts mostly of the effort to distribute FPGA accelerated applications in cluster. OMPC-F uses only standard OpenMP and supports single- and multi-FPGA applications as well as memory-based or streaming-accelerated kernels.
This research addresses the energy-efficient deployment of deep neural networks (DNNs) on FPGAs for edge-cloud computer vision applications. We evaluate and compare classic convolutional neural networks, Vision Transformers (ViTs), and alternative approaches that may be more suitable for FPGA deployment, such as differentiable boolean logic (DiffLogic) networks. Our methodology employs hardware-aware quantization techniques and specialized deployment frameworks to optimize performance on resource-constrained systems. The expected contributions include: i) energy characterization of DNN accelerators, ii) comparative analysis of FPGA toolchains, iii) optimization of modern networks with a focus on attention mechanisms, and iv) evaluation of DiffLogic networks as an alternative architecture. We demonstrate our approach through both toy cases (e.g., MNIST dataset) and a realistic Driver Distraction Detection (DDD) case study that requires real-time execution on edge devices. Preliminary results show that our optimized model achieves 66 % test accuracy on the DDD problem with very low power consumption on a PYNQ-Z2 board.
FABulous is an easy usable, yet powerful and complete eFPGA (embedded FPGA) Framework covering all aspects of an eFPGA ecosystem. FABulous eFPGAs had shown good area density in both standard-cell and custom cell flows and the framework allows various customizations, including the integration of custom primitives, I/O cells, or complex blocks like CPUs cores or ADCs. So far, more than 10 chips containing FABulous eFPGAs had been designed in technology nodes ranging from 28-180 nm by seven different universities. This demo shows how embedded FPGAs (eFPGAs) can be 1) specified, 2) verified and tested, 3) integrated into an ASIC, and 4) programmed in Verilog or VHDL using the all open FABulous eFPGA framework. Moreover, the demo will showcase two boards featuring FABulous FPGAs, including the first FABulous open-everything FPGA.
Achieving the lowest possible communication latency between an FPGA accelerator card and host CPU software is critical for many applications, such as high-frequency trading, in-memory storage systems, and 5G processing. However, no freely available PCIe DMA engine (DMAE) is currently optimized for low latency. To address this, we present DMA Calypte, an open-source, platform-independent, ultra-low-latency DMAE. We implemented numerous optimizations at the DMAE, software, and device driver levels to minimize communication latency over PCIe. DMA Calypte's functionality was validated on accelerator cards featuring AMD Kintex UltraScale+ and Agilex 7 F-Series chips with a PCIe Gen3 x8 interface. We achieved a round-trip time between software and FPGA of just 790 ns and a throughput of up to 35 Gbps.
For perception systems in mobile robotics - particularly in demanding, highly dynamic environments - event cameras (DVS - Dynamic Vision Sensors) are being increasingly utilised as an alternative to conventional vision sensors. The real-time processing of the registered spatio-temporal sparse point cloud must satisfy stringent requirements in terms of latency, throughput, and energy efficiency. In this demonstration, we present our method for implementing Graph Convolutional Neural Networks on a heterogeneous SoC FPGA platform, aimed at meeting these constraints. We address the challenges associated with integrating event-based sensors with reconfigurable hardware, as well as the intricate relationship between latency, performance, and hardware resource utilisation.
GEMM (General Matrix Multiplication) is a fundamental operation in deep learning (DL), serving as the key computing kernel for neural network layers, such as convolutional and fully connected layers. As the deployment of DL models expands beyond high-performance computing to resource-constrained embedded systems, there is a growing demand for efficient GEMM implementations that can meet strict power budgets and real-time processing requirements. This paper introduces a dual-issue in-order RISC-V core with an inpipeline GEMM accelerator based on the VeeR EH1 core [13], eliminating control signals and data transfer overhead inherent to coprocessors and standalone accelerators. By integrating GEMM execution directly into the pipeline, the design leverages core utilization and enables compiler-time preloading of operands, streamlining memory access and reducing dynamic branching, thereby minimizing branch mispredictions. Further, pipelined load/store operations mitigate memory access stalls and cache misses, thereby enhancing the pipeline throughput. Speedup, measured using Verilator simulation, demonstrates an average of similar to 5x over the reference VeeR EH1 baseline core. By integrating GEMM acceleration within the pipeline, this work bridges the gap between domain-specific accelerators and general-purpose cores, delivering scalable computing performance.
Field-Programmable Gate Array (FPGA) routing is computationally expensive, taking hours or days with no guarantee of success. While prior work has used machine learning (ML) to guide placement and routing or predict routing outcomes, the process remains challenging to model precisely. A recent work has proposed using ML to predict the number of iterations remaining in a negotiated congestion router while it runs, enabling early termination of routing runs unlikely to succeed. However, that approach has key limitations hindering its utility: (1) iteration count is poorly correlated with runtime, (2) it ignores prediction confidence when deciding whether to exit, and (3) it cannot assess whether extending a routing run past a predefined limit is worthwhile. This paper presents a new ML-based framework that addresses these limitations. We introduce a method for estimating router workload based on node traversals in the FPGA routing resource graph, which strongly correlates with runtime and enables more accurate early exit decisions. We also propose a tunable success-confidence threshold that allows users to trade off runtime against success rate and we design a ML mixture of experts architecture to enable this thresholding effectively. Finally, we show how our architecture can “look ahead” to determine whether a routing run is likely to succeed if allowed to delay termination and continue past its initial time limit. We implement our approach on top of the negotiated congestion routing algorithm and, in our experiments on very difficult-toroute circuits, we find that the number of circuits successfully routed within a fixed cumulative routing time budget increases by 215 % compared with the approach from prior work.
Recently, the Vision-LSTM (ViL) model, built upon Extended Long Short-Term Memory (xLSTM) building blocks, has attracted widespread attention due to its excellent performance and its linear complexity. Due to the unified compute architecture of GPUs, efficiently deploying the ViL layer on GPU is challenging because of its complex dataflow, thereby necessitating custom hardware accelerators. Moreover, significant differences in the activation distribution and quantization sensitivity among modules, making most existing quantization methods for CNNs not suitable for ViL layers. To address these challenges, we propose an intra-layer mixed-precision quantization method, termed ClipQuant, for the ViL layer. By introducing variable quantization range parameters a and scaling parameters ss, assessing the quantization sensitivity of each module, and imposing suitable quantization parameter constraints, we achieve near-lossless quantization of the ViL layer (accuracy loss < 1%) with low precision. Furthermore, to efficiently process the quantized low-bit-width data, we propose a self-correcting, packed DSP MAC calculation structure based on the Xilinx DSP48E2 slices. We first achieve error-free cascaded DSP MAC operations with minimal additional hardware resources. Based on this, we design an efficient FPGA-based stream-style accelerator, EViL. By crafting dedicated hardware units for the various modules within the ViL layer and allocating appropriate data streams and throughput, the accelerator attains high hardware resource utilization. We implement EViL on the Xilinx ZCU102 platform. Compared to the NVIDIA RTX 3090 baseline, EViL achieves a 1.10x similar to 1.49x performance improvement and a 36.95x similar to 45.50x improvement in energy efficiency.
Modulo scheduling is a highly effective approach for maximizing throughput in loops with static memory dependencies, interleaving computations across consecutive loop iterations. Despite substantial advancements in scheduling procedures, it remains the most computationally intensive phase for high-level synthesis flows. A recent approach encodes the modulo scheduling problem as a series of Boolean satisfiability (SAT) instances, capitalizing on the efficiency of modern SAT solvers. This approach significantly reduces solving time and increases the availability of throughput-optimal solutions compared to integer linear programming-based algorithms. This work introduces two enhancements for SAT-based modulo scheduling: (i) an algorithm to rapidly calculate a lower bound for the schedule length, improving the identification of latency-optimal schedules; and (ii) a streamlined SAT formulation with fewer clauses, facilitating quicker solver decisions. Extensive experimental evaluations show that these improvements lead to an increased number of throughput-optimal and latency-optimal solutions.
Machine Learning has been successfully adopted to estimate power consumption for FPGA designs using features from the early stages of High-Level synthesis. However, existing ML-based power models work only for the same FPGA the model is trained on, and do not generalize well for unseen or futuristic FPGAs due to the lack of consideration of architectural and technology features, which are highly relevant to the power consumption. In addition, state-of-the-art analytical models take hours to synthesize and simulate hardware designs to perform accurate power analysis. It leads to a long turnaround time and inefficiency for researchers who want to explore the effects of the FPGA architecture on power. In order to tackle the problem, we proposed ATAPP, a novel Graph Neural Network-based power model where both design switching activities and architecture features are encoded to make power prediction. The features of the FPGA architecture are extracted with RapidWright at the tile level of the layout, and the embeddings are generated using positional encoding. The design features are composed of switching activities at the intermediate representation operator level and the data flow of the design in a graph. With the two representations, ATAPP is able to predict the average dynamic power with a new design overlay on an unseen FPGA. Our experiments show that ATAPP demonstrates an average error of 13.09 % with unseen FPGAs and designs, while the best prior produces more than 40 % error and XPE produces more than 200 %.
While routing failures are rare in contemporary FPGAs, difficulties in routing a design can translate into timing closure failure. However, unlike many customer-visible metrics, such as performance, routability of a design on a given FPGA is challenging to quantify. Nevertheless, FPGA architects and CAD developers monitor several metrics, such as heap operations, wire usage, number of routing iterations, and outstanding congestion during routing iterations, to indirectly assess and compare the routability of different designs on various FPGAs. In academic research these metrics can be correlated against channel width sweeps and experiments varying a small set of connectivity parameters on a fixed architecture. In modern commercial FPGAs routing topology can vary significantly between architectures and detailed connectivity patterns are highly optimized to the physical characteristics of the target process node, making channel width sweeps not applicable to evaluating and optimizing commercial FPGAs. In this work, we introduce a new metric to quantify routability in FPGAs. To this end, we start by extracting routability metrics across a set of experiments on an Agilex device as we reduce its available routing tracks. Next, we introduce our proposed routing struggle metric, derived from heap operations trajectory, to quantify and compare the routability of designs as we progressively maim the routing fabric of the device. Next, we perform several comparisons between Stratix 10 and Agilex FPGAs and show that the Agilex device is significantly more routable, but this routability difference is not captured using traditional routability metrics. However, routing struggle can provide a comparison in terms of routability between the Agilex and Stratix 10 devices that aligns with the trend of maximum operating frequency in progressively shrinking placement regions in our area-constrained experiments. Finally, we show that routing struggle is not limited to Quartus experiments and can also be applied in the academic VTR flow.
Stereo visual Simultaneous Localization and Mapping (SLAM) enables autonomous agents to navigate complex environments by simultaneously building metric maps and localizing within them. In this paper, we present the first FPGA-based System-on-Chip (SoC) implementation of stereo visual SLAM. Our design reduces stereo feature matching latency by dynamically grouping descriptors based on localized image regions and descriptor similarity, while preserving matching accuracy. We also revisit conventional key-frame generation strategies and show that some criteria lead to unnecessary map expansion and runtime instability on resource-constrained platforms. To address this, we propose a new key-frame generation method based on the tracking status of local map points in the current frame. Extensive evaluations on standard benchmarks show that our system achieves high robustness and real-time performance with notable performance-power gains, while maintaining accuracy comparable to state-of-the-art stereo SLAM methods on embedded platforms. Real-world experiments further confirm our system's suitability for practical robotic applications.
Neuromorphic inference applications primarily rely on general matrix-matrix multiplication (GEMM) and twodimensional convolution (Conv2D) operations. When conventional artificial neural network (ANN) acceleration techniques are employed, these computations often necessitate extensive data reordering, which imposes significant overheads, especially in lightweight embedded systems with limited CPU and memory bandwidth. To address this challenge, we propose a unified accelerator architecture executing GEMM and Conv2D operations without data reordering. The accelerator is co-designed with a neuromorphic software framework tailored for lightweight embedded systems. To validate effectiveness, we implement a neuromorphic processor incorporating the proposed accelerator on an FPGA. Evaluation results across four representative neuromorphic applications demonstrate that the proposed design reduces execution time and energy consumption by 69% and 89%, respectively, compared to conventional ANN accelerators.
