Transformers evolved to dominate the state of the art in almost any deep leaning domain, and usually involve very large models that are computationally expensive for training and inference. Smaller Transformer variants are still useful, particularly for a wide range of embedded applications. Existing compilers for FPGAs often lack support for core operations of the Transformer, such as the attention mechanism. In this work, we leverage the FINN framework to enable the automatic synthesis of custom-tailored FPGA accelerators for quantized Transformer models. We describe the hardware design of the scaled dot-product attention operation following the streaming dataflow paradigm and the integration into the compiler infrastructure of the FINN framework. We demonstrate a small-scale design space characterization exploring scaling behavior and resource utilization of our implementation, followed by two exemplary case studies, covering a radio signal classification use case as well as text generation with small GPTs, to demonstrate the end to end toolflow starting from quantization aware training up to deployment on the device. We evaluate our approach in terms of resource utilization, throughput, latency, accuracy and perplexity.
In this paper, we present a design methodology for multi-model accelerators considering the similar nodes between the two DNN computational graphs. First, we use a maximal common subgraph search algorithm to automatically extract the similarities between computational graphs. Then, we merge the common subgraphs and map the merged computation graph to hardware. We also present an LSTM-GRU hybrid accelerator without the need for redesigning and redeployment. Experimental results indicate that the hybrid accelerator achieves an average reduction of 42.3 % in the total logical resource consumption compared to the independent LSTM and GRU accelerators.
The routing architecture significantly impacts FPGA performance and area. Traditionally, routing resources are divided into connection blocks (CBs) and switch blocks (SBs) to implement interconnects based on high-fanin multiplexers (MUXes). The number of MUX levels is typically fixed in past studies, resulting in signals traversing a consistent number of MUXes within defined routing blocks. Although there is a lot of research focusing on the routing model, how MUXes implement these connections is less explored. In this paper, we propose a novel routing architecture based on multi-level MUXes, allowing for flexible driving relationships based on connections through various numbers of MUX levels. We define parameters to describe the multi-level MUX architecture, covering the number of MUX levels and the type and number of MUX fan-ins, and conduct experiments to investigate the effectiveness based on these parameters. In addition, we employ Bayesian Optimization (BO) for design space exploration based on the above parameters and wire distribution. The experimental result demonstrates that the optimized multi-level architecture achieves a 13.6 % reduction in critical path delay (CPD) with a 4.3 % overhead in total routing area compared to General Routing Block (GRB) architecture using VTR benchmarks. The multi-level MUX result also surpasses the BO searched result without the exploration of MUX arrangement, showing improvements of 6.9 % in CPD and 2.1 % in total routing area. Additionally, it shows 8.6 % and 5.0% improvement in CPD and routing area respectively than the versatile interconnection block (VIB) architecture.
Vendors design the Programmable Logic Fabric (PL) of an FPGA for robustness over a wide range of customer designs, use cases, and operating scenarios. These specifications cover for Process-Voltage-Temperature (PVT) variations occurring on the PL and are meant to ensure reliable timing margins on the FPGA. With increasing power densities, advanced packaging techniques and the use of PL for data-intensive computations, there have been efforts to understand the effects of voltage transients on the PL and their dependence on workloads. In this work, we propose a scalable methodology based on soft digital sensor (SDS) to characterize on-die voltage transients for compute-intensive workloads. This method is scalable with respect to test, measurement, and debugging compared to prior work. In addition, we show how the proposed methodology can be extended to study trends of timing margins and voltagefrequency headroom on FPGAs. This can improve the energy efficiency of designs running at a fixed operating point or using dynamic voltage-frequency scaling. Our results gathered from an AMD Versal VCK190 FPGA board running multiple highactivity workloads show incidents of large voltage droops of nearly 100 mV using the SDS. We also observe between 18-31% frequency headroom available across the higher activity designs analyzed in the presence of droop and jitter. These margins potentially allow designers to reliably push the operating voltage and frequency beyond the limits specified by the FPGA vendors, leading to much improved energy efficiencies.
Table-lookup inference, enabling convolutions via table lookups instead of multiply-accumulate (MAC) operations, shows promise for energy-efficient binarized convolutional neural networks (BCNNs). However, existing FPGA-based table-lookup BCNN designs consume considerable hardware and power due to the low binding level and LUT reuse. To address this, we propose EGSBConv, a novel convolution engine optimized for tablelookup inferences. EGSBConv leverages separable convolution and channel grouping to reduce fan-ins, facilitating kernel-level binding, while mitigating accuracy loss through internal channel extension. Each convolutional kernel is bound into a single table stored in BRAMs to reduce power consumption. Experiments on CIFAR-10 demonstrate 86% and 94% LUT reductions as well as 4.5x and 15.3x power efficiency improvements compared to state-of-the-art MAC-based and table-lookup-based BCNN accelerators, respectively.
Directly performing high-accuracy remote sensing scene classification (RSSC) on the edge devices facilitates realtime processing of ground information. Current mainstream RSSC neural networks face challenges in hardware deployment or lack high classification accuracy. This paper proposes a hardware-friendly and high-accuracy multi-branch rotation convolutional neural network (mBR-CNN) and its FPGA accelerator for RSSC tasks. The proposed neural network leverages the rotational properties of 3x3 convolutional kernels to construct mBR-CNN with rotation invariance. Additionally, the proposed FPGA accelerator innovatively incorporates a multi-branch rotation convolution operator, supported by a multi-branch parallel pipeline structure to achieve optimal throughput. Deployed on the Xilinx Virtex7 XC7VX690T FPGA, it utilizes less than 40% of on-chip resources, achieving processing speeds exceeding 2000 frames per second (fps), with a state-of-the-art classification accuracy of 96.19% on the UC-Merced Land Use Dataset.
Hyperdimensional Computing (HDC), which leverages the distributed computational characteristics of hypervectors, is gaining significant attention due to its outstanding classification performance and hardware friendliness. However, the coarse-grained computational characteristics in HDC are largely unexplored. To this end, we present CGRA-HD, a coarse-grained reconfigurable architecture (CGRA) for accelerating the HDC algorithm. By analyzing the frequent operations and general sub-graphs, we explore the specialized Processing Element (PE) design for HDC to realize efficient acceleration. The specialized PE and the general one constitute a CGRA-HD array and have been integrated into a state-of-the-art RISC-V+CGRA SoC. Our evaluation shows that CGRA-HD greatly improves HDC performance over CPUs with decreased area and power consumption compared to the general CGRA.
Multi-layer perceptron (MLP) is often the preferred model for regression tasks. Recently, graph-based networks (GNN) have become popular for estimating post-implementation Quality of Results (QoR) in High-Level Synthesis (HLS). In addition to the node features composed of various HLS directives, local neighborhood information and global information in the topology are crucial for accurate QoR prediction. In this paper, we propose a Topology-aware MLP-Mixer Predictor (TMMP) based exclusively on an MLP structure to estimate postimplementation QoR. TMMP leverages correlation encoding and a feature-mixing structure to achieve the capability of extracting topological information similar to that of GNN. Combining this with a Gaussian Process-based Multi-objective Bayesian Optimization engine (GMBO), we ultimately achieve a design space exploration framework, TMM-DSE, which can rapidly and accurately identify optimal design parameters. Experimental results show that our simple and efficient TMMP achieves area prediction performance comparable to the current best GNNbased models, with power and critical path delay errors only between 4.64% and 6.77%, even surpassing the SOTA work. GMBO with TMMP can accelerate the DSE process by up to 69x, with an average speedup of 35x.
FPGA accelerators for convolutional neural networks (CNNs) have made significant progress in recent years. It is critical to fully utilize scarce digital signal processor (DSP) resources on FPGAs to implement CNNs dense multiplyand-accumulate (MAC) operations. Many works pack multiple parallel multiplications into a single DSP to improve the efficiency and performance of the DSP. However, these works usually pack two or four multiplications into the DSP, or use approximate computation to boost packing density at the expense of accuracy. In addition, neglecting the resource consumption of the packing and unpacking process leads to an increase in hardware area. To address these issues, this paper implements a single DSP packing six signed multiplications for lossless computation. Moreover, the resource consumption of the data packing and unpacking process is analyzed and optimized for the first time, which leads to a 48.7% decrease in look-up tables (LUTs) resource utilization. Compared to the state-of-the-art accelerator design, our system design achieves 2.03x throughput with almost the same resource consumption and lower energy efficiency.
Field programmable gate array (FPGA) boards equipped with high bandwidth memory (HBM) allow users to exploit the remarkable external memory bandwidth. Nonetheless, it may not be an easy task to optimize the memory usage, while fully utilizing the available HBM bandwidth. Dynamic memory management (DMM) in High-level Synthesis (HLS) has been introduced as a promising solution to achieve high memory usage at low design cost. However, existing DMM solutions only focus on BRAMs or DDR memories, lacking enough support for HBMs. To address these challenges, we propose HBMalloc, a source-to-source framework that supports both automatic HLS code transformation and systematic hardware generation. Experimental results show that the proposed allocators can achieve an average of 2x higher HBM bandwidth than prior works. Our pointer partition strategy can explore the parallelism and bring about a significant speedup in diverse memory-bounded applications.
This paper introduces Neural Connect, a novel approach for the run-time adaptation of dynamic Deep Neural Networks (dynamic DNNs) at both the high, application level, and the low-level of individual processing elements within each layer. At the application level we make neural network layers run-time configurable, supporting both mapping operationslike activation functions-and reduction operations-such as multiply-accumulate and convolution. Within layers we allow efficient, scalable sharing of low-level processing and memory resources among any number of neurons, employing compute pools of heterogeneous processing elements to carry out layer operations. We separate high-level application development from its low-level deployment, enabling neural networks and their processing elements to be developed independently.
Lower-upper decomposition (LUD) is one of the most popular matrix factorization techniques in linear algebra and has been widely used in many scientific and engineering applications. While prior studies have investigated various strategies to accelerate block LUD on FPGAs for arbitrary input sizes, they often suffer from one or more of the following limitations: 1) excessive resource utilization due to separate PE (processing element) designs for different matrix blocks with diverse computation patterns; 2) excessive on-chip memory usage due to buffer-based designs; and 3) insufficient parallelism as only one-level parallelism (either row-level or iteration-level parallelism) was exploited due to complex dependencies. To address those limitations, we propose FLUD, a streamingbased systolic array design on the FPGA to accelerate block LUD, which shares the systolic array to accelerate different matrix blocks and exploits both column-level parallelism and iterationlevel parallelism. First, FLUD implements a configurable systolic array that is shared by different LUD blocks and scalable to arbitrary input sizes. To further optimize its hardware resource efficiency, FLUD groups a column of PEs together to replace their FIFO connections with lightweight registers and reduce multiple copies of local control logic inside each PE (for the purpose of resource sharing among different LUD blocks) into a global one. Moreover, FLUD devises a computation schedule to effectively share the highly-optimized systolic array design among the execution of different LUD blocks. Lastly, to enable fast design space exploration on a given FPGA platform, we develop an automation tool to automatically generate the optimized FLUD design in Vitis high-level synthesis (HLS), where users can configure the design size and data precision based on their needs. Experimental results demonstrate that FLUD achieves a peak throughput of 427.95 GFLOPS for single-precision floatingpoint LUD, which is about 3x faster than state-of-the-art FPGA design. Compared to the LAPACK library running on a 12-core Xeon Silver 4214 CPU, FLUD achieves 4.71x higher throughput and 10.25x better throughput/watt.
Annealing processors have attracted attention as domain-specific computers to solve combinatorial optimization problems (COPs) efficiently. Their search performance can be further enhanced by the merge method that provides a capability of updating multiple variables simultaneously. However, directly implementing the merge method on annealing processors requires large-scale computational and memory resources. This paper proposes a parallel-trial double-update annealing (PDA) algorithm that efficiently integrates the merge method into the annealing computation flow. Additionally, a PDA processor is implemented on an FPGA by extending the existing near-memory architecture for annealing computation. Experimental results for several COPs demonstrate that the proposed PDA processor achieves superior solution search performance compared to a conventional annealing processor and GPU-based annealing computation.
Visual-Inertial Odometry (VIO) is a fundamental module in unmanned systems, and Convolutional Neural Network (CNN)-based feature extraction methods have been increasingly adopted in recent years. As Domain-Specific Architectures (DSA) for CNN have advanced, Bundle Adjustment (BA) in the VIO backend, rather than CNN, has become the computational bottleneck in embedded systems. However, the CNN accelerator consumes substantial FPGA resources, leaving limited capacity for BA. To address this issue, we design an efficient BA accelerator that utilizes the sparsity of the data and employs a workload segmentation and subtask scheduling architecture. Additionally, we propose a method to compress the frontend feature extraction algorithm. Experiments on the ZCU102 and a small drone demonstrate that our BA accelerator achieves 30.3x speedup over the ARM CPU, and higher resource efficiency compared to recent accelerators. The frontend and backend of the VIO system are accelerated by 3.1x and 3.7x, respectively, resulting in an overall processing speed of 19.7 fps in real world.
Resource dependency is essential for operation parallelization and resource reuse in scheduling. Existing reinforcement learning (RL)- and graph neural network (GNN)-based methods use data-driven approximation to determine resource dependency. The approximation accuracy is not guaranteed for different circuit structures. System of difference constraints (SDC) formulation has the potential to conduct exact resource dependency determination. Existing SDC method adopts heuristic transformation, degrading determination accuracy. In this paper, we propose a resource dependency-aware scheduling method. The method integrates SDC formulation with three GNN-based assistant tasks. We leverage GNN to perceive operation relations and conduct operation relation-aware resource dependency determination with three assistant tasks. These tasks enable resource optimization in SDC formulation and mitigate loss in the heuristic transformation, improving accuracy. Experiments show that our method provides an average 38.7% resource usage reduction, 8.5%, and 17.1% latency reduction, compared with the RL-based, GNN-based, and basic SDC methods, respectively.
This paper presents Compass, an end-to-end design space exploration (DSE) framework to automate highlevel synthesis (HLS) directive optimization efficiently. Compass incorporates a directed acyclic graph (DAG) based modeling method which utilizes a customized variational graph autoencoder (VGAE) to transform the original discrete space to a continuous embedding space to improve the optimization efficiency. An active-learning-based sampling algorithm PRD-AL is proposed to draw high-quality initial samples. To enhance the robustness of Compass, we propose a novel MOEBO algorithm based on ensemble Bayesian optimization which constructs multiple local agents in trust regions and maintains a global agent to rectify the optimization trend. Besides, a collaborative acquisition maximization strategy is developed to encourage multiple agents to select promising candidates competitively. The experimental results demonstrate the superior performance of our approach to approximate the Pareto frontier with more than 51.18 % ADRS improvement and 22.20% PPA gain compared to state-of-the-art DSE approaches on different benchmarks.
A high-quality routing algorithm is crucial to achieving high-speed FPGA designs, and it is one of the most timeconsuming steps in the FPGA CAD flow. Using multiple CPUs is one way to reduce route time. However, exploiting parallelism on the most performant algorithms incorporating negotiated congestion, directed searches, and incremental approaches has been challenging. We introduce two parallel routers extending the state-of-the-art PathFinder-based AIR router in VPR 8. The first is the baseline parallel router, based on the widely applied technique of recursively bi-partitioning the physical FPGA so nonoverlapping nets can be routed in parallel; however, scalability is limited by nets (often high-fanout) spanning large chip areas. The second router enhances the baseline by applying a new net decomposition method to enable fragments of nets to be routed in parallel for better scalability. For intra-cluster routing, Titan benchmarks, and eight threads, we obtain a speedup of 2.14x with the baseline and 2.38x with the net-decomposing router, compared to the latest VPR 8+ sequential router. On flat (singlestep) routing, the net-decomposing router achieves a speedup of 2.15x with eight threads. The routers are deterministic and serially equivalent, achieving wire length and critical path delay comparable to the sequential algorithm. The routers are being integrated into the open-source VTR framework, enabling the research community to build on this work.
Pruning is essential for deploying neural networks on edge devices, with unstructured pruning effectively compressing models without significant accuracy loss. Current methods focus mainly on weight magnitude, neglecting important factors like layer depth, activation nonlinearity, and gradient feedback, leading to suboptimal pruning. To overcome these issues, we propose a novel technique that adjusts layerwise sparsity dynamically using multiple contextual factors. Our score-based mechanism incorporates weight magnitude and adapts to modelspecific features such as layer depth, nonlinearity, and gradient information. This method preserves crucial connections, significantly reduces parameters, and consistently outperforms traditional pruning techniques, offering a robust, hyperparameter-free solution.
The performance of FPGA systems is increasingly limited by the latency and bandwidth of off-chip memory. The traditional ASIC solution of using caches has also been both studied and implemented on FPGAs. However, such works have largely focused on primary caches, and fail to optimize for the unique resources modern FPGAs offer like LUTRAMs. Additionally, many recent developments in ASIC caching (e.g. new cache replacement policies and victim caches) have not been evaluated on FPGA systems, which provide less opportunity for improvement because of lower cache miss latencies. This work presents TableCache: an open-source, configurable, last-level cache for FPGA systems that makes prolific use of LUTRAM-based structures. This allows our design to use 51% fewer LUTs than a directly-comparable commercial FPGA lastlevel cache while offering lower read-hit latency and running on average at 29% higher frequencies. The parameterization of TableCache also allows us to explore different cache replacement policies, where we find that Static Re-Reference Interval Prediction (SRRIP) can reduce average memory access time by 6.4% compared to Least Recently Used (LRU) while using fewer FPGA resources. We also find that the impact of last-level victim caching on FPGAs is largely negative, with a marginal 0.21% reduction in average memory access time (measured in cycles) significantly outweighed by a 32.4% increase to cache LUTs and a 16.9% reduction in operating frequency.
Generating custom hardware accelerators within a high-level programming framework is a complex task. However, compiling and generating accelerators for kernel programs that use recursion is often unsupported, necessitating manual refactoring to ensure compatibility with high-level synthesis (HLS) tools. The single-level abstraction in existing HLS frameworks limits recursive application transformations, hindering efficient acceleration. In this paper, we present an automated restructuring technique for recursive applications, built on multi-level intermediate representation (MLIR), aimed at generating high-quality hardware accelerators through HLS. We automatically detect recursive calls and analyze recursive data structures to generate loops, state machines, and stacks for simulating recursion, thus enabling an end-to-end compilation process. Experimental results show that, compared to a domain-specific language (DSL) compiler supporting recursion, our approach outperforms methods requiring manual modification of recursion into DSL representations in over 55% of the benchmark cases, achieving an average resource reduction of 51.3 %. Furthermore, compared to HeteroRefactor, we support more benchmarks and provide a synthesizable architecture.
