Reasoning with Large Language Models (LLMs) has become a pivotal research topic because their logical abilities significantly surpass those of standard LLMs. LLM reasoning typically forms multiple chains of thought, action-by-action, and selects the best one as the final answer. However, the inference overhead of LLM reasoning is more than an order of magnitude higher than that of LLM. Despite the emerging shift towards memory-optimized algorithms and near-memory hardware, we still face the following challenges: (1) Existing memory-centric algorithms (e.g., KV cache technique) have low computational utilization (< 4% on NVIDIA A100 GPU) due to intensive memory access for inter-action data. (2) Emerging hardware architectures (e.g., near-memory processing) fail to fully utilize the inherent parallelism due to dependencies among models, leading to low utilization of memory bandwidth.To tackle these challenges, we propose Endor, a hardware-algorithm co-design to accelerate the inference of LLM reasoning efficiently. We identify that the auto-regressive decoding of LLM reasoning changes from the token level to the action level in terms of the computing paradigm. At the algorithm level, we propose a "nearly-decode-only" method which encompasses an efficient inter-action cache reuse method and a prediction-based pipeline optimization to reduce computation overhead. At the hardware level, we propose Endor-NMP, a near-memory accelerator featuring a score-aware cache management architecture and a heterogeneous mapping dataflow. Endor fully exploits both interaction and intra-action parallelism to improve memory bandwidth utilization. Experimental results demonstrate that neither existing algorithms nor hardware can achieve the expected acceleration. Endor achieves an end-to-end average speedup of 2.97× and 2.52× compared to the NVIDIA A100 GPU and advanced LLM accelerators on multiple models and datasets.

Long-term LLM agents need persistent memory that can track changing facts and provide relevant evidence across sessions. Existing memory systems often store observations as isolated records, summaries, or indexed fragments, which makes evidence aggregation, fact revision, and memory maintenance difficult. We propose Infini Memory, a maintainable text-based persistent memory architecture that treats agent memory as topic-structured documents. Each topic document serves as a semantic unit for collecting related evidence, preserving metadata, and revising facts over time. New observations are first staged in a buffer and periodically consolidated into coherent textual contexts. At inference time, an agentic retrieval procedure lets the LLM read memory through iterative tool calls rather than a single retrieval step. On MemoryAgentBench, Infini Memory achieves 64.7

The rapid scaling of large language models (LLMs) has significantly increased GPU memory pressure, which is further aggravated by training optimization techniques such as virtual pipeline and recomputation that disrupt tensor lifespans and introduce considerable memory fragmentation. Such fragmentation stems from the use of online GPU memory allocators in popular deep learning frameworks like PyTorch, which disregard tensor lifespans. As a result, this inefficiency can waste as much as 43

Deep neural network (DNN) accelerators integrated with RISC-V Instruction Set Architecture (ISA) extensions have enabled efficient computing on resource-constrained platforms. However, their specialization in regular compute patterns limits effectiveness on irregular workloads, making it challenging to achieve high throughput and energy efficiency. To tackle these challenges, we present ReNN-RV, which integrates a computation-aware RISC-V ISA extension with an instruction-driven processing pipeline to efficiently accelerate run-time reconfigurable processing elements (RePEs). The computation-aware ISA employs configurable opcodes and custom encodings to support fine-grained task scheduling, while an instruction-driven pipeline implements it with minimal control complexity. Moreover, the RePE accelerator provides seamless switching between multiply-accumulate (MAC) and non-MAC operations by configuring a path multiplexer to realize multiple operators at run time. Experimental results demonstrate that ReNN-RV achieves average reductions of 14.6 & times; in cycle count and 15.3 & times; in execution time across representative DNN workloads compared with the baseline RISC-V design. On average, ReNN-RV outperforms state-of-the-art designs by 10.1 & times; for energy efficiency and 10.3 & times; for computational throughput.

Sparse activation is caused by the activation function (e.g., ReLU) in the feed-forward network (FFN) of large language models (LLMs), and recently emerges as a promising method for LLM inference acceleration in resource-constrained scenarios by effectively reducing computational workload and memory requirements with >80 % predicted dynamic sparsity. In this paper, we identify the heavy and dynamic data transfer is the primary reason for the significant synchronization and poor GPU utilization during decoding phase of LLM inference with sparse activation, and propose to apply the near-data-processing (NDP) architecture to handle the dynamic sparse activation, while addressing three critical challenges for further NDP-GPU collaboration optimization. (1) Under-utilization of DRAM bandwidth during memory access of NDP. (2) Workload imbalance across channels during computation of NDP. (3) Time-consuming parsing of the sparse predicted pattern during NDP-GPU collaboration. To tackle the above challenges, we present SpAct-NDP, the NDPGPU heterogeneous architecture for efficient LLM inference with sparse activation. (1) For the memory access during NDP, we design the the specific sparsity-aware weight mapping strategy considering the characteristics of sparse activation to improve DRAM bandwidth utilization by balancing the bank workload and eliminating redundant memory access. (2) For the computation during NDP, we propose two-level heuristic scheduling system to achieve channel-wise workload balance. (3) For the collaboration of NDP-GPU, we point out that the parsing of the predicted sparse pattern is more suitable for GPUs with high parallelism and propose the request-weight pair parsing mechanism according to the input requests and sparse pattern on GPU, reducing $\sim 3 \times$ execution time and $\sim 9 \times$ memory. Experiments show that SpActNDP achieves up to $2.17 \times$ and $1.92 \times$ end-to-end speedup and $1.53 \times$ and $1.45 \times$ energy efficiency compared with the SOTA software frameworks for LLM with sparse activation on NVIDIA RTX 3090 and NVIDIA Tesla A100.

AI and HPC infrastructure increasingly serves workload portfolios that combine dense tensor computation, sparse kernels, large memory footprints, and communication-intensive collectives. Supporting these portfolios requires coordinated choices across accelerators, memory tiers, scale-up fabrics, and cluster networks. The resulting Cross-layer Heterogeneous System (XHS) design space is difficult to explore: hardware choices change legal task mappings, while rack power, switch radix, cabling, and cost constraints invalidate many candidates. We present CHASE, an application-driven framework that searches physically feasible XHS architectures through the workloads they must execute. CHASE represents candidates as hierarchical typed graphs and rejects designs that violate deployment constraints. It avoids intractable joint hardware-mapping search with a decoupled two-level loop: an inner mapper translates hardware-independent workload DAGs into topology-aware event traces, a calibrated event-driven simulator evaluates each mapping, and an outer telemetry-guided optimizer evolves the hardware graph. We evaluate CHASE on sparse-computing and LLM workloads. Its mapper remains within 6.06% of exhaustive optima while reducing mapping time by 60.5% on average relative to PEFT. Compute-model errors average 4.4-7.5%, and communication validation reproduces key trends across physical platforms. The outer search reaches near-global optima within 64 iterations. End-to-end case studies show that sparse workloads favor criticality-aware heterogeneous pods, whereas LLM inference favors scale-up islands; the resulting designs deliver 6.20$\times$ and 2.12$\times$ geomean speedups, respectively, while reducing cost and power relative to the baselines.

Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GPU computing systems, particularly on consumer-grade GPUs. By exploiting concurrent hardware execution, overlapping computation and communication latency becomes an effective technique for mitigating the communication overhead. We identify that an efficient and adaptable overlapping design should satisfy (1) tile-wise overlapping to maximize the overlapping opportunity, (2) interference-free computation to maintain the original computational performance, and (3) communication agnosticism to reduce the development burden against varying communication primitives. Nevertheless, current designs fail to simultaneously optimize for all of those features. To address the issue, we propose FlashOverlap, which utilizes a novel signaling mechanism: when part of the output finishes, the computation kernel sends a signal to trigger the communication of that part, while continuing the computation of the remaining part (interference-free computation). Consequently, the communication of the finished part and the computation of the remaining part can be overlapped. On top of the signaling mechanism, FlashOverlap comprises two key components: (1) the determination of the signaling timing to boost the overlap efficiency (tile-wise overlapping), and (2) a pre-communication reordering to create the contiguous address for finished data, enabling communication by simply calling NCCL APIs (communication agnosticism), and a post-communication reordering to correct the data order. Experiments show that FlashOverlap achieves up to 1.65x speedup through overlap, outperforming existing works in most cases. Code is available at https://github.com/infinigence/FlashOverlap.

Optical computing is becoming a promising solution to meet the growing computational demands of increasingly large-scale deep neural networks (DNNs). However, high power consumption from analog-to-digital (ADC) and digital-to-analog (DAC) conversions poses significant challenges for optical computing. Spiking Neural Networks (SNNs), with their binary spike-based input and output, show the potential to address this issue by reducing the need for high-precision DAC/ADC.In order to exploit the complementary nature of optical computing and spike-based processing, this paper proposes the Partially Incoherent Configurable Optical Computing Architecture for SNN Acceleration (PICoSNN). We address three critical challenges: phase errors in coherent optical computing, limited configurability in weight-stationary architectures, and inefficient mapping of general SNNs to optical computing hardware. We integrate partially incoherent tensor cores with optical leaky integrate-and-fire neurons, minimizing ADC/DAC overhead while supporting dynamic weight mapping. Further, we propose KV Spiking Self-Attention to enable efficient attention with 1-bit multiplications. Experimental results show that PICoSNN achieves up to 70.54× higher throughput and 8.13× lower energy consumption compared to ASIC implementations, while delivering 15.46× better throughput per area and 17.67× better energy efficiency per area than state-of-the-art photonic accelerators.

As an emerging type of AI computing accelerator, SRAM Computing-In-Memory (CIM) accelerators feature high energy efficiency and throughput. However, various CIM designs and under-explored mapping strategies impede the full exploration of compute and storage balancing in SRAM-CIM accelerator, potentially leading to significant performance degradation. To address this issue, we propose CIM-Tuner, an automatic tool for hardware balancing and optimal mapping strategy under area constraint via hardware-mapping co-exploration. It ensures universality across various CIM designs through a matrix abstraction of CIM macros and a generalized accelerator template. For efficient mapping with different hardware configurations, it employs fine-grained two-level strategies comprising accelerator-level scheduling and macro-level tiling. Compared to prior CIM mapping, CIM-Tuner's extended strategy space achieves 1.58$\times$ higher energy efficiency and 2.11$\times$ higher throughput. Applied to SOTA CIM accelerators with identical area budget, CIM-Tuner also delivers comparable improvements. The simulation accuracy is silicon-verified and CIM-Tuner tool is open-sourced at https://github.com/champloo2878/CIM-Tuner.git.

Large Language Models (LLMs) with 70 billion or more parameters are increasingly being deployed in cloud-based Model-as-a-Service (MaaS) scenarios. To meet the demands of such deployments, MaaS providers require batched LLM decoding systems that can deliver high System Throughput (STP) while minimizing Total Cost of Ownership (TCO). However, existing FPGA-based solutions predominantly focus on small-batch or single-batch inference, which fails to meet the computational requirements of batched LLM decoding, resulting in performance gaps of up to 7.96 & times;. Moreover, the low utilization of multi-head attention operations in batched decoding scenarios, e.g., only 3.72% on A100 GPUs, further constrains throughput and inflates TCO. To address these challenges, this article introduces CD-LLM, a heterogeneous multi-FPGA system designed for efficient batched decoding of LLMs with 70B+ parameters, built upon a Compute-Dedicated architecture. First, we propose a memory-aligned mixed-precision quantization engine to reduce workload. By employing importance-aware quantization, we compress Llama-3.1-70B to an effective 3.45-bit representation and achieve 72.33% bandwidth utilization through memory-aligned data packing. Second, we present a compute-dedicated FPGA architecture that maximizes peak performance by leveraging FPGA-specific resources such as DSPs, BRAMs, and LUTs. The compute-dedicated architecture enables CD-LLM to reach a peak performance of 59.90 TOPS at 600 MHz on U250 FPGA. At last, we introduce a heterogeneous master-slave multi-FPGA system to achieve higher utilization. By pipelining attention and linear layer computations across master and slave FPGAs, CD-LLM achieves utilization rates of 83.08% for linear layers and 68.30% for attention layers. CD-LLM is designed with a heterogeneous multi-FPGA architecture, with an HBM-enabled FPGA as the master accelerator and eight DDR-based FPGAs as slave accelerators. When deployed for inference on the Llama-3.1-70B model with a batch size of 256, CD-LLM achieves a throughput of 2,721.79 tokens/s. This represents a 6.11 & times; improvement in STP and a 4.71 & times; reduction in TCO compared to an eight-card RTX3090 GPU system. Furthermore, CD-LLM substantially outperforms the state-of-the-art eight-card FPGA accelerator FlightLLM, delivering 16.15 & times; higher STP and 14.56 & times; lower TCO.

Neural networks (NNs) have exhibited excellent performance in various fields of artificial intelligence. However, the primary operations in these mainstream models, including matrix-vector multiplication (MVM), element-wise multiplication (EWM), and depth-wise convolution (DWConv), require massive data movements during computation, which greatly impacts NNs' inference performance. The emerging Processing-In-Memory (PIM) architectures have shown great potential to overcome the memory wall problem. However, constrained by the supported data format and operator type, directly adopting PIM architectures for neural network acceleration faces three challenges: (1) Floating-point (FP) format has been widely adopted for ensuring high algorithm accuracy. However, Resistive Random-Access Memory (RRAM)-based analog PIM architectures perform integer (INT) MVMs in the analog domain, limiting their application to the more accurate FP format; (2) Static Random-Access Memory (SRAM)-based digital PIM architectures require additional circuits to support the FP format, and the SRAM capacity cannot satisfy the storage requirement of latest large language models (LLMs); (3) When performing the operators with few accumulation steps, such as EWMs and DWConvs, only few memory units in PIM architecture are activated, resulting in severe device under-utilization. To tackle the above challenges, this article proposes an RRAM and 3D-SRAM-based hybrid PIM architecture, achieving FP-based algorithm accuracy, high device utilization, and high energy efficiency. At the software level, we first analyze the impact of quantization errors on NN's inference accuracy. For the quantization error-insensitive MVM operations, we propose the PIM-oriented exponent-free non-uniform (PN) data format. The proposed PN format can be flexibly adjusted to fit the non-uniform distribution and approach FP-based algorithm accuracy using bit-slicing-based full INT operations. For the quantization error-sensitive EWM/DWConv operations, we introduce the multiplication-free approximated FP multiplications to reduce the additional hardware overhead. At the hardware level, we propose a hybrid PIM architecture, including an RRAM analog PIM using shift-and-add for PN-based MVMs, and a 3D-SRAM digital PIM with high utilization for DWConv/EWM operations. Extensive experiments on CNNs and attention-free LLMs validate that the proposed PIM architecture achieves up to 99.4x and 33.9x speedup with 5697.7x and 8.2x energy efficiency improvement compared to GPU and PIM-baseline, respectively. With the proposed PN format and approximated FP multiplications, the algorithm accuracy of CNNs and attention-free LLMs can be improved by up to 3.01% and 10.18%, respectively.

Near data processing (NDP) based on DRAM has emerged to be a promising solution to the “memory wall” problem of machine learning models. From the algorithmic perspective, group-wise low-bit floating-point (FP) quantization has become an important trend for both efficient training and inference. Integrating low-bit FP quantization into NDP also notably shrinks the memory footprint of large models, alleviating the memory-capacity constraints of NDP architectures. However, existing NDP compilers struggle to support efficient low-bit FP computation on NDP. First, different quantization configurations exhibit different preferences for NDP compilation strategies. Second, the fine-grained grouping leads to frequent switching between quantized value access and group scale access during computation, increasing the DRAM row-buffer miss rate. Third, fine-grained grouping triggers frequent high-precision dequantization operations, causing significant latency overhead. To address these challenges, this paper proposes FlexQ-NDP, an NDP compiler tailored for general low-bit FP computation. Firstly, we develop an open-source simulation framework 11 Available at https://github.com/ISCA26-FlexQ-NDP-ae/flexqndp to model the low-bit FP computation overhead on NDP. Secondly, we design a scale-value interleaved FP layout, effectively reducing DRAM row-changing overhead. Thirdly, we propose a dequantization-hiding technique based on instruction reordering to reduce the DRAM idle time induced by frequent dequantization operations. Finally, we develop a lightweight compilationspace pruning and search strategy to enable efficient low-bit FP computation on NDP. Extensive experiments show that FlexQ-NDP achieves up to $3.29 \times$ speedup over existing compilation strategies on various low-bit FP quantization configurations.

With the continuous growth of deep learning model sizes, their training processes face significant challenges in fault tolerance efficiency. Existing solutions, however, struggle to deliver both low communication overhead and effective fault tolerance. This paper presents C ^3 FT, a Computation-Centric Checkpointing scheme for efficient Fault Tolerance of distributed large model training. Specifically, C ^3 FT utilizes the CPU resources to update the optimizer states, and then uses the optimizer states on CPU as the checkpoints. Therefore, updating the optimizer states (including the model parameters) automatically updates the checkpoints on CPU, which also avoids the costly communication overhead in existing fault tolerance solutions. Overall, C ^3 FT achieves the high efficiency of fault tolerance and communication simultaneously. Evaluation results show that when training a 70B model on 32 A100 GPUs (4 nodes × 8 cards), C ^3 FT improves the training throughput by 1.69 × compared to Gemini (the SOTA solution with in-CPU-memory checkpointing).

The Mixture-of-Experts(MoE)framework has become a pivotal approach for enhancing the knowledge capacity and inference efficiency of Large Language Models(LLMs).Conventional methods for scaling dense LLMs have reached significant limitations in training and inference due to computational and memory constraints.MoE addresses these challenges by distributing knowledge representation across specialized expert sub-networks,enabling parameter expansion while maintaining efficiency through sparse expert activation during inference.However,the dynamic nature of expert activation introduces substantial challenges in resource management and scheduling,necessitating targeted optimization at both the system and architectural levels.This survey focuses on the deployment of MoE-based LLMs.It first reviews the definitions and developmental trajectory of MoE,followed by an in-depth analysis of current system-level optimization strategies and architectural innovations tailored to MoE.The paper concludes by summarizing key findings and proposing prospective optimization techniques for MoE-based LLMs. Significance The MoE mechanism offers a promising solution to the computational and memory limitations of dense LLMs.By distributing knowledge representation across specialized expert sub-networks,MoE facilitates model scaling without incurring prohibitive computational costs.This architecture alleviates the bottlenecks associated with training and inference in traditional dense models,marking a notable advance in LLM research.Nonetheless,the dynamic expert activation patterns inherent to MoE introduce new challenges in resource scheduling and management.Overcoming these challenges requires targeted system-and architecture-level optimizations to fully harness the potential of MoE-based LLMs. Progress Recent advancements in MoE-based LLMs have led to the development of various optimization strategies.At the system level,approaches such as automatic parallelism,communication-computation pipelining,and communication operator fusion have been adopted to reduce communication overhead.Memory management has been improved through expert prefetching,caching mechanisms,and queue scheduling policies.To address computational load imbalance,both offline scheduling methods and runtime expert allocation strategies have been proposed,including designs that leverage heterogeneous CPU-GPU architectures.In terms of hardware architecture,innovations include dynamic adaptation to expert activation patterns,techniques to overcome bandwidth limitations,and near-memory computing schemes that improve deployment efficiency.In parallel,the open-source community has developed supporting tools and frameworks that facilitate the practical deployment and optimization of MoE-based models. Conclusions This survey presents a comprehensive review of system and architectural optimization techniques for MoE-based LLMs.It highlights the importance of reconciling parameter scalability with computational efficiency through the MoE framework.The dynamic nature of expert activation poses significant challenges in scheduling and resource management,which are systematically addressed in this survey.By evaluating current optimization techniques across both system and hardware layers,the paper offers key insights into the state of the field.It also proposes directions for future work,providing a reference for researchers and practitioners seeking to improve the performance and scalability of MoE-based models.The findings emphasize the need for continued innovation across algorithm development,system engineering,and architectural design to fully realize the potential of MoE in real-world applications. Prospects Future research on MoE-based LLMs is expected to advance the integration of algorithm design,system optimization,and hardware co-design.Key research directions include resolving load imbalance and maximizing resource utilization through adaptive expert scheduling algorithms,refining system frameworks to support dynamic sparse computation more effectively,and exploring hardware paradigms such as near-memory computing and hierarchical memory architectures.These developments aim to deliver more efficient and scalable MoE model deployments by fostering deeper synergy between software and hardware components.

Data transfer between processors and memory remains a critical bottleneck in improving application performance on traditional computing hardware, particularly for data-intensive workloads such as machine learning, bioinformatics, and security applications. In-memory computing (IMC), a paradigm where a substantial portion of data processing occurs directly within memory, has emerged as a promising solution to mitigate this bottleneck. The advancement of emerging non-volatile memory (NVM) technologies has further accelerated the development of IMC hardware fabrics. However, harnessing the full potential of IMC requires a cross-layer design approach that spans memory technologies, circuits, architectures, and systems. Essential cross-layer tools—including modeling and simulation, data partitioning and mapping, and operation scheduling—play a pivotal role in designing efficient IMC-based hardware. This article reviews key advancements in simulation and design tools for IMC fabrics, with a focus on NVM-based crossbar arrays and content-addressable memories, while highlighting the necessity of cross-layer collaboration. Additionally, we discuss current challenges and emerging opportunities in the field.

Near DRAM Processing (NDP) architectures have emerged to be a promising solution for commercializing in-memory computing and addressing the "memory wall" problem, especially for the memory-intensive machine learning (ML) workloads. In NDP architectures, the Processing Units (PUs) are distributed next to different memory units to exploit the high internal bandwidth. Therefore, in order to fully utilize the bandwidth advantage of NDP architectures for ML applications, meticulous evaluations and optimizations of data placement in DRAM and workload scheduling among different PUs are required. However, existing simulation and compilation tools face two insuperable obstacles to achieving these targets. On the one hand, tools for traditional von Neumann architectures only focus on the data access behaviors between the host and DRAM and treat DRAM as a whole part, which cannot support NDP architectures with multiple independent processing and memory units working simultaneously. On the other hand, existing NDP simulators and compilers are designed for specific DRAM technology and NDP architecture, lacking compatibility for various NDP architectures. In order to overcome these challenges and optimize data mapping and workload scheduling for different NDP architectures, we propose UniNDP, a unified NDP compilation and simulation tool for ML applications. Firstly, we propose a unified tree-based NDP hardware abstraction and the corresponding instruction set, enabling the support for various NDP architectures based on different DRAM technologies. Secondly, we design a cycle-accurate and instruction-driven NDP simulator to evaluate hardware performance by accurately tracking the working status of memory elements and PUs. The accurate simulation can provide effective guidance for compilation. Thirdly, we design an NDP compiler that optimizes data partition, mapping, and workload scheduling in different DRAM hierarchies. Furthermore, to enhance the compilation efficiency, we propose a hardware status-guided search space pruning strategy and a fast performance predictor using DRAM timing parameters. Extensive experimental results show that, compared to existing mapping and compilation methods, UniNDP can achieve 1.05-3.43x speedup across multiple NDP architectures and different ML workloads. Furthermore, based on the results of UniNDP, we provide insights for the future NDP architecture design and deployment in ML applications.

Scaling Diffusion Transformer (DiT) inference via sequence parallelism is critical for reducing latency in visual generation, but is severely hampered by workload imbalance when applied to models employing block-wise sparse attention. The imbalance stems from the inherent variation in sparsity across attention heads and the irregular distribution of dense blocks within the sparse mask, when sequence parallelism is applied along the head dimension (as in Ulysses) or the block dimension (as in Ring Attention). In this paper, we formalize a sparse imbalance ratio to quantify the imbalance, and propose db-SP, a sparsity-aware sequence parallelism technique that tackles the challenge. db-SP contains a dual-level partitioning approach that achieves near-perfect workload balance at both the head and block levels with negligible overhead. Furthermore, to handle the evolving sparsity patterns across denoising steps and layers, db-SP dynamically determines the parallel degrees for the head and block dimensions at runtime. Experimental results demonstrate that db-SP delivers an end-to-end speedup of 1.25x and an attention-specific speedup of 1.40x over state-of-the-art sequence parallel methods on average. Code is available at https://github.com/thu-nics/db-SP.

Hybrid convolutional neural network (CNN) and Transformer networks are emerging in computer vision, combining convolutional, linear, and attention layers to achieve high accuracies with moderate model sizes. Developing the accelerators for hybrid networks is pivotal to simultaneously optimize the static matrix multiplication (MM) in convolutional and linear layers, as well as dynamic MM in attention layers. However, the existing accelerators are primarily designed for either CNNs or Transformers, resulting in increased data movement to support dynamic MM and potential under-utilization of hardware for static MM. To enhance computational performance and energy efficiency for hybrid networks, we propose HyCTor, an accelerator featuring flexible output-stationary (OS) and weight-stationary (WS) dataflows, along with a multicore extension for higher throughput. The parallel array of HyCTor supports interlayer slicing and intralayer splicing to improve the utilization for static MM, and enables seamless switching between OS and WS dataflow to minimize the data movement in dynamic MM. By leveraging structured sparsity in OS dataflow and unstructured sparsity in WS dataflow, the computational efficiency is further boosted for each layer through flexible dataflow selection based on the sparsity ratio. Besides, a novel QuadLoop-mesh topology is proposed to address the complex data dependencies in hybrid networks and minimize data transmission distances in the multicore HyCTor. Experimental results on ResNet-18, ViT-B, and TransIAR-AF show that the proposed single-core HyCTor achieves 1.83x , 1.65x , and 2.41x speedup than state-of-the-art (SOTA) accelerators with 100% utilization rate in most layers, and 3.82x - 38.5x speedup than RTX4090 GPU. The energy efficiency of HyCTor is improved by 1.81x - 8.77x compared with SOTA accelerators. Moreover, the 4-core HyCTor achieves speedups of 3.32x , 2.58x , and 2.91x , while the 16-core HyCTor achieves speedups of 7.05x , 4.05x , and 9.64x compared to 1-core HyCTor on three networks.