Artificial intelligence systems are becoming more intelligent, but at a very high cost in terms of energy consumption and training requirements. By contrast, our brains only require 20 W of energy, they learn online and they can instantly adjust to changing contingencies. This begs the question what data structures, algorithms and learning methods enable brains to achieve that, and whether these can be ported into artificial devices. We are addressing this question for a core feature of intelligence: the capacity to plan and solve problems, including new problems that involve states that were never encountered before. Here we examine three tools that brains are likely to use for achieving that: cognitive maps, stochastic computing and compositional coding. We integrate these tools into a transparent neural network model, and demonstrate its power for flexible planning and problem-solving. Importantly, this approach is suitable for implementation by in-memory computing and other energy-efficient neuromorphic hardware. In particular, it only requires self-supervised local synaptic plasticity that is suited for on-chip learning. Hence, a core feature of brain intelligence-the capacity to generate solutions to problems that were never encountered before-does not require deep neural networks or large language models, and can be implemented in energy-efficient edge devices.

Adaptive behavior requires the brain to transition between distinct contexts while maintaining representations of prior experience. The ability to reconfigure neural representations without erasing previously acquired knowledge is central to learning in dynamic environments, yet the neural mechanisms that support this balance remain unclear. Understanding these mechanisms is also critical for addressing catastrophic forgetting in artificial systems designed for lifelong learning. Here, we identify joint sparse coding and temporal dynamics in both the mouse medial prefrontal cortex (mPFC) and computational networks as mechanisms that help preserve prior representations during context transitions. Specifically, sparsity in context-dependent representations reduces cross-context interference, whereas temporal dynamics within the network activity further enhance context separability across time. Strikingly, networks endowed with both properties, such as spiking neural networks, exhibit improved retention during lifelong learning without auxiliary heuristics. These findings establish joint sparse coding and temporal dynamics as a core mechanism supporting flexible context reconfiguration in lifelong learning and, through their activity constraining nature, as an energy-efficient architectural principle for stable adaptation. Together, they provide a mechanistic framework for understanding how the brain preserves prior knowledge while flexibly adapting to new contexts.

Spatial data structures (e.g., Kd-trees, R-trees, BVHs) are the fundamental abstraction for organizing geometric data and avoiding linear traversal, which underpin point cloud processing, ray tracing, and collision detection. We target the general problem of efficient spatial data structure search and take the point cloud as the primary case for analysis and evaluation. However, these structures introduce fundamental inefficiencies: the compute bottleneck from recursive searching and the memory bottleneck from irregular access patterns, which existing architectural solutions fail to address effectively. We present RoboCortex, a novel architecture that addresses these challenges through three synergistic designs. First, RoboCortex introduces a near-cache programmable accelerator that not only hardware-accelerates spatial data structure searches but also exposes physical coordinates to the cache hierarchy. It enables cache optimizations based on locality in physical coordinates rather than memory address patterns. Building on this coordinate visibility, RoboCortex further proposes the path buffer, a hardware structure that caches frequent searching paths, to exploit physical locality by bypassing redundant node visits. However, the path buffer may exacerbate memory access irregularity; thus, as a compensatory mechanism, RoboCortex designs a dedicated prefetching strategy to further improve search efficiency. Our experimental results demonstrate that RoboCortex achieves 2.74-13.07× speedup for autonomous driving oriented workloads and 12.73-77.94× improvement for object reconstruction oriented workload over baseline CPU implementations. To our knowledge, this is the first spatial-data-structure-oriented architecture to exploit physical locality without accuracy loss. We not only demonstrate its effectiveness on point cloud workloads, but also its generality to more domains, such as ray tracing in graphics.

Lifelong learning, the ability to acquire and retain knowledge continually, is essential for artificial intelligence operating in dynamic, open-ended environments, yet remains limited by catastrophic forgetting, limited generalization, and high energy costs. These challenges stem from a dual disconnect: between algorithms and hardware, and from the oversimplified adoption of singular biological mechanisms. We present a co-design paradigm that bridges this gap by integrating complementary neurobiological principles into a hardware-native learning system. Our approach unites a hybrid lifelong learning algorithm—fusing Drosophila -inspired dynamic active forgetting with human-derived metaplasticity—with a self-selective memristor (SSM) that intrinsically emulates these synaptic dynamics. The algorithm balances synaptic pruning for adaptability with consolidation for stability, while the SSM hardware physically instantiates this stability-plasticity continuum through its tunable conductance decay and self-rectifying properties, eliminating cross-talk without external selectors. With a 10% read margin, the passive crossbar array integrated with the proposed device achieves terabit-scale storage capacity. Evaluated on 20 sequential CIFAR-variant tasks using hardware-calibrated simulation with per-tile hardware validation, our system projects 3.42× speedup and 91× energy improvement over CMOS accelerators. This work establishes a scalable, hardware-algorithm-integrated foundation for the efficient deployment of lifelong learning in real-world systems.

Robust visual perception in open-world environments remains a fundamental challenge, as data representations degrade under extreme imaging conditions. By contrast, the human visual system maintains resilience through compositional representations built from visual primitives. Here we introduce a brain-inspired two-stage learning framework for adaptive and robust perception in open-world settings. In the bottom-up stage, structured visual priors are embedded into intermediate representations, enabling self-supervised learning of an internal model that fuses complementary cues. In the top-down stage, distilled visual knowledge from this model adaptively modulates downstream perception algorithms to enhance robustness and generalization. We instantiate the framework using the brain-inspired vision sensor Tianmouc and evaluate it on datasets capturing real-world extreme scenarios. Experiments show that bottom-up learning yields consistent generalization across diverse degradations, while top-down modulation substantially improves monocular depth estimation and video instance segmentation under severe interference. These results establish a principled brain-inspired computational approach for advancing artificial visual intelligence and understanding human vision. A brain-inspired two-stage sensing–perception framework couples bottom-up visual priors with top-down adaptive modulation, enabling robust open-world vision under extreme real-world degradations.

Recent advancements in brain-inspired complementary vision chips (CVS) with intensity, multi-bit temporal difference (TD) and spatial difference (SD) sensing capabilities offer a promising solution to the limitations of traditional image sensors by enabling high-speed, high-precision sensing with reduced bandwidth consumption. However, noise characterizations and denoising strategies for these sensors remain underexplored. In this study, leveraging a recently developed state-of-the-art CVS, Tianmouc, we present a theoretical analysis of its noise characteristics, revealing the main challenge for denoising: a distinctive distribution that varies with local illumination. Building on this analysis, we develop a suite of novel denoising algorithms named locally adaptive direction-aware filter (LADF). LADF implements multi-stage denoising algorithms consisting of preprocessing followed by an adaptive threshold filter that adjusts parameters locally to mitigate noise variability. Additionally, considering the distinct characteristics of SD, we develop a multi-directional and polarity-aware separation strategy, while for TD, we exploit the inherent time-space correlation between TD and SD to suppress noise further. To enable rigorous evaluation, we construct a large-scale paired dataset through a novel synthetic-real approach that combines accurately labeled synthetic images with real-world captured data. Experimental results demonstrate that LADF achieves an average signal-to-noise ratio (SNR) of 10.11 in SD, outperforming two baseline methods by factors of 1.54 & times; and 2.73 & times;, respectively, and an average SNR of 4.52 in TD, surpassing the baselines by 1.47 & times; and 3.57 & times;, respectively. Furthermore, our method reduces errors in motion estimation by 28.9%, and enhances the peak SNR in reconstruction by 3.35 dB, demonstrating its effectiveness in downstream tasks. Our algorithm establishes a new benchmark for CVS denoising and demonstrates significant potential to enhance the performance of application systems utilizing CVS.

Brain-inspired vision sensors (BVSs), drawing inspiration from the human visual system, produce sparse, high-temporal-resolution data stream capable of capturing rapid object motion. However, real-time processing of such data while preserving its inherent sparsity presents a critical challenge for practical deployment. A key challenge is to leverage the spatiotemporal correlations among event stream (ES). Overlooking these correlations leads to prohibitively high query costs that even negate the benefits of sparsity while exploiting them introduces complications such as input-output order conflicts and tradeoffs between memory usage and latency. To address this, we present Espresso, an efficient hardware architecture that leverages the spatiotemporal order of events while explicitly preserving sparsity, achieving low-latency stream processing of event data. We first formalize a spatiotemporal order representation that identifies key features for stream processing on sparse events. Building on this, Espresso decouples the output window address from the input event address, resolving order conflicts via a dedicated queue mechanism and minimizing memory overhead through an optimized hash table. This enables immediate window-wise processing with minimal latency. To coordinate the pipeline, we design the Event-Scheduler, a streamlined finite state machine (FSM) that prunes computations on zero values and aligns input-output stream order discrepancies. Integrated together, these modules deliver scalable, high-throughput processing for event-driven vision tasks. Espresso achieves up to 5000 frames/s, offering a 5.1 & times; performance improvement over embedded GPUs. With parallel instantiations, it exceeds over 7000 frames/s in structured scenes and maintains over 2000 frames/s under complex-environment scenarios with minimal hardware overhead. These results establish Espresso as an efficient and scalable solution for real-time event-based vision processing, demonstrating the importance of spatiotemporal ordering in unlocking the full potential of BVSs.

Model-Based Reinforcement Learning yields sample efficiency via latent imagination, yet remains constrained by Historical Tethering: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model's manifold discovery outpaces the policy's sparse-reward optimization. We propose Mind Dreamer (MD), a framework that instantiates Active Causal Intervention to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Expected Free Energy. Instead of initializing from historical data, it draws initial states from an adversarial generator s_0 ∼ p_gen(·), creating non-continuous latent jumps to epistemic blind spots that are physically plausible yet cognitively challenging. We derive Relay Value Function and Relay Uncertainty Function to resolve the credit assignment paradox across these spatial ruptures. Treating synthesized anchors as interventional intermediary states, these potentials propagate pragmatic and epistemic value through Bellman-style backups. Notably, we prove that uncertainty propagation across discontinuities necessitates a quadratic discount γ^2, establishing a formal epistemic horizon. Theoretically, MD approximates a variance-minimizing importance sampler that expands the manifold's spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a 1.67× average speedup over DreamerV3 on DeepMind Control Suite, reaching 8.8× in sparse-reward tasks.

Large Language Models (LLMs) have transformed society, but their computational and energy needs hinder efficient inference. The memory wall, the growing processor-memory speed disparity, remains a critical bottleneck for LLM. While Process-in-Memory (PIM) architectures address this challenge by co-locating computation with memory, achieving $\mathbf{5}-\mathbf{2 0} \times$ higher bandwidth than GPUs, existing scalable PIM solutions face critical trade-offs in flexibility, capacity, and efficiency when handling LLMs' dynamic memory-compute patterns and operator diversity. DRAM-PIM suffers from inter-bank communication overhead despite its vector parallelism. SRAM-PIM offers sub10ns latency for matrix operation but is constrained by limited capacity. This work introduces CompAir, a scalable PIM architecture that integrates DRAM-PIM and SRAM-PIM with hybrid bonding, enabling efficient linear computations while unlocking multi-granularity data pathways. We further develop CompAirNoC, an advanced network-on-chip (NoC) with an embedded arithmetic logic unit that performs non-linear operations during data movement. Such a design offloads the centralized communication bottleneck in the channel level to distributed banks, simultaneously reducing communication overhead and area cost for scalability. Finally, we develop a hierarchical Instruction Set Architecture that ensures both flexibility and programmability of the hybrid PIM. Experiments show CompAir delivers 1.83-7.98× faster prefill and $\mathbf{1. 9 5 - 6. 2 8} \times$ faster decoding versus state-ofthe-art PIM designs, with $\mathbf{3. 5 2} \times$ lower energy than GPU-PIM hybrids. This work presents the first systematic exploration of hybrid DRAM-PIM and SRAM-PIM architectures with innetwork computation, paving the way towards a scalable PIM system for LLM inference.

Motion blur arises when rapid scene changes occur during the exposure period, collapsing rich intra-exposure motion into a single RGB frame. Without explicit structural or temporal cues, RGB-only deblurring is highly ill-posed and often fails under extreme motion. Inspired by the human visual system, brain-inspired vision sensors introduce temporally dense information to alleviate this problem. However, event cameras still suffer from event rate saturation under rapid motion, while the event modality entangles edge features and motion cues, which limits their effectiveness. As a recent breakthrough, the complementary vision sensor (CVS), Tianmouc, captures synchronized RGB frames together with high-frame-rate, multi-bit spatial difference (SD, encoding structural edges) and temporal difference (TD, encoding motion cues) data within a single RGB exposure, offering a promising solution for RGB deblurring under extreme dynamic scenes. To fully leverage these complementary modalities, we propose Spatio-Temporal Difference Guided Deblur Net (STGDNet), which adopts a recurrent multi-branch architecture that iteratively encodes and fuses SD and TD sequences to restore structure and color details lost in blurry RGB inputs. Our method outperforms current RGB or event-based approaches in both synthetic CVS dataset and real-world evaluations. Moreover, STGDNet exhibits strong generalization capability across over 100 extreme real-world scenarios. Project page: https://tmcDeblur.github.io/

The rapid advancement of Large Language Models (LLMs) has revolutionized various aspects of human life, yet their immense computational and energy demands pose significant challenges for efficient inference. The memory wall, the growing processor-memory speed disparity, remains a critical bottleneck for LLM. Process-In-Memory (PIM) architectures overcome limitations by co-locating compute units with memory, leveraging 5-20× higher internal bandwidth and enabling greater energy efficiency than GPUs. However, existing PIMs struggle to balance flexibility, performance, and cost-efficiency for LLMs' dynamic memory-compute patterns and operator diversity. DRAM-PIM suffers from inter-bank communication overhead despite its vector parallelism. SRAM-PIM offers sub-10ns latency for matrix operation but is constrained by limited capacity. This work introduces CompAir, a novel PIM architecture that integrates DRAM-PIM and SRAM-PIM with hybrid bonding, enabling efficient linear computations while unlocking multi-granularity data pathways. We further develop CompAir-NoC, an advanced network-on-chip with an embedded arithmetic logic unit that performs non-linear operations during data movement, simultaneously reducing communication overhead and area cost. Finally, we develop a hierarchical Instruction Set Architecture that ensures both flexibility and programmability of the hybrid PIM. Experimental results demonstrate that CompAir achieves 1.83-7.98× prefill and 1.95-6.28× decode improvement over the current state-of-the-art fully PIM architecture. Compared to the hybrid A100 and HBM-PIM system, CompAir achieves 3.52× energy consumption reduction with comparable throughput. This work represents the first systematic exploration of hybrid DRAM-PIM and SRAM-PIM architectures with in-network computation capabilities, offering a high-efficiency solution for LLM.

With the rapid development of artificial intelligence (AI) applications, a new type of AI accelerators, known as the Brain-inspired Chip (BIC), has emerged in Nonvon Neumann computing environments, such as SpiNNaker, Loihi, TrueNorth and Tianjic. However, contemporary BICs often exhibit low resource utilization due to the imbalance in hardware resource allocation across various models, which is exacerbated by the lack of virtualization support. Although prior efforts have explored (para-) virtualization techniques to share resources of other AI accelerators such as GPU, these attempts have failed to account for key characteristics of BIC such as interconnection between multiple BIC cores and scratchpad-centric memory, leading to suboptimal performance. This paper presents vBIC, a comprehensive BIC virtualization solution featuring two key architectural extensions: (1) BIC memory virtualization which reduces TLB misses and table walking times for scratchpad-centric memory access; and (2) BIC interconnection virtualization which establishes a virtual topology among multiple BIC cores. We have implemented a prototype of vBIC on an FPGA platform. Evaluation results show that vBIC maintains end-to-end performance across various AI workloads. Furthermore, when compared with alternative approaches such as unified virtual memory, vBIC achieves a 2 x performance improvement in transformer-based models.

Training large language models (LLMs) has become increasingly expensive due to the rapid expansion in model size. Pipeline parallelism is a widely used distributed training technique. However, as LLMs with larger context become prevalent and memory optimization techniques advance, traditional PP methods encounter greater communication challenges due to the increased size of activations and gradients of activations. To address this issue, we introduce weight-pipeline parallelism (WeiPipe) that transitions from an activation-passing pipeline to a weight-passing pipeline. WeiPipe reduces communication costs and achieves a more balanced utilization by transmitting only weights and their gradients between workers in a pipeline manner. WeiPipe does not rely on collective communication primitives, thus ensuring scalability. We present four variations of WeiPipe parallelism, including WeiPipe-Interleave, which emphasizes communication efficiency, and WeiPipe-zero-bubble, discussing the potential for minimal bubble ratios. Our implementation of WeiPipe-Interleave, performed on up to 32 GPUs and tested in various model configurations, including large-context LLM training, demonstrates a significant improvement in throughput compared to state-of-the-art pipeline parallelism and fully sharded data parallelism with different underlying infrastructures, including NVLink connections within cluster with Ethernet among cluster, and PCIe within cluster and Ethernet among cluster. Additionally, WeiPipe also shows greater scalability in communication-constrained scenarios compared to state-of-art strategies.

Recording and reconstructing high-speed scenes poses a significant challenge. While high-speed cameras can capture fine temporal details, their extremely high bandwidth demands make continuous recording unsustainable. Conversely, traditional RGB cameras, typically operating at 30 FPS, rely on frame interpolation to synthesize high-speed motion, often introducing artifacts and motion blur. Human visual system inspired sensors, like event cameras, offer high-speed sparse temporal or spatial variation data, partially alleviating these issues. However, existing methods still suffer from RGB blur, temporal aliasing, and loss of event information. To overcome these challenges, we leverage a novel complementary vision sensor, Tianmouc, which outputs high-speed, multi-bit, sparse spatio-temporal difference information with RGB frames. Building on this unique sensing modality, we introduce a Cascaded Bi-directional Recurrent Diffusion Model (CBRDM) that achieves accurate, sharp, color-rich video frames reconstruction. Our method outperforms state-of-the-art RGB interpolation algorithms in quantitative evaluations and surpasses eventbased methods in real-world comparisons. Code and dataset are at https://github.com/Tianmouc/GenRec.

Visual Odometry (VO) estimates the pose and motion trajectory of the camera based on visual input, serving as a fundamental technique for robotic positioning and navigation. However, existing VO methods face challenges in visual degradation in extreme environments, e.g., high dynamic range or fast-motion conditions. Although event-based sensing schemes offer partial solutions to this problem, they are limited by unstable features and noise. Recently, a novel brain-inspired vision sensor, Tianmouc, has been reported, incorporating two complementary pathways: a cognition-oriented pathway (COP) for precise color intensity and an action-oriented pathway (AOP) for fast spatiotemporal sensing, considered a promising visual input for VO tasks. Here, we develop Complementary Pathway Spatial Enhanced Visual Odometry (CSVO) to cope with extreme scenarios by fusing the COP and AOP information of Tianmouc. To leverage the dynamic range expansion brought about by dual-pathway fusion, as well as the low-latency spatial difference data in AOP to address high-speed motion, we design an asynchronous dual-pathway feature encoder considering synchronous multimodal fusion and asynchronous cross-modal feature matching. To train and evaluate CSVO, we transform two conventional VO datasets, TartanAir and Apollo, to Tianmouc modality through simulation and collect a real- world Tianmouc-VO dataset in challenging scenes. Our results demonstrate state-of-the-art performance over existing methods on these datasets. Our work sheds light on the generalizability of agents working in extreme scenarios. The codes and data sets are available at https://github.com/Tianmouc/CSVO.

Shortcut learning poses a significant challenge to both the interpretability and robustness of artificial intelligence, arising from dataset biases that lead models to exploit unintended correlations, or shortcuts, which undermine performance evaluations. Addressing these inherent biases is particularly difficult due to the complex, high-dimensional nature of data. Here, we introduce shortcut hull learning, a diagnostic paradigm that unifies shortcut representations in probability space and utilizes diverse models with different inductive biases to efficiently learn and identify shortcuts. This paradigm establishes a comprehensive, shortcut-free evaluation framework, validated by developing a shortcut-free topological dataset to assess deep neural networks' global capabilities, enabling a shift from Minsky and Papert's representational analysis to an empirical investigation of learning capacity. Unexpectedly, our experimental results suggest that under this framework, convolutional models-typically considered weak in global capabilities-outperform transformer-based models, challenging prevailing beliefs. By enabling robust and bias-free evaluation, our framework uncovers the true model capabilities beyond architectural preferences, offering a foundation for advancing AI interpretability and reliability.

Given the escalating intricacy and multifaceted nature of contemporary social systems, manually generating solutions to address pertinent social issues has become a formidable task. In response to this challenge, the rapid development of artificial intelligence has spurred the exploration of computational methodologies aimed at automatically generating solutions. However, current methods for the auto-generation of solutions mainly concentrate on local social regulations that pertain to specific scenarios. Here, we report an automatic social operating system (ASOS) designed for general social solution generation built upon agent-based models that enables both global and local analyses and regulations of social problems across spatial and temporal dimensions. ASOS adopts a hypergraph with extensible social semantics for a comprehensive and structured representation of social dynamics. It also incorporates a generalized protocol for standardized hypergraph operations and a symbolic hybrid framework that delivers interpretable solutions, yielding a balance between regulatory efficacy and functional viability. To demonstrate the effectiveness of the ASOS, we apply it to the domain of averting extreme events within international oil futures markets. By generating a new trading role supplemented by new mechanisms, ASOS can adeptly discern precarious market conditions and make front-running interventions for nonprofit purposes. This study demonstrated that ASOS provides an efficient and systematic approach for generating solutions for enhancing our society.

This paper presents a heterogeneous graph attention network for predicting routing congestion on the netlists, crucial for chip design. It features a two-level attention mechanism to account for netlist structure and design specifications, which previous studies overlooked. Our approach achieves faster and more accurate congestion prediction, with up to 19% increase in accuracy and up to 5.22x speedup. It is the first work that integrates design specifications, enabling a comprehensive prediction of congestion.

Routing congestion is a critical concern that detrimentally impacts chip performance, necessitating substantial time investment in detection and alleviation during the chip design process. Predicting congestion on the netlist significantly reduces time consumption by obtaining routing feedback immediately after design completion without going through subsequent steps. However, the constraints on chip production imposed by design specifications significantly influence routing, and it is ignored by previous studies, resulting in imprecise predictions and sub-optimal optimization. In this study, we design a heterogeneous graph introducing the influence of design specifications to aid netlist congestion prediction. Subsequently, we develop a heterogeneous graph attention network that uses a two-level attention mechanism to capture the effects of netlist structure and design specifications. Experimental results demonstrate our method achieves fast and accurate congestion prediction with up to 19% accuracy improvement and 5.22× prediction speedup. To the best of our knowledge, it is the first work that integrates design specifications, enabling a comprehensive prediction of congestion arising from both the netlist structure and the design specifications.