On-device extended-reality (XR) eye tracking requires accurate pupil localization and low-latency updates under constrained computation, memory, and power budgets. Hybrid event–frame sensing provides complementary semantic and temporal information, but exploiting it efficiently requires an explicit runtime distinction between infrequent semantic recovery and frequent local tracking. This paper presents HBTXR, a runtime-aware hybrid event–frame eye-tracking system based on algorithm–hardware co-design. Within a unified geometric state space, frame-guided Search establishes or refreshes an anchor, and event-guided Track predicts anchor-relative pupilstate updates. The time-synchronous and geometry-stable super-vision method aligns the training targets of the two modes, and a modality-aware shared vision transformer with a reliability-driven scheduler selects the active path at runtime. The resulting asymmetric workload is mapped onto a ZCU104 CPU–FPGA platform through a mode-asymmetric cyclic-streaming architecture with mixed-precision post-training quantization and LUT-based nonlinear approximation. The proposed method achieves a mean pupil-center error of 1.158 px, with Search and Track accounting for 10.88% and 89.12% of the measured invocations and a workload-average computation of 137.57MOP per invocation. Board-level evaluation with pre-staged inputs achieves a measured system inference latency of 1.025 ms at an estimated platform power of 4.648W, showing that explicit Search/Track asymmetry yields a balanced accuracy–computation–latency operating point for programmable hybrid eye tracking.
Ultra-low-rate image compression is essentially a problem of visual description: when only a few bits are available, a codec should preserve global layout, object structure, and semantic organization before refining local texture. Inspired by hierarchical visual representations, we formulate progressive compression as learning compact descriptions that can be successively transmitted and completed. Visual Autoregressive (VAR) models offer a natural basis for this paradigm, as their multi-scale discrete token hierarchy evolves from coarse structures to fine details and enables next-scale completion from received prefixes. However, existing VAR models are mainly optimized for full-scale image generation, leaving early prefixes insufficiently meaningful, semantic, and reliable under constrained bitrate budgets. To address this mismatch, we propose a hierarchical visual description learning framework for ultra-low-rate progressive compression. We introduce Bitrate-Adaptive Truncation (BAT) to encourage early scales to encode critical visual content, Multi-scale Semantic Distillation (MSD) to enhance scale-wise semantic representation with emphasis on coarse tokens, and Prefix-Boundary Bitwise Self-Correction (PBSC) to improve decoding robustness by perturbing the first predicted boundary scale while keeping transmitted prefixes clean. These components align training with practical progressive decoding under compressed-prefix conditions. Extensive experiments show that our method achieves superior perceptual fidelity and more efficient bitrate utilization, especially in ultra-low-bitrate regimes. On DIV2K, it achieves up to 60.0% BD-rate saving compared with existing anchor methods. LVLM-based evaluation further confirms that our reconstructions preserve more task-relevant semantic information at ultra-low bitrates. The code is available at https: //github.com/NJUVISION/Hierarchical-Visual-Compression.
Transformer-based models achieve strong performance on blind compressed image restoration (CIR) under unknown compression settings, but their densely stacked projection operators make deployment costly. We study training-free acceleration for such models through an empirical, operator-level analysis. Specifically, we profile where computation is concentrated, characterize channel-wise activation distributions under compressed inputs, and examine the singular-value decay of representative projection weights. These measurements identify the recurring projections in the attention, feed-forward, convolutional, and prompt-conditioning pathways as effective low-rank replacement targets that preserve the original inference graph. They further establish truncated SVD (TSVD) as a principled training-free baseline for blind CIR and reveal why a uniform low-rank objective is mismatched to the channel-imbalanced feature distributions of compressed images. Accordingly, we formulate Activation-Calibrated Singular Value Decomposition (ASVD) for this restoration setting, deriving low-rank factors entirely offline from pretrained weights and lightweight activation statistics gathered from only a few compressed calibration images. By weighting the decomposition with channel-wise activation magnitudes, our ASVD assigns larger penalties to errors on more active channels, requiring neither retraining nor online calibration and leaving the restoration pipeline intact. When instantiated on PromptCIR at rank 64, ASVD consistently improves PSNR-B over TSVD across four standard benchmarks under identical parameter and MAC budgets. At this operating point, the factorized model cuts parameters by 59
Microscaling (MX) is an OCP-standardized dynamic quantization format that reduces the computational and memory costs of deep learning by sharing an 8-bit exponent across multiple operands. Existing MX formats expose a practical trade-off: MXINT provides high mantissa precision for direct-cast inference, whereas MXFP provides local exponent bits that are important for preserving small gradients during training. Moreover, conventional MX formats use a symmetric block representation, which can waste representational capacity when activations or weights exhibit sign-dependent scale imbalance. This paper proposes MX-SAFE+, an efficient asymmetric microscaling format that combines adaptive per-element bit allocation (MXSF+) with sign-aware shared exponent encoding (EAMX). The proposed MXSF+ format switches among three numerical modes, E2M5, E2M3, E3M0, within a block to reduce quantization error near the shared exponent while extending dynamic range for small values. In addition, the proposed EAMX encoding packs positive and negative shared scales into the original 8-bit scale budget, enabling asymmetric scaling without increasing the MX block footprint. To support the format efficiently, we design an MXSF+- aware systolic tensor array and an EAMX-compatible MAC datapath that replace costly pre-accumulation FP-based scale multiplications with lightweight shift operations. Overall, the proposed EAMXSF+ format improves vision task accuracy by 2.77%/7.94% and 22.3%/3.83%, on average, for inference/full-training compared to MXINT8 and MXFP8 E4M3, respectively, while keeping the accuracy drop against BF16 below 0.6% even in complex language and multimodal tasks. Moreover, with the EAMXSF+ training-inference accelerator, it achieves similar accuracy to the BF16 accelerator while using 23.5/11.0% less total energy consumption with vision/LLM training task.
Floating-point compute-in-memory (FP-CIM) has gained prominence as a critical technology for mitigating the "memory wall" in next-generation Extended Reality (XR). Its appeal lies in its ability to handle the high-dynamic-range data essential for real-time three-dimensional reconstruction and foveated rendering. However, the multi-stage complexity of floating-point arithmetic poses significant challenges for circuit overhead and precision-energy trade-offs within memory arrays. This paper provides a systematic review of FP-CIM from a dataflow-centric perspective, deconstructing the processing pipeline into five essential operations: exponent addition, mantissa multiplication, maximum value extraction, shifting, and accumulation. For each stage, we analyze state-of-the-art circuit implementations and fundamental design trade-offs. Furthermore, the paper identifies critical hurdles for large-scale deployment and explores emerging trends, including hybrid memory hierarchies, post-alignment architectures for high-precision accumulation, and machine-learning-assisted Electronic Design Automation toolchains. This review serves as a comprehensive guide for optimizing FP-CIM macros, establishing a hardware foundation for the low-latency, energy-efficient immersive experiences required by future XR devices.
