Recent years have seen a surge in the popularity of large language models. Such models are typically run in datacenters using high-end GPUs. However, due to latency and privacy concerns, companies have proposed ‘small’ language models (SLMs), designed to run locally on edge devices. We characterize two notable SLMs (i.e., Gemma 3 and Qwen3) on an edge CPU and find that the final layer accounts for up to $31.5 \%$ of the total SLM inference runtime. This final layer maps the output of the last decoder block to a specific token in the model’s vocabulary. Therefore, this layer’s runtime scales with the vocabulary size, which can be very large even for SLMs (e.g., 262k tokens for Gemma 3). Building on this observation, we present HeadSkip, two complimentary techniques to speed-up the final layer. First, we execute the final layer in chunks in order from most frequently to least frequently seen tokens. As we run, we check the logit output for tokens in a chunk and if any exceed a pre-set threshold, we output the token with the largest logit and skip executing the rest of the layer, obtaining a significant speedup. We perform this re-ordering statically based on the frequency of words in English. However, this does not capture words which are generally rare but may appear frequently in some contexts. For such words, a static order may still execute most of the layer until we check this ‘infrequent’ token, diminishing our speedup gains. To avoid this, HeadSkip also caches tokens which appear in the current context. This helps us achieve even greater speed-ups (up to $45 \%$) on modern SLMs, when evaluated on two edge CPUs.

Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality. We test this practice on a 28-quant cohort of Qwen3.6-35B-A3B and a 41-quant cohort of Devstral-Small-2-24B, evaluated across a suite of downstream benchmarks. We find that KLD is strongly correlated with benchmark score over the full cohort (ρ=-0.72 on Qwen and ρ=-0.86 on Devstral, both with p<0.001). However, this relationship collapses to non-significance in the near-baseline silent zone (ρ=+0.00 on Qwen and ρ=-0.24, p=0.36, on Devstral). This collapse persists across 14 measurement variants, including different KLD aggregations, perplexity formulations, top-1 agreement, calibration corpora, and context lengths. At the per-prompt level, KLD has only weak failure-prediction power on code, with failed-vs-passed geometric-mean ratios in [1.08,1.22] across five models on LiveCodeBench, and fails as a cross-model router, achieving only 42.3%-49.4% accuracy on disagreement prompts. We trace the collapse to a structural decomposition: KLD primarily measures the volume of disagreement with the reference, with silent-zone composite ρ=+0.94 (p<0.001) on Qwen and +0.55 (p=0.03) on Devstral, while its relationship to the direction of those disagreements is weak and task-conditional.

Work-skipping methods accelerate neural network training by selectively skipping work that is deemed not to contribute significantly to learning. The goal of such methods is to reduce training time while incurring little or negligible reduction in output accuracy. We identify a “blind-spot” in current best-practice methodologies used to evaluate the effectiveness of work-skipping methods. Current methodologies fail to establish objective ways of determining whether a time reduction vs. accuracy drop trade-off is indeed beneficial. We propose a set of guidelines for evaluating the effectiveness of workload skipping techniques. Our guidelines emphasize the importance of using wall clock time, comparing with random skipping baselines, incorporating early stopping or time-to-accuracy measures, and utilizing Pareto curves. By providing a structured framework, we aim to assist practitioners in accurately determining the true speed advantages of training acceleration algorithms that involve workload skipping. To illustrate the appropriateness of our guidelines we study two work-skipping methods: GSkip, which skips complete layer’s gradient computations and weight updates based on their relative changes, and DeadHorse, which selects data samples for backpropagation according to output confidence. We demonstrate how our methodology can establish when these methods are indeed beneficial. We find that on many occasions, random skipping, early termination, or hyperparameter tuning may be as effective if not more.

Machine Learning (ML) has empowered computing devices to perform tasks traditionally associated with human intelli-gence, such as “thinking,” “seeing,” “hearing,” “reading,” and “writing.” This capability allows systems to interact with the physical world and process information in ways that can rival human abilities, enhancing discovery, learning, and decision-making. Although the foundational techniques of ML have existed for decades, its recent surge in attention is partly due to advancements in computing hardware. Around 2009, computing hardware in the form of graphics processors had reached a level of performance and storage capacity that made implementing core ML methods practical in terms of cost and time. Despite these advances, the performance and storage capacities of current hardware still constrain ML applications. As ML applications are being deployed across the full spectrum of computing devices, from the data center to Internet-of- Things systems, enhancing hardware remains a critical factor for unlocking ML's full potential.

The transfer of tensors from/to memory during neural network training dominates time and energy. To improve energy efficiency and performance, research has been exploring ways to use narrower data representations. So far, these attempts relied on user-directed trial-and-error to achieve convergence. We present methods that relieve users from this responsibility. Our methods dynamically adjust the size and format of the floating-point containers used for activations and weights during training, achieving adaptivity across three dimensions: i) which datatype to use, ii) on which tensor, and iii) how it changes over time. The different meanings and distributions of exponent and mantissas lead us to tailored approaches for each. We present two lossy pairs of methods to eliminate as many mantissa and exponent bits as possible without affecting accuracy. Quantum Mantissa and Quantum Exponent are machine learning compression methods that tap into the gradient descent algorithm to learn the minimal mantissa and exponent bitlengths on a per-layer granularity. They automatically learn that many tensors can use just 1 or 2 mantissa bits and 3 or 4 exponent bits. Overall, the two machine learning methods reduce the footprint by 4.74×. Alternatively, BitWave observes changes in the loss function during training to adjust mantissa and exponent bitlengths network-wide, yielding a 3.19× reduction in footprint. Finally, we present an optional method, Gecko, to exploit the naturally emerging, lop-sided exponent distribution to losslessly compress resulting exponents from Quantum Exponent or BitWave and, on average, improve compression rates to 5.64× and 4.56×.

Spike sorting is the process of parsing electrophysiological signals from neurons to identify if, when, and which particular neurons fire. Spike sorting is a particularly difficult task in computational neuroscience due to the growing scale of recording technologies and complexity in traditional spike sorting algorithms. Previous spike sorters can be divided into software-based and hardware-based solutions. Software solutions are highly accurate but operate on recordings after-the-fact, and often require utilization of high-power GPUs to process in a timely fashion, and they cannot be used in portable applications. Hardware solutions suffer in terms of accuracy due to the simplification of mechanisms for implementation's sake and process only up to 128 inputs. This work answers the question: "How much computation power and memory storage is needed to sort spikes from 1000s of channels to keep up with advances in probe technology?" We analyze the computational and memory requirements for modern software spike sorters to identify their potential bottlenecks - namely in the template memory storage. We architect Marple, a highly optimized hardware pipeline for spike sorting which incorporates a novel mechanism to reduce the template memory storage from 8 - 11x. Marple is scalable, uses a flexible vector-based back-end to perform neuron identification, and a fixed-function front-end to filter the incoming streams into areas of interest. The implementation is projected to use just 79mW in 7nm, when spike sorting 10K channels at peak activity. We further demonstrate, for the first time, a machine learning replacement for the template matching stage.

Diffusion models are emerging models that generate images by iteratively denoising random Gaussian noise using deep neural networks. These models typically exhibit high computational and memory demands, necessitating effective post-training quantization for high-performance inference. Recent works propose low-bitwidth (e.g., 8-bit or 4-bit) quantization for diffusion models, however 4-bit integer quantization typically results in low-quality images. We observe that on several widely used hardware platforms, there is little or no difference in compute capability between floating-point and integer arithmetic operations of the same bitwidth (e.g., 8-bit or 4-bit). Therefore, we propose an effective floating-point quantization method for diffusion models that provides better image quality compared to integer quantization methods. We employ a floating-point quantization method that was effective for other processing tasks, specifically computer vision and natural language tasks, and tailor it for diffusion models by integrating weight rounding learning during the mapping of the full-precision values to the quantized values in the quantization process. We comprehensively study integer and floating-point quantization methods in state-of-the-art diffusion models. Our floating-point quantization method not only generates higher-quality images than that of integer quantization methods, but also shows no noticeable degradation compared to full-precision models (32-bit floating-point), when both weights and activations are quantized to 8-bit floating-point values, while has minimal degradation with 4-bit weights and 8-bit activations. Additionally, we introduce a better methodology to evaluate quantization effects, highlighting shortcomings with existing output quality metrics and experimental methodologies. Finally, as an additional potential benefit, our floating-point quantization method increases model sparsity by an order of magnitude, enabling further optimization opportunities.

Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time and energy benefits on existing hardware designs that support short bitlengths. However, the question of finding the minimum bitlength for a desired accuracy remains open. We introduce a training method for minimizing inference bitlength at any granularity while maintaining accuracy. Namely, we propose a regularizer that penalizes large bitlength representations throughout the architecture and show how it can be modified to minimize other quantifiable criteria, such as number of operations or memory footprint. We demonstrate that our method learns thrifty representations while maintaining accuracy. With ImageNet, the method produces an average per layer bitlength of 4.13, 3.76 and 4.36 bits on AlexNet, ResNet18 and MobileNet V2 respectively, remaining within 2.0%, 0.5% and 0.5% of the base TOP-1 accuracy.

Atalanta is a lossless , hardware/software co-designed compression technique for the tensors of fixed-point quantized deep neural networks. Atalanta increases effective memory capacity, reduces off-die traffic, and/or helps to achieve the desired performance/energy targets while using smaller off-die memories during inference. Atalanta is architected to deliver nearly identical coding efficiency compared to Arithmetic Coding while avoiding its complexity, overhead, and bandwidth limitations. Indicatively, the Atalanta decoder and encoder units each use less than 50B of internal storage. In hardware, Atalanta is implemented as an assist over any machine learning accelerator transparently compressing/decompressing tensors just before the off-die memory controller. This work shows the performance and energy efficiency of Atalanta when implemented in a 65nm technology node. Atalanta reduces data footprint of weights and activations to 60% and 48% respectively on average over a wide set of 8-bit quantized models and complements a wide range of quantization methods. Integrated with a Tensorcore-based accelerator, Atalanta boosts the speedup and energy efficiency to 1.44× and 1.37×, respectively. Atalanta is effective at compressing the stashed activations during training for fixed-point inference.

We introduce a software-hardware co-design approach to reduce memory traffic and footprint during training with BFloat16 or FP32, in order to boost energy efficiency and execution time performance. Our methods dynamically adjust the size and format of the floating-point containers used to store activations and weights during training. The different value distributions lead us to different approaches for exponents and mantissas. Gecko exploits the favourable exponent distribution with a lossless delta encoding approach to reduce the total exponent footprint by up to 58% in comparison to the FP32 baseline. To contend with the noisy mantissa distributions, we present two lossy methods to eliminate as many as possible least significant bits without affecting accuracy. Quantum Mantissa is a machine learning mantissa compression method that taps onto the gradient descent algorithm to learn the minimal mantissa bitlengths on a per-layer granularity, and obtain up to 92% reduction in total mantissa footprint. Alternatively, BitChop observes changes in the loss function during training to adjust mantissa bitlength network-wide, yielding a reduction of 81% in footprint. Schrödinger's FP implements hardware encoders/decoders that, guided by Gecko/Quantum Mantissa or Gecko/BitChop, transparently encode/decode values when transferring to/from off-chip memory, boosting energy efficiency and reducing execution time.

A dual-tap coded-exposure-pixel (CEP) image sensor is presented and validated in two computational imaging applications. The NMOS-only data-memory pixel (DMP) reduces the transistor count yielding a $7{-}\mu \text {m}$ pitch. One frame period can include up to 900 subexposures when operating at 30 frames/s, corresponding to 39 000 coded subexposures/s. The $320\times 320$ -pixel sensor features two readout modes using column-parallel analog-to-digital converters (ADCs). ADC1 is a conventional high-accuracy $\Delta \Sigma $ -modulated ADC that digitizes pixel voltage at the end of every frame period, and ADC2 is a fast energy-efficient comparator that compares the pixel voltage with a constant reference voltage during each subexposure. The outputs of the 12-bit frame-rate ADC1 and the 1-bit subexposure-rate ADC2 are adaptively combined to boost the native dynamic range of the uncoded pixel by over 57 dB, demonstrating over 101-dB dynamic range in intensity imaging. In the second demonstrated application, combined with machine-learned projected illumination patterns, the CEP camera enables single-shot structured-light 3-D imaging at the native resolution and the nominal 30 frames/s video rate.

Deep Neural Network models are becoming much larger which greatly increases their computation and memory requirements. Sparsity offers great opportunities to reduce unnecessary data transfers and computations. However, exploiting sparsity in CNN inference presents challenges such as irregularities in memory access patterns. To overcome this challenge, we propose cuSCNN, an efficient sparse CNN inference engine that leverages the sparsity of both models and activations using optimized sparse-sparse matrix convolution kernels with compressed operands. cuSCNN is motivated by the concepts introduced by the SCNN hardware accelerator[21] but modified appropriately to achieve an efficient software implementation for GPUs. We develop GPU optimizations that boost execution performance and reduce the required memory size and bandwidth. cuSCNN achieves a speedup of up to 171 × compared to an efficient CPU implementation and 30 × speedup compared to a multi-threaded CPU implementation without batching, enabling the use of inexpensive low-end memory-constrained GPUs to implement large networks with near real-time latency. Although GPU throughput can benefit from larger batch sizes, batch size 1 achieves the lowest latency and hence we focus on it.

We introduce a software-hardware co-design approach to reduce memory traffic and footprint during training with BFloat16 or FP32, in order to boost energy efficiency and execution time performance. Our methods dynamically adjust the size and format of the floating-point containers used to store activations and weights during training. The different value distributions lead us to different approaches for exponents and mantissas. Gecko exploits the favourable exponent distribution with a lossless delta encoding approach to reduce the total exponent footprint by up to 58% in comparison to the FP32 baseline. To contend with the noisy mantissa distributions, we present two lossy methods to eliminate as many as possible least significant bits without affecting accuracy. Quantum Mantissa is a machine learning mantissa compression method that taps onto the gradient descent algorithm to learn the minimal mantissa bitlengths on a per-layer granularity, and obtain up to 92% reduction in total mantissa footprint. Alternatively, BitChop observes changes in the loss function during training to adjust mantissa bitlength network-wide, yielding a reduction of 81% in footprint. Schrödinger's FP implements hardware encoders/decoders that, guided by Gecko/Quantum Mantissa or Gecko/BitChop, transparently encode/decode values when transferring to/from off-chip memory, boosting energy efficiency and reducing execution time.

Increasingly larger and better Transformer models keep advancing state-of-the-art accuracy and capability for Natural Language Processing applications. These models demand more computational power, storage, and energy. Mokey reduces the footprint of state-of-the-art 32-bit or 16-bit floating-point transformer models by quantizing all values to 4-bit indexes into dictionaries of representative 16-bit fixed-point centroids. Mokey does not need fine-tuning, an essential feature as often the training resources or datasets are not available to many. Exploiting the range of values that naturally occur in transformer models, Mokey selects centroid values to also fit an exponential curve. This unique feature enables Mokey to replace the bulk of the original multiply-accumulate operations with narrow 3b fixed-point additions resulting in an area- and energy-efficient hardware accelerator design. Over a set of state-of-the-art transformer models, the Mokey accelerator delivers an order of magnitude improvements in energy efficiency over a Tensor Cores-based accelerator while improving performance by at least 4× and as much as 15× depending on the model and on-chip buffering capacity. Optionally, Mokey can be used as memory compression assist for any other accelerator transparently stashing wide floating-point or fixed-point activations or weights into narrow 4-bit indexes. Mokey proves superior to prior state-of-the-art quantization methods for Transformers.

Presents a powerpoint on the topic of a massive-scale brain activity decoding chip.

A dual-tap coded-exposure-pixel (CEP) image sensor is presented and demonstrated in several computational imaging applications. The NMOS-only data-memory pixel (DMP) reduces transistor count in CEP yielding 39,000 subexposures/s at 320x320 sensor resolution with 7 µm pitch. The outputs of a frame-rate 12-bit ADC1 and a 1-bit subexposure-rate ADC2 are adaptively combined to boost the native dynamic range of coded-imaging modalities by over 57dB, demonstrating over 101dB dynamic range in intensity imaging. The CEP camera combined with machine learnt projection patterns enables single-shot structured-light 3D imaging at native resolution and video rate.

We introduce a software-hardware co-design approach to reduce memory traffic and footprint during training with BFloat16 or FP32 boosting energy efficiency and execution time performance. We introduce methods to dynamically adjust the size and format of the floating-point containers used to store activations and weights during training. The different value distributions lead us to different approaches for exponents and mantissas. Gecko exploits the favourable exponent distribution with a loss-less delta encoding approach to reduce the total exponent footprint by up to $58\%$ in comparison to a 32 bit floating point baseline. To content with the noisy mantissa distributions, we present two lossy methods to eliminate as many as possible least significant bits while not affecting accuracy. Quantum Mantissa, is a machine learning-first mantissa compression method that taps on training's gradient descent algorithm to also learn minimal mantissa bitlengths on a per-layer granularity, and obtain up to $92\%$ reduction in total mantissa footprint. Alternatively, BitChop observes changes in the loss function during training to adjust mantissa bit-length network-wide yielding a reduction of $81\%$ in footprint. Schrödinger's FP implements hardware encoders/decoders that guided by Gecko/Quantum Mantissa or Gecko/BitChop transparently encode/decode values when transferring to/from off-chip memory boosting energy efficiency and reducing execution time.

Data accesses between on- and off-chip memories account for a large fraction of overall energy consumption during inference with deep learning networks. We present APack, a simple and effective, lossless, off-chip memory compression technique for fixed-point quantized models. APack reduces data widths by exploiting the non-uniform value distribution in deep learning applications. APack can be used to increase the effective memory capacity, to reduce off-chip traffic, and/or to achieve the desired performance/energy targets while using smaller off-chip memories. APack builds upon arithmetic coding, encoding each value as an arithmetically coded variable length prefix, plus an offset. To maximize compression ratio a heuristic software algorithm partitions the value space into groups each sharing a common prefix. APack exploits memory access parallelism by using several, pipelined encoder/decoder units in parallel and keeps up with the high data bandwidth demands of deep learning. APack can be used with any machine learning accelerator. In the demonstrated configuration, APack is placed just before the off-chip memory controller so that he rest of the on-chip memory and compute units thus see the original data stream. We implemented the APack compressor and decompressor in Verilog and in a 65nm tech node demonstrating its performance and energy efficiency. Indicatively, APack reduces data footprint of weights and activations to 60% and 48% respectively on average over a wide set of 8-bit quantized models. It naturally adapts and compresses models that use even more aggressive quantization methods. When integrated with a Tensorcore-based accelerator, APack boosts the speedup and energy efficiency to 1.44X and 1.37X respectively.

We present FPRaker, a processing element for composing training accelerators. FPRaker processes several floating-point multiply-accumulation operations concurrently and accumulates their result into a higher precision accumulator. FPRaker boosts performance and energy efficiency during training by taking advantage of the values that naturally appear during training. Specifically, it processes the significand of the operands of each multiply-accumulate as a series of signed powers of two. The conversion to this form is done on-the-fly. This exposes ineffectual work that can be skipped: values when encoded have few terms and some of them can be discarded as they would fall outside the range of the accumulator given the limited precision of floating-point. We demonstrate that FPRaker can be used to compose an accelerator for training and that it can improve performance and energy efficiency compared to using conventional floating-point units under ISO-compute area constraints. We also demonstrate that FPRaker delivers additional benefits when training incorporates pruning and quantization. Finally, we show that FPRaker naturally amplifies performance with training methods that use a different precision per layer.