Reservoir computing advances the intriguing idea that a nonlinear recurrent neural circuit-the reservoir-can encode spatio-temporal input signals to enable efficient ways to perform tasks like classification or regression. However, recently the idea of a monolithic reservoir network that simultaneously buffers input signals and expands them into nonlinear features has been challenged. A representation scheme in which memory buffer and expansion into higher-order polynomial features can be configured separately has been shown to significantly outperform traditional reservoir computing in prediction of multivariate time-series. Here we propose a configurable neuromorphic representation scheme that provides competitive performance on prediction, but with significantly better scaling properties than directly materializing higher-order features as in prior work. Our approach combines the use of randomized representations from traditional reservoir computing with mathematical principles for approximating polynomial kernels via such representations. While the memory buffer can be realized with standard reservoir networks, computing higher-order features requires networks of 'Sigma-Pi' neurons, i.e., neurons that enable both summation as well as multiplication of inputs. Finally, we provide an implementation of the memory buffer and Sigma-Pi networks on Loihi 2, an existing neuromorphic hardware platform.

There is a growing interest in the early prediction of outcomes in ongoing business processes. Predictive process monitoring distills knowledge from the sequence of event data generated and stored during the execution of processes and trains models on this knowledge to predict outcomes of ongoing processes. However, most state-of-the-art methods require the training of complex and inefficient machine learning models and hyper-parameter optimization as well as numerous input data to achieve high performance. In this paper, we present a novel approach based on Hyperdimensional Computing (HDC) for predicting the outcome of ongoing processes before their completion. We highlight its simplicity, efficiency, and high performance while utilizing only a subset of the input data, which helps in achieving a lower memory demand and faster and more effective corrective measures. We evaluate our proposed method on four publicly available datasets with a total of 12 binary prediction tasks. Our proposed method achieves an average 6% higher area under the ROC curve (AUC) and up to a 14% higher F1-score, while yielding a 20× earlier prediction than state-of-the-art conventional machine learning- and neural network-based models.

On-device learning has emerged as a prevailing trend that avoids the slow response time and costly communication of cloud-based learning. The ability to learn continuously and indefinitely in a changing environment, and with resource constraints, is critical for real sensor deployments. However, existing designs are inadequate for practical scenarios with (i) streaming data input, (ii) lack of supervision and (iii) limited on-board resources. In this paper, we design and deploy the first on-device lifelong learning system called LifeHD for general IoT applications with limited supervision. LifeHD is designed based on a novel neurally-inspired and lightweight learning paradigm called Hyperdimensional Computing (HDC). We utilize a two-tier associative memory organization to intelligently store and manage high-dimensional, low-precision vectors, which represent the historical patterns as cluster centroids. We additionally propose two variants of LifeHD to cope with scarce labeled inputs and power constraints. We implement LifeHD on off-the-shelf edge platforms and perform extensive evaluations across three scenarios. Our measurements show that LifeHD improves the unsupervised clustering accuracy by up to 74.8% compared to the state-of-the-art NN-based unsupervised lifelong learning baselines with as much as 34.3x better energy efficiency. Our code is available at https://github.com/Orienfish/LifeHD.

Recent years have seen growing interest in leveraging deep learning models for monitoring epilepsy patients based on electroencephalographic (EEG) signals. However, these approaches often exhibit poor generalization when applied outside of the setting in which training data was collected. Furthermore, manual labeling of EEG signals is a time-consuming process requiring expert analysis, making fine-tuning patient-specific models to new settings a costly proposition. In this work, we propose the Maximum-Mean-Discrepancy Decoder (M2D2) for automatic temporal localization and labeling of seizures in long EEG recordings to assist medical experts. We show that M2D2 achieves 76.0% and 70.4% of F1-score for temporal localization when evaluated on EEG data gathered in a different clinical setting than the training data. The results demonstrate that M2D2 yields substantially higher generalization performance than other state-of-the-art deep learning-based approaches.

Deep learning-based recommendation systems (e.g., DLRMs) are widely used AI models to provide high-quality personalized recommendations. Training data used for modern recommendation systems commonly includes categorical features taking on tens-of-millions of possible distinct values. These categorical tokens are typically assigned learned vector representations, that are stored in large embedding tables, on the order of 100s of GB. Storing and accessing these tables represent a substantial burden in commercial deployments. Our work proposes MEM-REC, a novel alternative representation approach for embedding tables. MEM-REC leverages bloom filters and hashing methods to encode categorical features using two cache-friendly embedding tables. The first table (token embedding) contains raw embeddings (i.e. learned vector representation), and the second table (weight embedding), which is much smaller, contains weights to scale these raw embeddings to provide better discriminative capability to each data point. We provide a detailed architecture, design and analysis of MEM-REC addressing trade-offs in accuracy and computation requirements, in comparison with state-of-the-art techniques. We show that MEM-REC can not only maintain the recommendation quality and significantly reduce the memory footprint for commercial scale recommendation models but can also improve the embedding latency. In particular, based on our results, MEM-REC compresses the MLPerf CriteoTB benchmark DLRM model size by 2900x and performs up to 3.4x faster embeddings while achieving the same AUC as that of the full uncompressed model.

Today’s machine learning (ML) systems, running workloads, such as deep neural networks, which require billions of parameters and many hours to train a model, consume a significant amount of energy. Due to the complexity of computation and topology, even the quantized models are hard to deploy on edge devices under energy constraints. To combat this, researchers have been focusing on new emerging neuromorphic computing models. Two of those models are hyperdimensional computing (HDC) and spiking neural networks (SNNs), both with their own benefits. HDC has various desirable properties that other ML algorithms lack, such as robustness to noise, simple operations, and high parallelism. SNNs are able to process event-based signal data in an efficient manner. This work develops $\mathsf {HyperSpike}$ , which utilizes a single, randomly initialized, and untrained SNN layer as a feature extractor connected to a trained HDC classifier. HDC is used to enable more efficient classification as well as provide robustness to errors. We experimentally show that $\mathsf {HyperSpike}$ is on average $31.5\times $ more robust to errors than traditional SNNs. On Intel’s Loihi (Davies et al., 2018), $\mathsf {HyperSpike}$ is $10\times $ faster and $2.6\times $ more energy efficient over traditional SNN networks. We further develop $\mathsf {HyperSpikeASIC}$ , a customized accelerator for $\mathsf {HyperSpike}$ . By decoupling the neuron and synapses, $\mathsf {HyperSpikeASIC}$ skips the inactive neurons and limits the neuron state updating to once per time step at most. $\mathsf {HyperSpikeASIC}$ is $601\times $ faster and $3467\times $ more energy efficient than $\mathsf {HyperSpike}$ running on Intel’s Loihi for SNN acceleration, and $12.2\times $ faster and $211\times $ more energy efficient than the state-of-the-art SNN ASIC implementation (Wang et al., 2022).

Today's Machine Learning(ML) systems, especially those running in server farms running workloads such as Deep Neural Networks, which require billions of parameters and many hours to train a model, consume a significant amount of energy. To combat this, researchers have been focusing on new emerging neuromorphic computing models. Two of those models are Hyperdimensional Computing (HDC) and Spiking Neural Networks (SNNs), both with their own benefits. HDC has various desirable properties that other Machine Learning (ML) algorithms lack such as: robustness to noise in the system, simple operations, and high parallelism. SNNs are able to process event based signal data in an efficient manner. In this paper, we create HyperSpike, which utilizes a single, randomly initialized and untrained SNN layer as feature extractor connected to a trained HDC classifier. HDC is used to enable more efficient classification as well as provide robustness to errors. We experimentally show that HyperSpike is on average 31.5× more robust to errors than traditional SNNs. We also implement HyperSpike in hardware, and show that it is 10x faster and 2.6× more energy efficient over traditional SNN networks run on Intel's Loihi [1].

Hyperdimensional computing (HDC) is a paradigm for data representation and learning originating in computational neuroscience. HDC represents data as high-dimensional, low-precision vectors which can be used for a variety of information processing tasks like learning or recall. The mapping to high-dimensional space is a fundamental problem in HDC, and existing methods encounter scalability issues when the input data itself is high-dimensional. In this work, we explore a family of streaming encoding techniques based on hashing. We show formally that these methods enjoy comparable guarantees on performance for learning applications while being substantially more efficient than existing alternatives. We validate these results experimentally on a popular high-dimensional classification problem and show that our approach easily scales to very large data sets.

We apply deep learning to daytime satellite imagery to predict changes in income and population at high spatial resolution in US data. For grid cells with lateral dimensions of 1.2 km and 2.4 km (where the average US county has dimension of 51.9 km), our model predictions achieve R2 values of 0.85 to 0.91 in levels, which far exceed the accuracy of existing models, and 0.32 to 0.46 in decadal changes, which have no counterpart in the literature and are 3–4 times larger than for commonly used nighttime lights. Our network has wide application for analyzing localized shocks. (JEL C45, R11, R23)

Advances in graph neural network (GNN)-based algorithms enable machine learning on relational data. GNNs are computationally demanding since they rely upon backpropagation over the graph data that has sparse and irregular characteristics. In this paper, we propose a lightweight graph-based machine learning framework based on hyperdimensional computing (HDC) called RelHD. It maps the features of each node into a high-dimensional space and embeds relationships between nodes. Using lightweight HDC operations, RelHD enables both training and inference on graph data without backpropagation. Furthermore, we design a scalable processing in-memory (PIM) architecture based on the emerging FeFET technology to accelerate the proposed algorithm. Our strategy optimizes data allocation and operation scheduling that maximizes the accelerator performance by addressing the sparseness and irregularity of the graph. Experimental results show that RelHD offers comparable accuracy to the popular GNN-based algorithms while being up to 32× faster on GPU. Also, our FeFET-based accelerator achieves 33× of speedup and 59287× energy efficiency improvement on average over the GPU. It is 10× faster and 986× more energy efficient on average compared to the state-of-the-art in-memory processing-based GNN accelerator.

Hyperdimensional (HD) computing is a set of neurally inspired methods for computing on high-dimensional, low-precision, distributed representations of data. These representations can be combined with simple, neurally plausible algorithms to effect a variety of information processing tasks. HD computing has recently garnered significant interest from the computer hardware community as an energy-efficient, low-latency, and noise-robust tool for solving learning problems. We present a novel mathematical framework that unifies analysis of HD computing architectures, and provides general, non-asymptotic, sufficient conditions under which HD information processing techniques will succeed.

Hyperdimensional (HD) computing is a set of neurally inspired methods for obtaining highdimensional, low-precision, distributed representations of data. These representations can be combined with simple, neurally plausible algorithms to effect a variety of information processing tasks. HD computing has recently garnered significant interest from the computer hardware community as an energy-efficient, low-latency, and noise-robust tool for solving learning problems. In this review, we present a unified treatment of the theoretical foundations of HD computing with a focus on the suitability of representations for learning.

Hyperdimensional (HD) computing is an emerging paradigm for machine learning based on biologically plausible models of memory. HD computing represents data as random points with i.i.d. components in a high-dimensional, low-precision, space. These points are then used as input to learning algorithms. The i.i.d. and low-precision nature of HD representations accord naturally with highly parallel hardware like FPGAs and PIM architectures and HD computing has recently gained significant popularity in the hardware community [1]–[3]. Existing work on HD computing has focused primarily on learning linear functions over the encodings. However, much data of practical interest is nonlinear, meaning that linear methods may fail to capture important structure in the data. Our goal is to assess the utility of using nonlinear learning algorithms on HD representations of data. A complication arises because nonlinear models are generally more complex computationally than simple linear methods which are traditionally used with HD computing. In this work, we show how nonlinear learning on HD representations can be formulated as a sparse convex optimization problem using the Boolean Fourier transform. We additionally present results from a preliminary empirical study motivated by this formulation designed to assess the possible practical benefits of this approach.

Hyperdimensional (HD) computing is a set of neurally inspired methods for obtaining high-dimensional, low-precision, distributed representations of data. These representations can be combined with simple, neurally plausible algorithms to effect a variety of information processing tasks. HD computing has recently garnered significant interest from the computer hardware community as an energy-efficient, low-latency, and noise-robust tool for solving learning problems. In this review, we present a unified treatment of the theoretical foundations of HD computing with a focus on the suitability of representations for learning.

Hyperdimensional computing (HD) is an emerging paradigm for machine learning based on the evidence that the brain computes on high-dimensional, distributed, representations of data. The main operation of HD is encoding, which transfers the input data to hyperspace by mapping each input feature to a hypervector, followed by a bundling procedure that adds up the hypervectors to realize the encoding hypervector. The operations of HD are simple and highly parallelizable, but the large number of operations hampers the efficiency of HD in embedded domain. In this paper, we propose SHEARer, an algorithmhardware co-optimization to improve the performance and energy consumption of HD computing. We gain insight from a prudent scheme of approximating the hypervectors that, thanks to error resiliency of HD, has minimal impact on accuracy while provides high prospect for hardware optimization. Unlike previous works that generate the encoding hypervectors in full precision and then and then perform ex-post quantization, we compute the encoding hypervectors in an approximate manner that saves resources yet affords high accuracy. We also propose a novel FPGA architecture that achieves striking performance through massive parallelism with low power consumption. Moreover, we develop a software framework that enables training HD models by emulating the proposed approximate encodings. The FPGA implementation of SHEARer achieves an average throughput boost of 104,904× (15.7×) and energy savings of up to 56,044× (301×) compared to state-of-the-art encoding methods implemented on Raspberry Pi 3 (GeForce GTX 1080 Ti) using practical machine learning datasets.

Recent years have seen a growing interest in the development of non-invasive devices capable of detecting seizures which can be worn in everyday life. Such devices must be lightweight and unobtrusive which severely limit their on-board computing power and battery life. In this paper, we propose a novel technique based on hyperdimensional (HD) computing to detect epileptic seizures from 2-channel surface EEG recordings. The proposed technique eliminates the need for complicated feature extraction techniques required in conventional ML algorithms. The HD algorithm is also simple to implement and does not require expert knowledge for architectural optimizations needed for approaches based on neural networks. In addition, our proposed technique is light-weight and meets the computation and memory constraints of ultra-small devices. Experimental results on a publicly available dataset indicates our approach improves the accuracy compared to state-of-the-art techniques while consuming smaller or comparable power.

Hyperdimensional computing (HD) is an emerging paradigm for machine learning based on the evidence that the brain computes on high-dimensional, distributed, representations of data. The main operation of HD is encoding, which transfers the input data to hyperspace by mapping each input feature to a hypervector, accompanied by so-called bundling procedure that simply adds up the hypervectors to realize encoding hypervector. Although the operations of HD are highly parallelizable, the massive number of operations hampers the efficiency of HD in embedded domain. In this paper, we propose SHEARer, an algorithm-hardware co-optimization to improve the performance and energy consumption of HD computing. We gain insight from a prudent scheme of approximating the hypervectors that, thanks to inherent error resiliency of HD, has minimal impact on accuracy while provides high prospect for hardware optimization. In contrast to previous works that generate the encoding hypervectors in full precision and then ex-post quantizing, we compute the encoding hypervectors in an approximate manner that saves a significant amount of resources yet affords high accuracy. We also propose a novel FPGA implementation that achieves striking performance through massive parallelism with low power consumption. Moreover, we develop a software framework that enables training HD models by emulating the proposed approximate encodings. The FPGA implementation of SHEARer achieves an average throughput boost of 104,904x (15.7x) and energy savings of up to 56,044x (301x) compared to state-of-the-art encoding methods implemented on Raspberry Pi 3 (GeForce GTX 1080 Ti) using practical machine learning datasets.

In recent years, smart home technology has become prevalant and important for various applications. A typical smart home system consists of sensing nodes sending raw data to a cloud server which performs inference using a Machine Learning (ML) model trained offline. This approach suffers from high energy and communication costs and raises privacy concerns. To address these issues researchers proposed hierarchy aware models which distributes the inference computations across the sensor network with each node processing a part of the inference. While hierarchical models reduce these overheads significantly they are computationally intensive to run on resource constrained devices which are typical to smart home deployments. In this work we present a novel approach combining Hierarchy aware Neural Networks (HNN) with variational dropout technique to generate sparse models which have low computational overhead allowing them to be run on edge devices with limited resources. We evaluate our approach using an extensive real-world smart home deployment consisting of several edge devices. Measurements across different devices show that without significant loss of accuracy, energy consumption can be reduced by up to 35% over state-of-the-art.

Hyperdimensional (HD) computing is a mathematical framework, inspired by neuroscience, which can be used to represent many machine learning (ML) problems. Data is first encoded into high dimensional space (on the order of 103 or 104 dimensions) to create hypervectors. HD computing combines these hypervectors to create a model used for inference. However, due to the high dimensionality of the hypervectors, inference in HD is very expensive, especially when it runs on embedded devices with limited resources. One naive approach to improve the efficiency of HD computing is to simply lower the dimensionality of hypervectors, which comes with a corresponding loss in accuracy. However, if the data is compressed intelligently, we can reduce the dimensionality of an HD model without sacrificing accuracy. To that end, we propose CompHD, a novel approach for compressing HD models while maintaining the accuracy of the original model. CompHD utilizes the mathematics of high-dimensional spaces to compress hypervectors into shorter vectors while maintaining the information of full length hypervectors. We evaluated the efficiency of CompHD on a variety of applications. Our results show that CompHD can reduce model size by an average of 69.7%, resulting in a execution time speed up of 4.1 × and improving energy efficiency by 74% while maintaining the accuracy of the original model. This enables more low powered IoT devices to utilize HD computing for ML problems.

Networked applications with heterogeneous sensors are a growing source of data. Such applications use machine learning (ML) to make real-time predictions. Currently, features from all sensors are collected in a centralized cloud-based tier to form the whole feature vector for ML prediction. This approach has high communication cost, which wastes energy and often bottlenecks the network. In this work, we study an alternative approach that mitigates such issues by “pushing” ML inference computations out of the cloud and onto a hierarchy of IoT devices. Our approach presents a new technical challenge of “rewriting” an ML inference computation to factor it over a network of devices without significantly reducing prediction accuracy. We introduce novel exact factoring algorithms for some popular models that preserve accuracy. We also create novel approximate variants of other models that offer high accuracy. Measurements on a common IoT device show that energy use and latency can be reduced by up to 63% and 67% respectively without reducing accuracy relative to sending all data to the cloud.