Modern memory hierarchies work well with applications that have good spatial locality. Evolving (dynamic) graphs are important applications widely used to model graphs and networks with edge and vertex changes. They exhibit irregular memory access patterns and suffer from a high miss ratio and long miss penalty. Prefetching can be employed to predict and fetch future demand misses. However, current hardware prefetchers can not efficiently predict for applications with irregular memory accesses. In evolving graph applications, vertices that do not change during graph changes exhibit the same access correlation patterns. Current temporal prefetchers use one-to-one or one-to-many correlation to exploit these patterns. Similar patterns are recorded in the same entry, which causes aliasing and can lead to poor prefetch accuracy and coverage. This work proposes a software-assisted hardware prefetcher for evolving graphs. The key idea is to record the correlations between a sequence of vertex accesses and the following misses and then prefetch when the same vertex access sequence occurs in the future. The proposed Access-to-Miss Correlation (AMC) prefetcher provides a lightweight programming interface to identify the data structures of interest and sets the iteration boundary to update the correlation table. For the evaluated applications, AMC achieves a geomean speedup of 1.5x as compared to the best-performing prefetcher in prior work (VLDP). AMC can achieve an average of 62 coverage of 23

The state-of-the-art deep neural network (DNN) models use pruning to avoid over-fitting and reduce the number of parameters. In order to improve storage and computational efficiency, only nonzero elements are stored, and their locations are encoded into a sparse format. Sparse General Matrix Multiplication (SpGEMM) is the kernel computation of DNN-based applications. One challenge of computing SpGEMM is to avoid multiplying zero elements while keeping hardware utilization high in hardware accelerators that consist of processing element (PE) arrays. Prior work tackling this challenge typically requires complex interconnection networks, which adds high area and energy costs.This work proposes a HW/SW co-design architecture to compute SpGEMM efficiently without requiring complex interconnection networks. A novel fast packing algorithm, SorPack, is proposed to convert a sparse matrix into a dense matrix that increases PE utilization. The key idea is to sort columns and rows inside each submatrix based on the number of nonzero elements. The goal is to keep the partial sums that need to be added together close to each other, hence can be added locally and avoid the use of complex interconnection networks. In addition, a new tile-based hierarchical architecture, HIRAC, is proposed to provide a scalable system that maximizes the parallelism of the PEs. The HIRAC architecture consists of a novel PE array design and interconnection network tailored for DNN applications. The SorPack algorithm complements the HIRAC to further improve hardware utilization and overall system performance. Based on the evaluation results, HIRAC achieves an average of 3.2× speedup on a single layer of DNN as compared to the state-of-the-art sparse DNN accelerator SIGMA. In addition, HIRAC has a 9.5% area reduction and a 32% power reduction as compared to SIGMA. An end-to-end evaluation on a DNN model shows an 8.2× runtime reduction over the TPU.

Sparse linear algebra is an important kernel in many different applications. Among various sparse general matrix-matrix multiplication (SpGEMM) algorithms, Gustavson’s column-wise SpGEMM has good locality when reading input matrix and can be easily parallelized by distributing the computation of different columns of an output matrix to different processors. However, the sparse accumulation (SPA) step in column-wise SpGEMM, which merges partial sums from each of the multiplications by the row indices, is still a performance bottleneck. The state-of-the-art software implementation uses a hash table for partial sum search in the SPA, which makes SPA the largest contributor to the execution time of SpGEMM. There are three reasons that cause the SPA to become the bottleneck: (1) hash probing requires data-dependent branches that are difficult for a branch predictor to predict correctly; (2) the accumulation of partial sum is dependent on the results of the hash probing, which makes it difficult to hide the hash probing latency; and (3) hash collision requires time-consuming linear search and optimizations to reduce these collisions require an accurate estimation of the number of non-zeros in each column of the output matrix. This work proposes ASA architecture to accelerate the SPA. ASA overcomes the challenges of SPA by (1) executing the partial sum search and accumulate with a single instruction through ISA extension to eliminate data-dependent branches in hash probing, (2) using a dedicated on-chip cache to perform the search and accumulation in a pipelined fashion, (3) relying on the parallel search capability of a set-associative cache to reduce search latency, and (4) delaying the merging of overflowed entries. As a result, ASA achieves an average of 2.25× and 5.05× speedup as compared to the state-of-the-art software implementation of a Markov clustering application and its SpGEMM kernel, respectively. As compared to a state-of-the-art hashing accelerator design, ASA achieves an average of 1.95× speedup in the SpGEMM kernel.

In the past decade, monolithic three dimensional integrated circuits (M3D-ICs) advance fast and demonstrate several important breakthroughs in the fabrication process and circuit level design. This article surveys recent research works on M3D technology from the computer system perspective with several case studies, and presents the opportunities as well as the challenges brought by this emerging technology. We also discuss possible applications of M3D based computer architectures. As the M3D technology is attracting enormous attention from both industry and academia, we expect that this article can be a good reference for the researchers and industrial partners who are interested in this fast evolving field, and promote the research activities of computer system/architecture design with this emerging technology.

Recurrent spiking neural network (RSNN) is a frequently studied model to understand biological neural networks, as well as to develop energy efficient neuromorphic systems. Deep learning optimization approach, such as backpropogation through time (BPTT), equipped with surrogate gradient, can be used as an efficient optimization method for RSNN. Including dynamic properties of biological neurons into the neuron model may improve the network's temporal learning capability. Earlier work only considers the spike frequency adaptation behavior with a large adaptation time constant that may be unsuitable for neuromorphic implementation. Besides adaptation, synapse is also an important structure for information transfer between neurons and its dynamics may influence network performance. In this work, a Leaky Integrate and Fire neuron model with dynamic synapses and spike frequency adaptation is used for temporal tasks. A step-by-step experiment is designed to understand the impact of recurrent connections, synapse model, and adaptation model on the network accuracy. For each step, a hyper-parameters tuning tool is used to find the best set of neuron parameters. In addition, the influence of the synapse and adaptation time constants is studied. Results suggest that, dynamic synapse is more efficient than adaptation in improving the network's learning capability. When incorporating adaptation and synapse model together, the network can achieve a similar accuracy as the sate-of-the-art RSNN works while requiring fewer neurons and smaller time constants.

Sparse linear algebra is an important kernel in many different applications. Among various sparse general matrix-matrix multiplication (SpGEMM) algorithms, Gustavson’s column-wise SpGEMM has good locality when reading input matrix and can be easily parallelized by distributing the computation of different columns of an output matrix to different processors. However, the sparse accumulation (SPA) step in column-wise SpGEMM, which merges partial sums from each of the multiplications by the row indices, is still a performance bottleneck. The state-of-the-art software implementation uses a hash table for partial sum search in the SPA, which makes SPA the largest contributor to the execution time of SpGEMM. There are three reasons that cause the SPA to become the bottleneck: (1) hash probing requires data-dependent branches that are difficult for a branch predictor to predict correctly; (2) the accumulation of partial sum is dependent on the results of the hash probing, which makes it difficult to hide the hash probing latency; and (3) hash collision requires time-consuming linear search and optimizations to reduce these collisions require an accurate estimation of the number of non-zeros in each column of the output matrix. This work proposes ASA architecture to accelerate the SPA. ASA overcomes the challenges of SPA by (1) executing the partial sum search and accumulate with a single instruction through ISA extension to eliminate data-dependent branches in hash probing, (2) using a dedicated on-chip cache to perform the search and accumulation in a pipelined fashion, (3) relying on the parallel search capability of a set-associative cache to reduce search latency, and (4) delaying the merging of overflowed entries. As a result, ASA achieves an average of 2.25× and 5.05× speedup as compared to the state-of-the-art software implementation of a Markov clustering application and its SpGEMM kernel, respectively. As compared to a state-of-the-art hashing accelerator design, ASA achieves an average of 1.95× speedup in the SpGEMM kernel.

Multi-Level Cell Phase Change Memory (MLC PCM) is a non-volatile memory technology that promises high-density data storage. However, MLC PCM suffers from long write latency because changing the state of each memory cell takes a long time and the number of cells that can be written concurrently is limited due to the relatively large size and power consumption of the write drivers. MLC PCM typically divides the write data into multiple cell groups to program cells in batches. The group mapping has a significant impact on the overall latency of write requests. Prior work proposed a group mapping technique for single-level cell (SLC) PCM based on the data pattern of a set of applications. In this work, a larger design space of group mapping is explored for MLC PCM. The specific group mapping for each benchmark is dynamically customized. Our technique leads to a 13.14% reduction in write latency compared to the state-of-the-art method (Du et al., 2013) and shows good temporal consistence in different benchmarks.

Spiking Neural Networks (SNN) can model biological neural networks with different levels of details. There are trade-offs between model fidelity and computation efficiency. Which model is the most appropriate one to use depends on the goal and the computation task. Temporal learning is an important feature of the brain, which requires neural networks to integrate information from the past to solve present computation tasks. Prior work has proposed different SNN models for temporal learning, which includes the Leaky-Integrate-and-Fire (LIF), the Adaptive Leaky-Integrate-and-Fire (ALIF), and the Exponential Adaptive Leaky-Integrate-and-Fire (AdEx). These models capture different biological details and exhibit different learning properties. This work aims to compare the model fidelity and learning performance of these three SNN models. Experimental data for in vitro living neural networks is used to first fit parameters of these three models. An automatic fitting tool is used to match the precise spike timing of the in vitro neurons and the modeled neurons. ALIF and AdEX can match with the spiking timing of the biological neuron better than the LIF does. The fitted models are then compared on a delay task, where the network needs to output values that were input into the network in the recent past. To compute the delay task, the Neural Engineering Framework (NEF) is used to implement a Legendre Memory Unit. Good performance is demonstrated on the delay task using ALIF, which suggests the possibility of implementing the algorithm on in vitro living neural networks. This work proposes a new neuron parameter fitting approach, compares three SNN models, and is the first to use detailed adaptive neurons on the delay task with the NEF approach.

Spontaneous synchronized neuronal activity is found in the neocortical areas of the developing as well as epileptic brain. Synchronized neuronal activity can also be found in dissociated neural networks in vitro. To study the mechanistic origin of this behavior, we built a simple and fast computational model of large network of approximately 80% excitatory and 20% inhibitory neurons. Our model precisely matched the spike times and spike frequency adaptation of cortical pyramidal neurons obtained via whole cell recordings in confined in vitro networks. High accuracy of our model was achieved by implementing an AHP current with sigmoid dynamics that approximated Ca2+ dependent spike frequency adaptation. Then, we implemented short term synaptic depression of the synaptic weights. Modeling of depression was verified by matching the experimental post synaptic responses with three different frequencies of spikes evocation in the presynaptic neuron using dual whole cell recording. Investigations with this computational network were then verified in an all-optical experimental system. This experimental setup was designed by confining a cortical network with polydimethylsiloxane (PDMS), and by co-transfecting the neurons with channel-rhodopsin 2 (ChR2) and jRgeco1a. Patterned stimulator with spatial resolution of somatic compartment (approximately 15 μm) was used to excite individual neuron in a 2D confined network consisting of approximately 300 neurons confined to an area of about 1 mm2. We explored different stimulation paradigms in silico and in vitro to achieve a burst free neuronal network of different size by activating different numbers and patterns of input neurons. Action potential adaptation of each spiking excitatory neuron and the short term synaptic depression of each activated synapses could lead the network into a non-bursting mode. Distributed stimulation method that was developed in this work could potentially be used in suppressing epileptic seizures. Keywords: burst free network, patterned stimulation, optogenetics, synaptic depression

General-purpose computing systems employ memory hierarchies to provide the appearance of a single large, fast, coherent memory. In special-purpose CPUs, programmers manually manage distinct, non-coherent scratchpad memories. In this article, we combine these mechanisms by adding a virtually addressed, set-associative scratchpad to a general purpose CPU. Our scratchpad exists alongside a traditional cache and is able to avoid many of the programming challenges associated with traditional scratchpads without sacrificing generality (e.g., virtualization). Furthermore, our design delivers increased security and improves performance, especially for workloads with high locality or that interact with nonvolatile memory.

As device integration density increases exponentially as predicted by Moore’s law, power consumption becomes a bottleneck for system scaling where leakage power of on-chip cache occupies a large fraction of the total power budget. Spin transfer torque magnetic random access memory (STT-MRAM) is a promising candidate to replace static random access memory (SRAM) as an on-chip last level cache (LLC) due to its ultralow leakage power, high integration density, and nonvolatility. Moreover, with the prevalence of edge computing and Internet-of-Things (IoT) applications, it can be beneficial to build a total nonvolatile cache hierarchy, including the L1 cache. However, building an L1 cache with STT-MRAM still faces severe challenges particularly because reducing its relatively high write latency by increasing write voltage can accelerate oxide breakdown of the MTJ device and threaten the L1 cache lifetime significantly due to intensive accesses. In our previous work, we proposed a dynamic overwriting voltage adjustment (DOVA) technique to deal with this challenge. In this article, we improve this technique by a DOVA promotion (DOVA PRO) technique for the STT-MRAM L1 cache, considering the cache write endurance and performance simultaneously. A high write voltage is used for performance-critical cache lines, while a low write voltage is used for other cache lines to approach an optimal tradeoff between reliability and performance. Experimental results show that the proposed technique DOVA PRO can improve cache performance by 23.5%, on average, compared to the DOVA technique. In the meantime, the average degradation of cache lifetime remains almost unchanged compared with the DOVA technique on average. Furthermore, DOVA PRO can support flexible configurations to achieve various optimization targets, such as higher performance or a longer lifetime.

Simplified models of neurons are widely used in computational investigations of large networks. One of the most important performance metrics of simplified models is their accuracy in reproducing action potential (spike) timing. In this article, we developed a simple, computationally efficient neuron model by modifying the adaptive exponential integrate and fire (AdEx) model [1] with sigmoid afterhyperpolarization current (Sigmoid AHP). Our model can precisely match the spike times and spike frequency adaptation of cortical pyramidal neurons. The accuracy was similar to a more complex two compartment biophysically realistic model of the same neurons. This work provides a simplified neuronal model with improved spike timing accuracy for use in modeling of large neural networks.Clinical Relevance— Accurate and computationally efficient single neuron model will enable large network modeling of brain regions involved in neurological and psychiatric disorders and may lead to a better understanding of the disorder mechanisms.

In this paper, we develop an in-memory analog computing (IMAC) architecture realizing both synaptic behavior and activation functions within non-volatile memory arrays. Spin-orbit torque magnetoresistive random-access memory (SOT-MRAM) devices are leveraged to realize sigmoidal neurons as well as binarized synapses. First, it is shown the proposed IMAC architecture can be utilized to realize a multilayer perceptron (MLP) classifier achieving orders of magnitude performance improvement compared to previous mixed-signal and digital implementations. Next, a heterogeneous mixed-signal and mixed-precision CPU-IMAC architecture is proposed for convolutional neural networks (CNNs) inference on mobile processors, in which IMAC is designed as a co-processor to realize fully-connected (FC) layers whereas convolution layers are executed in CPU. Architecture-level analytical models are developed to evaluate the performance and energy consumption of the CPU-IMAC architecture. Simulation results exhibit 6.5% and 10% energy savings for CPU-IMAC based realizations of LeNet and VGG CNN models, for MNIST and CIFAR-10 pattern recognition tasks, respectively.

Recent research suggests that in vitro neural networks created from dissociated neurons may be used for computing and performing machine learning tasks. To develop a better artificial intelligent system, a hybrid bio-silicon computer is worth exploring, but its performance is still inferior to that of a silicon-based computer. One reason may be that a living neural network has many intrinsic properties, such as random network connectivity, high network sparsity, and large neural and synaptic variability. These properties may lead to new design considerations, and existing algorithms need to be adjusted for living neural network implementation. This work investigates the impact of neural variations and random connections on inference with learning algorithms. A two-layer hybrid bio-silicon platform is constructed and a five-step design method is proposed for the fast development of living neural network algorithms. Neural variations and dynamics are verified by fitting model parameters with biological experimental results. Random connections are generated under different connection probabilities to vary network sparsity. A multi-layer perceptron algorithm is tested with biological constraints on the MNIST dataset. The results show that a reasonable inference accuracy can be achieved despite the presence of neural variations and random network connections. A new adaptive pre-processing technique is proposed to ensure good learning accuracy with different living neural network sparsity.

Machine learning models have been widely deployed in many real-world tasks. When a non-expert data holder wants to use a third-party machine learning service for model training, it is critical to preserve the confidentiality of the training data. In this paper, we for the first time explore the potential privacy leakage in a scenario that a malicious ML provider offers data holder customized training code including model compression which is essential in practical deployment The provider is unable to access the training process hosted by the secured third party, but could inquire models when they are released in public. As a result, adversary can extract sensitive training data with high quality even from these deeply compressed models that are tailored for resource-limited devices. Our investigation shows that existing compressions like quantization, can serve as a defense against such an attack, by degrading the model accuracy and memorized data quality simultaneously. To overcome this defense, we take an initial attempt to design a simple but stealthy quantized correlation encoding attack flow from an adversary perspective. Three integrated components-data pre-processing, layer-wise data-weight correlation regularization, data-aware quantization, are developed accordingly. Extensive experimental results show that our framework can preserve the evasiveness and effectiveness of stealing data from compressed models.

DRAM scaling has been slowed down. Emerging non-volatile memories (e.g., Phase-Change Memory) promises higher density, better scalability, and persistence. However, endurance is a fundamental issue that hinders the broad adoption of PCM-after repeated writes, a PCM cell can get stuck at a value and be no longer programmable. The prevalence of this stuck-at fault issue requires error detection and correction mechanisms for PCM. Existing solutions such as verify-after-write adds additional latency to PCM writes, which degrades overall system performance. Other solutions like in-memory error-correcting code (ECC) requires a high storage overhead and introduces more reliability issue because ECC bits tend to wear out faster than the protected data bits. In this paper, a novel stuck-at faults detection technique is proposed to improve performance and reliability simultaneously. Since stuck-at faults can only be detected after new writes, ECC does not need to be stored permanently and can be deleted immediately after a one-time detection, which helps to reduce ECC storage overhead. Therefore, this work proposes to use a small on-chip ECC cache to store the temporary ECC entries, which does not suffer from endurance issue. To maximize the utilization of the limited cache space, this work optimizes ECC entry insertion and deletion mechanisms and exploits memory bank-level parallelism to minimize performance impact. For the evaluated workloads, the proposed ECC cache achieves an average of 9.4% of the performance improvement over the baseline with a verify-after-write detection.

Applications with irregular memory access patterns do not benefit well from the memory hierarchy as applications that have good locality do. Relatively high miss ratio and long memory access latency can cause the processor to stall and degrade system performance. Prefetching can help to hide the miss penalty by predicting which memory addresses will be accessed in the near future and issuing memory requests ahead of the time. However, software prefetchers add instruction overhead, whereas hardware prefetchers cannot efficiently predict irregular memory access sequences with high accuracy. Fortunately, in many important irregular applications (e.g., iterative solvers, graph algorithms, and sparse matrix-vector multiplication), memory access sequences repeat over multiple iterations or program phases. When the patterns are long, a conventional spatial-temporal prefetcher can not achieve high prefetching accuracy, but these repeating patterns can be identified by programmers.In this work, we propose a software-assisted hardware prefetcher that focuses on repeating irregular memory access patterns for data structures that cannot benefit from conventional hardware prefetchers. The key idea is to provide a programming interface to record cache miss sequence on the first appearance of a memory access pattern and prefetch through replaying the pattern on the following repeats. The proposed Record-and-Replay (RnR) prefetcher provides a lightweight software interface so that the programmers can specify in the application code: 1) which data structures have irregular memory accesses, 2) when to start the recording, and 3) when to start the replay (prefetching). This work evaluated three irregular workloads with different inputs. For the evaluated workloads and inputs, the proposed RnR prefetcher can achieve on average 2.16× speedup for graph applications and 2.91× speedup for an iterative solver with a sparse matrix-vector multiplication kernel. By leveraging the knowledge from the programmers, the proposed RnR prefetcher can achieve over 95% prefetching accuracy and miss coverage.

Graph analytics is an important class of applications and is one of the cornerstone of big-data workloads. Unfortunately, due to poor data locality in most graph applications, conventional general-purpose computer architectures are unable to perform the best of their processing abilities. The main source of poor locality comes from accessing vertex properties. Upper-level caches cannot hold data blocks long enough due to their limited capacity and the long reuse distance of vertex properties. Moreover, accesses to properties can evict other useful data with good locality, which causes more conflicting misses. In this work, a small cache is added exclusively for the properties to solve this problem. We further enhance this structure with prefetchers to increase the hit rate of properties and improve performance of system. Experimental results show that compared to two state-of-the-art prefetcher and accelerator for graph computing, our proposed architecture achieves 1.13x-2.54x and 1.04x-1.27x performance improvements. In the meanwhile, the energy consumptions can be saved by 6.41%-13.43% and 34.67%-43.92% respectively.

As device integration density increases exponentially predicted by Moore's law, power consumption becomes a bottleneck for system scaling. On the other hand, leakage power of on-chip cache occupies a large fraction of the total power budget. STT-RAM is a promising candidate to replace SRAM as on-chip cache due to its ultra-low leakage power, high integration density and non-volatility. However, building L1 cache with STT-RAM still faces severe challenges especially because of its high write latency and energy overheads. Moreover, intensive accesses in L1 cache accelerate oxide breakdown and threaten the lifetime of STT-RAM significantly. In this paper, we propose a Dynamic Overwriting Voltage Adjustment (DOVA) technique for STT-RAM L1 cache. A high write voltage is used for performance critical cache lines while a low write voltage is used for other cache lines to approach an optimal trade-off between reliability and performance. Experimental results show that the proposed technique can improve cache performance up to 18%, and 9% on average with almost the same reliability level as in the case when only the low write voltage is used.

DRAM refresh is responsible for significant performance and energy overheads in a wide range of computer systems, from mobile platforms to datacenters [1] . With the growing demand for DRAM capacity and the worsening retention time characteristics of deeply scaled DRAM, refresh is expected to become an even more pronounced problem in future technology generations [2] . This paper examines content aware refresh , a new technique that reduces the refresh frequency by exploiting the unidirectional nature of DRAM retention errors: assuming that a logical 1 and 0 respectively are represented by the presence and absence of charge, 1 -to- 0 failures are much more likely than 0 -to- 1 failures. As a result, in a DRAM system that uses a block error correcting code (ECC) to protect memory, blocks with fewer 1 s can attain a specified reliability target (i.e., mean time to failure) with a refresh rate lower than that which is required for a block with all 1 s. Leveraging this key insight, and without compromising memory reliability, the proposed content aware refresh mechanism refreshes memory blocks with fewer 1 s less frequently. To keep the overhead of tracking multiple refresh rates manageable, refresh groupsgroups of DRAM rows refreshed togetherare dynamically arranged into one of a predefined number of refresh bins and refreshed at the rate determined by the ECC block with the greatest number of 1 s in that bin. By tailoring the refresh rate to the actual content of a memory block rather than assuming a worst case data pattern, content aware refresh respectively outperforms DRAM systems that employ RAS-only Refresh, all-bank Auto Refresh, and per-bank Auto Refresh mechanisms by 12, 8, and 13 percent. It also reduces DRAM system energy by 15, 13, and 16 percent as compared to these systems.