Many fields of scientific simulation, such as chemistry and condensed matter physics, are increasingly eschewing dense tensor contraction in favor of sparse tensor contraction. In this work, we center around binary sparse tensor contraction (SpTC) which has the challenges of index matching and accumulation. To address these difficulties, we present GSpTC, an efficient element- wise SpTC framework on CPU-GPU heterogeneous systems. GSpTC first introduces a fine-grained partitioning strategy based on element- wise tensor contraction. By analyzing and selecting appropriate dimension partitioning strategies, we can efficiently utilize the multi-threading parallelism on GPUs and optimize the overall performance of GSpTC. In particular, GSpTC leverages multi-threading parallelism on GPUs for the contraction phase and merging phase, which greatly accelerates the computation phase in sparse tensor contraction computations. Furthermore, GSpTC employs parallel pipeline technology to hide the data transmission time between the host and the device, further enhancing its performance. As a result, GSpTC achieves an average performance improvement of 267% compared to the previous state-of-the-art framework Sparta.

Sparse linear algebra includes the fundamental and important operations in various large-scale scientific computing and real-world applications. There exists performance bottleneck for sparse linear algebra since it mainly contains the memory-bound computations with low arithmetic intensity. How to improve its performance has increasingly become a focus of research efforts. Using parallel computing techniques to accelerate sparse linear algebra is currently the most popular method, while facing various challenges, e.g., large-scale data brings difficulties in storage, and the sparsity of data leads to irregular memory accesses and parallel load imbalance. Therefore, this article provides a comprehensive overview on acceleration of sparse linear algebra operations using parallel computing platforms, where we focus on four main classifications: sparse matrix-vector multiplication (SpMV), sparse matrix-sparse vector multiplication (SpMSpV), sparse general matrix-matrix multiplication (SpGEMM), and sparse tensor algebra. The takeaways from this article include the following: understanding the challenges of accelerating linear sparse algebra on various hardware platforms; understanding how structured data sparsity can improve storage efficiency; understanding how to optimize parallel load balance; understanding how to improve the efficiency of memory accesses; understanding how do the adaptive frameworks automatically select the optimal algorithms; and understanding recent design trends for acceleration of parallel sparse linear algebra.

In recommendation systems, Graph Convolutional Networks (GCNs) often suffer from significant computational and memory cost when propagating features across the entire user-item graph. While various sampling strategies have been introduced to reduce the cost, the challenge of neighbor explosion persists, primarily due to the iterative nature of neighbor aggregation. This work focuses on exploring subgraph propagation for scalable recommendation by addressing two primary challenges: efficient and effective subgraph construction and subgraph sparsity. To address these challenges, we propose a novel GCN model for recommendation based on Subgraph propagation, called SubGCN. One key component of SubGCN is BiPPR, a technique that fuses both source- and target-based Personalized PageRank (PPR) approximations, to overcome the challenge of efficient and effective subgraph construction. Furthermore, we propose a source-target contrastive learning scheme to mitigate the impact of subgraph sparsity for SubGCN. We conduct extensive experiments on two large and two medium-sized datasets to evaluate the scalability, efficiency, and effectiveness of SubGCN. On medium-sized datasets, compared to full-graph GCNs, SubGCN achieves competitive accuracy while using only 23.79% training time on Gowalla and 16.3% on Yelp2018. On large datasets, where full-graph GCNs ran out of the GPU memory, our proposed SubGCN outperforms widely used sampling strategies in terms of training efficiency and recommendation accuracy.

Sparse Matrix and Vector multiplication (SpMV) is one of the core algorithms in various large-scale scientific computing and real-world applications. With the rapid development of AI and big data, the input vector in SpMV becomes sparse in many application fields. Especially in some graph analysis calculations, the sparsity of the input vector will change with the running of the program, and the non-zero element distribution of the adjacency matrix of some graph data has the power law property, leading to serious load imbalance, which requires additional optimization means. Therefore, the optimal SpMV kernel may be different, and a single SpMV kernel can no longer meet the acceleration requirements. In this article, we propose a decision tree-based adaptive SpMV framework, named DTSpMV, that can automatically select appropriate SpMV kernels according to different input data in iterations of graph computation. Based on the analysis of computing patterns, bit-array compression algorithms, and serial and parallel algorithms, we encapsulate nine SpMV kernels within the framework. We explore machine learning-based kernel selectors in terms of both accuracy and runtime overhead. Experimental results on NVIDIA Tesla T4 GPU show that our adaptive framework achieves the arithmetic average performance improvement of 152× compared to the SpMV kernel in cuSPARSE.

Convolutional neural networks (CNNs) are widely utilized in intelligent edge computing applications such as computational vision and image processing. However, as the number of layers of the CNN model increases, the number of parameters and computations gets larger, making it increasingly challenging to accelerate in edge computing applications. To effectively adapt to the tradeoff between the speed and accuracy of CNNs inference for smart applications. This paper proposes an FPGA-based adaptive CNNs inference accelerator synergistically utilizing filter pruning, fixed-point parameter quantization, and multi-computing unit parallelism called APPQ-CNN. First, the article devises a hybrid pruning algorithm based on the L1-norm and APoZ to measure the filter impact degree and a configurable parameter quantization fixed-point computing architecture instead of floating-point architecture. Then, design a cascade of the CNN pipelined kernel architecture and configurable multiple computation units. Finally, conduct extensive performance exploration and comparison experiments on various real and synthetic datasets. With negligible accuracy loss, the speed performance of our accelerator APPQ-CNN compares with current state-of-the-art FPGA-based accelerators PipeCNN and OctCNN by 2.15x and 1.91x, respectively. Furthermore, APPQ-CNN provides settable fixed-point quantization bit-width parameters, filter pruning rate, and multiple computation unit counts to cope with practical application performance requirements in edge computing.

The need for multi-GPU platforms in graph neural networks (GNNs) has been driven by the growing size of input graphs. However, although the existing multi-gpu GNN framework has been optimized from the perspective of optimizing computing and communication operations, communication competition still exists. To this end, we introduce PUSHGNN, a runtime system designed to reduce communication overhead across GPUs, boosting GNN performance. Therefore, we designed a push-based pipeline communication model and made custom tuning to significantly reduce pipeline contention. Comparative assessments demonstrate that PUSHGNN consistently outperforms leading full-graph GNN systems on average 1.97× and 7.57× faster than MGG and MGG-UVM, respectively.

Graph convolutional network (GCN) has been extensively applied to recommender systems (RS) and achieved significant performance improvements through iteratively aggregating high-order neighbors to model the relevance between users and items as well as their characteristics. In the aggregation process, GCN models usually give neighbors the same or trainable weights based on implicit features, ignoring explicit ones. In this work, we take the features with explicit meanings or extracted with specific purpose as explicit ones (e.g., temporal features) and the others contained in user-item network as implicit ones (e.g., user preferences). However, some explicit features and knowledge play an essential role in improving the model representation ability and explainability in recommendation systems. To deal with the limitation, we propose a GCN based framework to embed the explicit features or those extracted with explicit intentions in this work. We also provide specific implementations based on two commonly researched features, temporal evolution and popularity bias. Specifically, we first experimentally analyze the popularity bias of the representation learning in RS based on two commonly used GCN models. Secondly, we propose a general framework to weigh neighbors based on explicit features or intentions. Thirdly, we implement a Temporal and Popularity weighted Aggregator (TPA) for GCN. The Interest-Forgetting Curve is utilized to capture temporal evolution as temporal weights and the data-driven Beta distribution is employed to tune the weights based on the node popularity flexibly. At last, we conduct extensive experiments on three real-world datasets to demonstrate the effectiveness of TPA in improving recommendation accuracy and alleviating the popularity bias.

Nowadays, more data are represented and stored by a graph structure, and subgraph matching is a fundamental problem in a variety of scientific machine learning and industrial applications, such as remote sensing image registration, industrial inspection, etc. Due to the nondeterministic polynominal hard (NP-hard) problem of subgraph matching, the explosive growth of graph data, the disadvantages of high energy consumption, and the high overhead of CPU and graphics processing unit (GPU) platforms, computing subgraph matching is becoming more and more challenging. To alleviate this problem, we propose a phased hybrid algorithm to accelerate the enumeration task of subgraph matching, called PH-CF, based on the CPU-field programmable gate arrays (FPGA) heterogeneous platform. This approach can make full use of the pipeline and data flow mechanism, low power consumption, and configurable characteristics of FPGA. First, the matching order of query vertices automatically selects GraphQL (GQL) or RI methods according to the sparsity of the data (query) graph. Second, a candidate vertex auxiliary data structure set partitioning method is designed to effectively realize the load balance of multiple computing units at the FPGA and CPU host sides. Third, FPGA's pipeline and data flow mechanism is used to accelerate the enumeration phase of subgraph matching. Experimental results on real-world and synthetic datasets show that the performance of the PH-CF outperforms the state-of-the-arts. PH-CF can obtain the average performance improvement of up to $16.07\times$, $38.61\times$, and $11.46\times$ over CFL, CECI, and DP-iso, respectively. Moreover, our approach has good stability and robustness on various datasets.

Sparse Matrix and Vector multiplication (SpMV) is one of the core algorithms in various large-scale scientific computing and real-world applications. With the rapid development of AI and big data, the input vector in SpMV becomes more and more sparse in many application fields. Especially in some graph analysis calculations, the sparsity of the input vector will change with the running of the program. At this time, the optimal SpMV kernel may be different, and a single SpMV kernel can no longer meet the acceleration requirements. In this paper, we propose a decision tree-based adaptive SpMV framework, named DTSpMV, that can automatically select appropriate SpMV kernels based on different input data in iterations of graph computation. Based on the analysis of computational patterns and serial and parallel algorithms, we encapsulate six SpMV cores within the proposed framework. We explore machine learning-based kernel selectors in terms of both accuracy and runtime overhead. Experimental results on NVIDIA Tesla P100 GPU show that our adaptive framework achieves the arithmetic average performance improvement of 60% compared to the state-of-the-art SpMV kernel benchmark, and the average ratio of the runtime prediction overhead of the framework to the total computing overhead is 1.9%.

Sparse matrix-sparse vector (SpMSpV) multiplication is one of the fundamental and important operations in many high-performance scientific and engineering applications. The inherent irregularity and poor data locality lead to two main challenges to scaling SpMSpV over high-performance computing (HPC) systems: (i) a large amount of redundant data limits the utilization of bandwidth and parallel resources; (ii) the irregular access pattern limits the exploitation of computing resources. This paper proposes a fine-grained parallel SpMSpV ( fgSpMSpV ) framework on Sunway TaihuLight supercomputer to alleviate the challenges for large-scale real-world applications. First, fgSpMSpV adopts an MPI \( + \)OpenMP \( +X \) parallelization model to exploit the multi-stage and hybrid parallelism of heterogeneous HPC architectures and accelerate both pre-/post-processing and main SpMSpV computation. Second, fgSpMSpV utilizes an adaptive parallel execution to reduce the pre-processing, adapt to the parallelism and memory hierarchy of the Sunway system, while still tame redundant and random memory accesses in SpMSpV, including a set of techniques like the fine-grained partitioner, re-collection method, and Compressed Sparse Column Vector (CSCV) matrix format. Third, fgSpMSpV uses several optimization techniques to further utilize the computing resources. fgSpMSpV on the Sunway TaihuLight gains a noticeable performance improvement from the key optimization techniques with various sparsity of the input. Additionally, fgSpMSpV is implemented on an NVIDIA Tesal P100 GPU and applied to the breath-first-search (BFS) application. fgSpMSpV on a P100 GPU obtains the speedup of up to \( 134.38\times \) over the state-of-the-art SpMSpV algorithms, and the BFS application using fgSpMSpV achieves the speedup of up to \( 21.68\times \) over the state-of-the-arts.

SpMM (Sparse-Dense Matrix Matrix Multiplication) is the key computing core in many machine learning, big data analysis, and neural network applications. Although SpMM has received more attention in recent years, it is still a challenge to effectively improve the performance of SpMM. For example, existing work has largely given little consideration to improving performance by optimizing memory accesses. To alleviate the problems, we propose a matrix partition method, then, a SpMM algorithm is customized to match the partition, which makes full use of the architecture of GPU, and it easily improves the parallelism to achieve the best performance by adjusting parameters. Furthermore, we present two optimized algorithms for memory access, named rsSpMM and rrSpMM, which not only combine the advantages of the above algorithms, but also greatly exploit the advantage of different memory spaces of GPU. Experimental results on NVIDIA Tesla T4 demonstrate that the proposed algorithms in this paper has a significant improvement over the previous state-of-the-art cuSPARSE libraries, and preferably delivers 4. 6x, 8.9x, and 10. 1x improvements, respectively.

Canonical Polyadic Decomposition (CPD) of sparse tensors is an effective tool in various machine learning and data analytics applications, in which sparse Matricized Tensor Times Khatri-Rao Product (MTTKRP) is the major performance bottleneck. To overcome this bottleneck and support efficient applications, this paper presents HPSpTM, an efficient sparse MTTKRP framework, to exploit the multi-level parallelism and reusability on heterogeneous HPC systems. HPSpTM incorporates: (1) a multi-level matrix-driven tiling engine that leverages the process- and thread-level parallelism of the underlying platform and data reusability based on the derived factor matrix-driven MTTKRP algorithm; (2) a tensor-driven parallel execution that enables buffering-aware scheduling and pipeline scheduling to optimize the performance in the tile granularity; (3) a partition-aware light weight data storage that exploits better data locality based on the proposed hierarchical and fine-grained execution; and (4) a performance auto-tuning technique that offers large flexibility for tile size auto-adjusting across various input datasets based on a designed runtime model. Our experiments show that HPSpTM on a Nvidia Tesla P100 obtains the average performance improvement of up to 76.46% over the state-of-the-arts, and HPSpTM achieves the speedup of up to 15.39× when scaling from 8 to 128 core groups, corresponding to processes, on the Sunway TaihuLight supercomputer.

Many fields of scientific simulation, such as chem-istry and condensed matter physics, are increasingly eschewing dense tensor contraction in favor of sparse tensor contraction. In this work, we centre around binary sparse tensor contraction (SpTC) which has the challenges of index matching and accumulation. To address and alleviate the above difficulties, we present GSpTC, an efficient element-wise SpTC framework on CPU-GPU heterogeneous systems. GSpTC first introduces a fine-grained index partition based on element-wise tensor con-traction, which facilitates efficient use of the parallel computing power of GPUs. In particular, GSpTC leverages multi-threading parallelism on GPUs for the contraction phase and merge phase, which greatly accelerates the computation phase in sparse tensor contraction. Through evaluation of five datasets, we find that the proposed GSpTC achieves an average performance improvement of 74.32% compared to the previous state-of-the-art Sparta framework.

Efficient execution of SPARQL queries over large RDF datasets is a topic of considerable interest due to increased use of RDF to encode data. Most of this work has followed either relational or graph-based approaches. In this paper, we propose an alternative query engine, called gSmart, based on matrix algebra. This approach can potentially better exploit the computing power of high-performance heterogeneous architectures that we target. gSmart incorporates: (1) grouped incident edge-based SPARQL query evaluation, in which all unevaluated edges of a vertex are evaluated together using a series of matrix operations to fully utilize query constraints and narrow down the solution space; (2) a graph query planner that determines the order in which vertices in query graphs should be evaluated; (3) memory- and computation-efficient data structures including the light-weight sparse matrix (LSpM) storage for RDF data and the tree-based representation for evaluation results; (4) a multi-stage data partitioner to map the incident edge-based query evaluation into heterogeneous HPC architectures and develop multi-level parallelism; and (5) a parallel executor that uses the fine-grained processing scheme, pre-pruning technique, and tree-pruning technique to lower inter-node communication and enable high throughput. Evaluations of gSmart on a CPU+GPU HPC architecture show execution time speedups of up to 46920.00x compared to the existing SPARQL query engines on a single node machine. Additionally, gSmart on the Tianhe-1A supercomputer achieves a maximum speedup of 6.90x scaling from 2 to 16 CPU+GPU nodes.

The Sunway TaihuLight, equipped with 10 million cores, is currently the world's third fastest supercomputer. SpMV is one of core algorithms in many high-performance computing applications. This paper implements a fine-grained design for generic parallel SpMV based on the special Sunway architecture and finds three main performance limitations, i.e., storage limitation, load imbalance, and huge overhead of irregular memory accesses. To address these problems, this paper introduces a customized and accelerative framework for SpMV (CASpMV) on the Sunway. The CASpMV customizes an auto-tuning four-way partition scheme for SpMV based on the proposed statistical model, which describes the sparse matrix structure characteristics, to make it better fit in with the computing architecture and memory hierarchy of the Sunway. Moreover, the CASpMV provides an accelerative method and customized optimizations to avoid irregular memory accesses and further improve its performance on the Sunway. Our CASpMV achieves a performance improvement that ranges from 588.05 to 2118.62 percent over the generic parallel SpMV on a CG (which corresponds to an MPI process) of the Sunway on average and has good scalability on multiple CGs. The performance comparisons of the CASpMV with state-of-the-art methods on the Sunway indicate that the sparsity and irregularity of data structures have less impact on CASpMV.

Identifying the family of malware can determine their malicious intent and attack patterns, which helps to efficiently analyze large numbers of malware variants. Methods based on traditional machine learning often require a lot of time and resources in feature engineering. Virtually all existing static analysis methods based on malware visualization are derived from grayscale images, while a single low-order feature representation may be detrimental to discovering hidden features in a malware family. Based on these problems, this paper proposes an effective malware classification framework (MalFCS) based on malware visualization and automated feature extraction. MalFCS includes mainly three modules: malware visualization, feature extraction, and classification. First, we visualize malware binaries as entropy graphs based on structural entropy. Second, we present a feature extractor based on deep convolutional neural networks to extract patterns shared by a family from entropy graphs automatically. Finally, we propose an SVM classifier to classify malware based on the extracted features. We evaluate the proposed MalFCS over two widely studied benchmark datasets, i.e., Malimg and Microsoft. Experimental results show that compared with the state-of-the-art methods, MalFCS can obtain excellent classification performance with accuracy of 0.997 and 1, respectively, achieving the state-of-the-art performance.

General sparse matrix-sparse matrix (SpGEMM) multiplication is one of the basic kernels in a great many applications. Several works focus on various optimizations for SpGEMM. To fully exploit the powerful computing capability of the Sunway TaihuLight supercomputer for SpGEMM, this paper designs the partitioning method and parallelization of CSR-based SpGEMM to make it well match to the Sunway architecture. In addition, this paper optimizes the partitioning method based on the distribution of the floating-point calculations of the CSR-based SpGEMM to achieve the load balance and performance improvement on the Sunway. We, respectively, analyze the performance, including the memory footprint and the execution time, of the parallel CSR-based SpGEMM and the optimized CSR-based SpGEMM on the Sunway. The experimental results show that the optimized CSR-based SpGEMM outperforms over the parallel CSR-based SpGEMM and has good scalability on the Sunway.

Sparse matrix-vector multiplication (SpMV) is one of the important subroutines in numerical linear algebras widely used in lots of large-scale applications. Accelerating SpMV on multicore and manycore architectures based on Compressed Sparse Row (CSR) format via row-wise parallelization is one of the most popular directions. However, there are three main challenges in optimizing parallel CSR-based SpMV: (a) limited local memory of each computing unit can be overwhelmed by assignments to long rows of large-scale sparse matrices; (b) irregular accesses to the input vector result in expensive memory access latency; (c) sparse data structure leads to low bandwidth usage. This paper proposes a two-phase large-scale SpMV, called tpSpMV, based on the memory structure and computing architecture of multicore and manycore architectures to alleviate the three main difficulties. First, we propose the two-phase parallel execution technique for tpSpMV that performs parallel CSR-based SpMV into two separate phases to overcome the computational scale limitation. Second, we respectively propose the adaptive partitioning methods and parallelization designs using the local memory caching technique for the two phases to exploit the architectural advantages of the high-performance computing platforms and alleviate the problem of high memory access latency. Third, we design several optimizations, such as data reduction, aligned memory accessing, and pipeline technique, to improve bandwidth usage and optimize tpSpMV’s performance. Experimental results on SW26010 CPUs of the Sunway TaihuLight supercomputer prove that tpSpMV achieves up to 28.61 speedups and yields the performance improvement of 13.16% over the state-of-the-art work on average.

The prevalence of the Internet of Things (IoT) and the explosion of available information on the Web have led to an enormous amount of widely available IoT data sets with sparsity. Sparse matrix-vector multiplication (SpMV) is one of the most essential algorithms in various kinds of IoT applications. This article designs an autotuning hybrid computing scheme for SpMV, named ahSpMV, on the powerful and unique architecture of Sunway TaihuLight supercomputer, to combine the advantages of the heterogeneous parallel Sunway architecture and the Hybrid (HYB) sparse matrix format and optimize the SpMV's performance. First, we propose a heterogeneous parallelization design for ahSpMV based on the heterogeneous manycore architecture of the SW26010 of Sunway TaihuLight and the hybrid feature of the HYB format. Second, we propose several optimization techniques for computation and communication of ahSpMV, to fully utilize the computing power of Sunway. Third, we analyze the execution time of ahSpMV on Sunway. Fourth, based on the performance analysis, we propose an autotuning scheme for ahSpMV to set the proper parameter for the HYB format. We evaluate ahSpMV's performance on the Sunway architecture. The result analysis indicates that ahSpMV has obvious performance improvement over parallel SpMV based on other related sparse matrix formats. The optimization techniques and the autotuning scheme for ahSpMV also yield expected optimization effects. Moreover, the experimental results illustrate that ahSpMV has good scalability on the Sunway architecture.

Multi-dimensional, large-scale, and sparse data, which can be neatly represented by sparse tensors, are increasingly used in various applications such as data analysis and machine learning. A high-performance sparse tensor-vector product (SpTV), one of the most fundamental operations of processing sparse tensors, is necessary for improving efficiency of related applications. In this article, we propose aeSpTV, an adaptive and efficient SpTV framework on Sunway TaihuLight supercomputer, to solve several challenges of optimizing SpTV on high-performance computing platforms. First, to map SpTV to Sunway architecture and tame expensive memory access latency and parallel writing conflict due to the intrinsic irregularity of SpTV, we introduce an adaptive SpTV parallelization. Second, to co-execute with the parallelization design while still ensuring high efficiency, we design a sparse tensor data structure named . Third, based on the adaptive SpTV parallelization with the novel tensor data structure, we present an auto-tuner that chooses the most befitting tensor partitioning method for aeSpTV using the variance analysis theory of mathematical statistics to achieve load balance. Fourth, to further leverage the computing power of Sunway, we propose customized optimizations for aeSpTV. Experimental results show that aeSpTV yields good sacalability on both thread-level and process-level parallelism of Sunway. It achieves a maximum GFLOPS of 195.69 on 128 processes. Additionally, it is proved that optimization effects of the partitioning auto-tuner and optimization techniques are remarkable.