Manycore has become the mainstream processor architecture for building HPC supercomputer systems,providing powerful computing power for High Performance Computing(HPC)exascale supercomputers.With the increasing number of cores integrated on manycore processor chips,the competition for large-scale cores for memory resources has become more intense.Manycore on-chip memory hierarchy is an important structure that alleviates the"memory wall"problem,aids HPC applications better play the computing advantages of manycore processors,and improves the performance of practical applications.The design has a significant impact on the performance,power consumption,and area of an on-chip system.The design of a many-call on-chip memory hierarchy has a significant impact on the performance,power consumption,and area of manycore systems.It is an important part of the structural design of manycore systems and is a research interest in the industry.Owing to the differences in the development history of manycore chips,the design technology of on-chip microarchitecture,and the different requirements of the application fields,the current HPC mainstream manycore on-chip storage hierarchy is different;however,from the perspective of horizontal comparison and the vertical development trend of each processor,as well as from the changes in application requirements brought by the continuous integration and development of HPC,data science,and machine learning,the hybrid structure of the SPM+Cache would most likely become the mainstream choice for the on-chip storage hierarchy designs of manycore processors in HPC exascale supercomputer systems in the future.For exascale computing software and algorithms,the designs and optimization based on the characteristics of the manycore memory hierarchy can aid HPC applications benefit from the computing advantages of manycore processors,thus effectively improving the performance of practical applications.Therefore,software,algorithm design,and optimization technology for the characteristics of the manycore on-chip storage hierarchy is also a research interest in the industry.This study first partitioned the on-chip memory hierarchy into multilevel Cache,SPM,and SPM+Cache hybrid structures according to different organizations,and then summarized and analyzed the advantages and disadvantages of these structures.This study analyzed the current status and development trend of the memory hierarchy designs of the chips of mainstream exascale supercomputer systems,such as the international mainstream GPU,homogeneous manycore,and domestic manycore.In summary,the research status of software and hardware technologies is related to the design and optimization of the memory hierarchy from the manycore of the manycore LLC management and cache consistency protocol,SPM management and data movement optimization,and the global perspective optimization of the SPM+cache hybrid architecture.Thus,this study looks forward to the future research direction of on-chip memory hierarchy based on different perspectives,such as hardware,software,and algorithm designs.

Alltoall is an important collective communication mode that is widely used in scientific and engineering computing applications. In applications such as FFT, using as much memory space as possible can improve applications’ efficiency and expand the scale of problems to be solved. High-bandwidth alltoall_inplace communication under space-constrained buffer conditions can significantly improve the performance of such applications. In this paper, two alltoall_inpalce communication algorithms under the condition of limited buffer space are designed for the domestic parallel computing system based on Sunway many-core processor. The alltoall_inpalce communication interfaces for both the core array and the whole CPU node are realized. The experimental results show that algorithms implemented in this paper can achieve high bandwidth using limited buffer space.

高性能计算中,硬件支持的多播操作对应用程序性能具有至关重要的影响.Infiniband网络中现有的两类多播路由算法中,MINIHOP-MC未考虑路由均衡性问题,导致链路edge forwarding index(EFI)指数很大,严重影响多播消息性能;SSSP-MC虽然部分考虑了路由均衡性问题,但其运行时间很长,不能满足超大规模互连网络的需求.提出一种负载均衡的快速多播路由算法FULB-MC,可用于存在大量多播组的超大规模互连网络环境.该算法采用自底向上的多播树构建方法,并综合利用两种负载均衡策略根据局部负载信息进行多播路由选择.还提出了新的加入/离开多播组机制,避免频繁计算多播路由问题.在各种拓扑结构、各种通信模式下对FULB-MC进行了大量测试.结果表明,在运行时间方面,FULB-MC显著低于SSSP-MC;在链路EFI指数方面,FULB-MC明显优于MINIHOP-MC,而与SSSP-MC基本相当.

Using in-network-computation capabilities of the network devices (also called hardware collectives) to optimize MPI collectives has become popular in high-performance computing, and shows significant performance advantages. However, the hardware collectives are not flawless in practical use scenarios. One of the problems is that it is difficult to use. In order to obtain the performance advantage of hardware collectives, the network management software need to create dedicated aggregate tree for each MPI communicator, which is a complicated task. One solution is to make MPI communicators sharing the global imprecise aggregate trees created by the management software when initiating networks, but it leads to heavy interference between MPI communicators and causes significant performance degradation. So we have to make tradeoff between performance and ease of use. We propose a hybrid approach to optimize MPI collectives by in-network-computation and point-to-point messages. On the one hand, we use the pre-created aggregate trees in each super-node, rather than sending requests to the network management software to create dedicated aggregate trees. On the other hand, the hardware collectives are transferred only in the local super-node, so it cannot disturb the jobs running on other super-nodes. We provide a cost model to evaluate the overhead of the hybrid collective algorithms. We also test its performance in the new generation Sunway supercomputer. The results show that our approach reduces the median latency by 18%~74% compared to collectives implemented by point-to-point messages, although the performance decrease slightly compared to the original hardware collectives. In addition, the tail latency of our approach is significantly lower than that of the original hardware collectives in the presence of heavy interference.

在高性能计算领域,多播路由算法对硬件集合操作的性能具有至关重要的影响.随着系统规模的不断扩大,多播组的个数急剧增加,可能会超过硬件支持的多播表条目数,而现有的多播路由算法要么没有给出解决方案,要么存在时间开销大、多播路由经常变化等问题.为此,首先对胖树中的无冲突多播生成树数量进行了量化研究,并以此为基础提出了一种适用于胖树的高效实用的定制多播路由算法(customized multicast routing for limited multicast forwarding table size,C-MR4LMS).C-MR4LMS在构建多播树时,根据多播组的 MGID(multicast global identification)静态地将多播组映射到1棵生成树中,从而快速完成多播树的构建;而在合并多播树时,仅需合并使用同一生成树的多播组,且不会改变被合并多播组的路由.然后提出了 2种减少多播树冲突的方法:一是分层的MGID分配策略,以避免出现同一终端节点使用同一颜色加入多个多播组的情况;二是相互无干扰的作业节点分配策略,保证2个作业的多播组互不干扰.最后,在ibsim模拟器及神威E级原型机上对C-MR4LMS进行了测试,该多播路由算法计算多播路由的时间比现有的多播路由算法有了显著下降,最大下降了 94％.

高性能计算中,硬件支持的多播操作对集合通信的性能具有重要影响.随着超级计算机系统规模的不断扩大,多播组的个数急剧增加,可能会超过硬件支持的多播表条目数,而现有的多播路由算法没有给出相应的解决方案.提出一种面向有限多播表条目数的多播路由算法MR4LMS(multicast routing for limited MFT size),该算法使用先构造后染色、先染色后构造2种多播树构建方法,可显著降低所需的多播表条目数;而在多播表条目数不足时,通过合并算法将多个相似的多播组合并到一起以进一步减少所需的多播表条目数.在多种典型拓扑结构及通信模式下对MR4LMS进行了测试,结果表明仅需设置256个多播表条目就能够支持数千甚至数万个多播组,满足典型应用的需求.还对多播路由算法的最大EFI(edge forwarding index)及运行时间进行了测试,获得了令人满意的结果,表明MR4LMS可用于超大规模互连网络.

本文描述了神威E级原型机的互连网络和消息机制.神威E级原型机是继神威蓝光、神威?太湖之光之后神威家族的第三代计算机.该计算机作为一台E级计算机的原型机,峰值性能3.13PFlops,其最大的特色之一就是采用28Gbps传输技术,设计开发了新一代的神威高阶路由器和神威高性能网络接口两款芯片,在传统胖树的基础上,设计了双轨泛树拓扑结构,定义实现了新颖的神威消息原语和消息库,实现了一种基于包级粒度动态切换的双轨乱序消息机制,通信性能比神威?太湖之光互连网络提升了4倍,为神威E级计算机互连网络的研制奠定了基础.

The Sunway TaihuLight, equipped with 10 million cores, is currently the world's third fastest supercomputer. SpMV is one of core algorithms in many high-performance computing applications. This paper implements a fine-grained design for generic parallel SpMV based on the special Sunway architecture and finds three main performance limitations, i.e., storage limitation, load imbalance, and huge overhead of irregular memory accesses. To address these problems, this paper introduces a customized and accelerative framework for SpMV (CASpMV) on the Sunway. The CASpMV customizes an auto-tuning four-way partition scheme for SpMV based on the proposed statistical model, which describes the sparse matrix structure characteristics, to make it better fit in with the computing architecture and memory hierarchy of the Sunway. Moreover, the CASpMV provides an accelerative method and customized optimizations to avoid irregular memory accesses and further improve its performance on the Sunway. Our CASpMV achieves a performance improvement that ranges from 588.05 to 2118.62 percent over the generic parallel SpMV on a CG (which corresponds to an MPI process) of the Sunway on average and has good scalability on multiple CGs. The performance comparisons of the CASpMV with state-of-the-art methods on the Sunway indicate that the sparsity and irregularity of data structures have less impact on CASpMV.

In recent years, the improvements of system performance and energy efficiency for supercomputers have faced increasing challenges, which create more intensive demands on the architecture design for realizing exascale computing. This paper first analyzes the main requirements of exascale computing on the aspects of the parallel computing application and supercomputing center operation. Afterwards, a mapping scheme of “demands-challenges-architecture” is proposed. Then, the major challenges of exascale supercomputer, such as scalability, power consumption, data movement, programming and availability, are thoroughly analyzed, and the corresponding appropriate solutions are proposed. Moreover, this paper proposes the Sunway computer architecture towards exascale computing in which the many-core processor, network chipset and software system are all domestically-designed. The technology roadmap of Sunway supercomputer will hold the comprehensive design methods for the architecture, including the processor, interconnect network, assembly structure, power supply, cooling system, system software, parallel algorithm and application support, promising great advances for exascale supercomputing.

随着系统规模与集成度的快速增加,可靠性与可用性问题成为构建E级计算机系统所面临的重大挑战.针对神威太湖之光超级计算机可靠性与可用性设计与实现开展全面的分析.首先概要描述神威太湖之光超级计算机系统结构.其次,系统提出神威太湖之光超级计算机可靠性增强技术以及故障预测、主动迁移、任务局部降级等主被动容错技术,建立神威太湖之光超级计算机多层次主被动协同的容错系统.再次,根据系统故障统计信息,分析失效分布及主要失效来源,结合指数、对数正态与韦布尔3种典型寿命周期分布,对神威太湖之光系统故障间隔时间分布进行数据拟合分析.最大似然估计与K-S(Kolmogorov Smirnov)检验结果表明,对数正态分布与系统失效经验数据取得了最好的拟合度,建立神威太湖之光系统失效分布模型,并计算得出系统平均无故障时间.通过系统运行统计与实际应用测试,分析了故障预测精确度以及主动迁移、局部降低等容错技术的时间开销与容错效果.最后,在神威太湖之光超级计算机可靠性与可用性分析的基础上,提出E级计算机系统高可靠与高可用技术发展建议.

IPoIB是一种在InfiniBand网络上支持IP的协议,使IP应用程序可以运行在InfiniBand网络上.我们在国产并行系统上实现了IPoIB,通过乱序处理、内存拷贝优化、网络参数调优和避免应答延迟的优化手段,实现了IPoIB在国产并行系统上的性能提升.实验结果表明,IPoIB在国产并行系统上正确运行,网络带宽与优化前相比提高近6倍,与10GbE万兆以太网相比,IPoIB更具优势,乱序处理机制减少乱序效果明显.

大规模异构众核计算机系统具有计算能力强、性能功耗比高等突出优点,已成为超级计算机的发展方向,但其复杂的异构结构和庞大的系统规模,也使系统的可用性面临巨大挑战,因此研究面向大规模异构众核系统的轻量级容错技术具有重要意义.针对传统基于检查点的系统级容错开销过大的问题,在Parallel C语言中设计并实现了故障局部感知的轻量级降级、编译指导与自动分析的检查点等语言支持的容错机制,兼顾了好用性和高效性.局部故障感知的轻量级降级结合动态任务调度框架实现,支持众核系统,可扩展到百万以上并行规模;编译指导与自动分析的检查点通过程序员插入简单的编译指示,由编译器进行分析,提示不需要保留的数据,可有效降低保留恢复的数据量.神威太湖之光超级计算机上的测试数据表明,两种容错措施相对于传统容错方法效果良好,轻量级降级的容错开销小于1％,相对于传统回卷容错方法单次故障执行时间可减少3.5％以上,编译指导与自动分析的检查点在典型应用中最多可将保留量降低至1/10,具有很好的实用性.

高性能计算系统的体系结构日益复杂和现有性能分析工具的智能程度不足,导致高性能计算应用的程序性能分析和优化的成本代价日益高昂。所幸,人工智能领域目前取得了重要进展,其中深度学习技术发挥了重要作用,它给性能分析工具的智能化带来了契机。提出一种基于深度学习的程序性能智能分析框架,其核心思想是将程序的性能分析问题抽象成可用机器学习技术描述的分类问题,使用处理器支持的PMU采集分类所需的性能数据并标准化,使用簇评估技术结合簇的实际含义确定性能问题类别,通过稀疏编码自动学习性能数据特征并构建性能问题分类模型。在神威太湖之光超级计算机上实现了程序性能分析框架原型。实验结果表明,该性能分析方法能够直观地指导程序员快速把握当前应用最为突出的性能瓶颈问题,提高应用优化的效率,降低用户调优代码的成本。

Heterogeneous many-core architecture,with ultra-high performance to power consumption ratio,has become an important trend of supercomputer architecture development.However,many-core systems always have more complex parallel hierarchy and memory hierarchy,hence posing a great challenge to programming and optimization.Therefore,the study of many-core-oriented parallel programming techniques is of great significance,since it can reduce the difficulty of parallel programming on domestic many-core systems and improve the performance of parallel programs.This work proposes a multi-model parallel programming model upon unified architecture,including heterogeneous-fused speedup programming model and isomorphic independent programming model.Based on this model,Parallel C programming language is designed to effectively describe heterogeneous parallelism of the domestic many-core system.Compared to MPI+X programming pattern,programming with Parallel C has a global perspective,as well as advantages in the hierarchy locality description,one-side message passing and multi-core applications compatibility.The Parallel C compiler system constructed with Open64 fully supports the heterogeneous-fused speedup programming model and isomorphic independent programming model.In addition,the design and implementation of data layout and automatic DMA optimization,compiler-directed thread proxy optimization and topology-aware collective communications optimization are presented.The performance of the proposed method is evaluated with the Miro Benchmark and practical applications on Sunway Taihu Light computer system.Experimental results show that Parallel C language and the compile system have good performance and scalability to effectively support large-scale applications.

The design of efficient dynamic task scheduling and fault‐tolerant mechanism is an issue of crucial importance in high‐performance computing field .Most existing methods ,however ,can hardly achieve good scalability on large‐scale system .In this paper , we propose a scalable dynamic task scheduling model via N‐level queuing theory ,which dramatically reduces the programming burden by providing programmer with concise parallel programming framework .On one hand ,we utilize the Poisson process theory to analyze the average wait time of tasks ,and then decide the task layers according to threshold .On the other hand ,we reduce the fault tolerance overhead using region‐aware light‐weight degradation model .Experimental results with Micro Benchmark on Bluelight system with 32 768 cores show that our method achieves good scalability when the tasks take 3.4 s on average and the overhead is just 7 .2% of traditional model .Running on 16 384 cores ,pharmacological application DOCK achieves performance improvement by 34.3% with our scheduling .Moreover ,the results of DOCK show our fault‐tolerant model achieves 3.75% ~ 5.13% performance improvements over traditional mechanism .

在大规模并行系统上,并行应用涉及的文件操作的可靠性和文件数据的完整性验证很重要.基于传统串行MD5摘要的文件验证手段无法满足大规模并行应用的需求,提出一种多层MD5并行摘要思想.基于此实现了文件完整性实时检测接口,并测试了串行MD5摘要和多层MD5摘要的性能,以及相应的文件完整性检测技术的性能.

本文从E级计算运行时系统的需求和挑战出发,跟踪国内外相关研究热点,分析大型高性能计算机系统运行时组成的显著特点或技术发展趋势,阐述知名研究机构的运行时系统研究情况,并对E级计算运行时系统的结构组成和设计思路进行了总结.

Dynamic execution mode of OpenCL asks the underlying platform to support the dynamic generation,compiling and loading and running of the device files.For the platforms without these characteristics,they have to consider the support approach in terms of software.By using function-rename technology to solve the correct identification issue for the functions with same name,the predo strategy based on dynamic execution flow can implement the dynamic execution mode of OpenCL in static compiling environment.

The API interfaces provided by CUDA help programmers to get high performance CUDA applications in GPU, but they cannot support most I/O operations in device codes. The characteristics of CUDA's mapped memory are used here to create a dynamic polling service model in the host which can satisfy most I/O functions such as read/write file and “printf”. The technique to implement these I/O functions has some influence on the performance of the original applications. These functions quickly respond to the users' I/O requirements with the “printf” performance better than CUDA's. An easy and effective real-time method is given for users to debug their programs using the I/O functions. These functions improve productivity of converting legacy C/C++ codes to CUDA and broaden CUDA's functions.

面向处理器多层次存储子系统的访存数据流优化是编译优化的关键技术.传统处理器一般采用硬件管理的高速缓存的存储结构.当前一些多核、众核处理器为了能够获得更大的片上存储器容量和更高的访存速度,倾向采用软件管理的流存储模型,例如Cell、GPU和Cyclops64等.相比较硬件管理的高速缓存的存储结构,软件既要规划好程序中的变量在各级存储层的存储布局,又要充分优化程序数据访问对离计算核近、速度快的存储层局部性利用,具有较大的挑战性.本文对这种存储结构上的访存数据流优化技术进行了归纳总结,然后预测了该研究领域的技术发展趋势.