To analyze the efficiency of supercomputer functioning, it is useful to collect information from performance monitoring counters available in all modern processors. However, the ability to obtain such data is very limited—usually no more than 4 counters can be accessed simultaneously. To overcome this, multiplexing technology can be used, which allows collecting more data thanks to switching between counters—at any time, data from a specific set of counters is collected, and such sets repeatedly alternate. However, the use of this technology comes at a price of growing overheads—the execution time of supercomputer applications increases. Unfortunately, this topic has not been sufficiently studied so far. In this paper, we have carried out a detailed analysis and comparison of overheads caused by three different variants of multiplexing implemented using PAPI and LIKWID libraries. The obtained results show that the average overhead is ∼ 3–5

Any modern supercomputer has an extremely complex architecture, and efficient usage of its resources is often a very difficult task, even for experienced users. At the same time, the field of high-performance computing is becoming more and more in demand, so the issue of efficient utilization of supercomputers is very urgent. Therefore, users should know everything important about performance of their jobs running on a supercomputer in order to be able to optimize them, and administrators should be able to monitor and analyze all the nuances of the efficient functioning of such systems. However, there is currently no complete understanding of what data are best to be studied (and how it should be analyzed) in order to have a whole picture of the state of the supercomputer and the processes taking place there. In this paper, we make our first attempt to answer this question. To do this, we are developing a model that describes all the potential factors that may be important when analyzing the performance of supercomputer applications and the HPC system as a whole. The paper provides both a detailed description of this model for users and administrators and some interesting real-life examples discovered on the Lomonosov-2 supercomputer using a software implementation based on the proposed model. Любой современный суперкомпьютер имеет крайне сложную архитектуру, и эффективное использование его ресурсов зачастую является очень сложной задачей даже для опытных пользователей. В то же время высокопроизводительные вычисления становятся все более востребованными и вопрос эффективного использования суперкомпьютеров очень актуален. Поэтому пользователи должны знать все самое важное о производительности их работы, выполняемой на суперкомпьютере, чтобы иметь возможность ее оптимизировать, а администраторы должны уметь отслеживать и анализировать все нюансы эффективного функционирования таких систем. Однако в настоящее время нет полного понимания того, какие данные лучше всего для этого изучать (и как их следует анализировать), чтобы иметь полную картину состояния суперкомпьютера и происходящих на нем процессов. В этой статье мы делаем нашу первую попытку ответить на этот вопрос. Для этого мы разрабатываем модель, которая описывает все потенциальные факторы, которые могут быть важными при анализе производительности суперкомпьютерных приложений и системы HPC в целом. В документе представлено как подробное описание этой модели для пользователей и администраторов, так и несколько интересных реальных примеров, обнаруженных на суперкомпьютере Ломоносов-2 с помощью программного обеспечения, реализованного на основе предложенной модели.

The architecture of modern supercomputers is extremely complex, so it is exceedingly difficult to monitor and maintain the efficiency of their functioning. And even if it is possible to collect the necessary data on the operation of all important supercomputer components, how not to drown in this ‘‘sea of information’’ and not miss the onset of a critical situation? This requires the automation of the workload analysis process. One of the possible solutions is to create a set of rules that automatically detect and notify supercomputer administrators about the occurrence of certain critical situations or cases of a significant decrease in the efficiency of supercomputer functioning. Such approach allows quickly identifying the most interesting and important situations for the administrator, as well as correctly prioritizing the workload analysis process in whole. This article describes the process of developing a set of 19 rules, each of which determines a way to detect the onset of a certain critical situation, provides a description of the possible causes of its occurrence, and also specifies the criticality of the situation that has arisen. These rules allow monitoring different aspects of supercomputer behavior: the efficiency of using application packages, the operation of the queue system, the load and availability of service servers, the presence of global performance issues in user applications, and the peculiarities of using separate partitions of the supercomputer. The developed rules formed the basis of the software solution that was implemented and evaluated on the Petaflop-level Lomonosov-2 supercomputer.

Supercomputer technologies are in demand for solving many important and computationallyintensive tasks in various fields of science and technology. Therefore, it is not surprising that there are several dozen supercomputer centers only in Russia. However, the goals of creating such centers, as well as the range of tasks solved in them, can vary greatly, therefore the structure of supercomputers and the policies for their usage can significantly differ. This leads to the fact that many supercomputer centers live an isolated life – the administrators of such centers tend to solve administration-related tasks on their own, despite the fact that solutions for many similar tasks have already been developed and applied in other centers. This can happen due to different reasons, but in any case, this situation could and should be improved. To do this, it is worth establishing a closer connection between supercomputer centers, which will allow more actively exchanging experience or jointly developing desired system software. In order to understand the current situation in this area, a survey was conducted of representatives among 10 large supercomputer centers in Russia, and its results are presented in this paper. Two relevant topics about using monitoring data in practice and real-life examples of supercomputer functioning improvement are also discussed here in more detail. Their vision on these topics is provided by the system administrators of HSE University, Skoltech and Moscow State University.

Managing and administering an HPC center is a real challenge. The Supercomputing Center at Moscow State University is the largest in Russia, with petascale machines running in the interest of thousands of users engaged in hundreds of research projects. Since its early stages of development, the Octoshell system has been instrumental in mastering the equipment and tackling technical issues. The more problems are solved, the more new ideas arise and chances for improvement are discovered. In this paper, we discuss major enhancements recently made to Octoshell, including the integration of new modules (such as extended object description, license management, hardware accounting, and others), and outline plans for the near future.

Just after the computer era has been started, the Research Computing Center of the Moscow State University was equipped with the most modern computing hardware. These days RCC MSU still operates large scale supercomputers including Lomonosov and Lomonosov-2. Supercomputers are open for research and education society supporting hundreds of projects. The huge numbers of hardware and software components and parameters together with the complexity of architectures implemented raise an extremely important question – how efficiently the supercomputers are used. The efficiency study requires deep monitoring and analysis of all processes inside supercomputers. To improve the efficiency, a set of tools and techniques should be created to make a quick and automated decisions in all sides of supercomputer functioning. The paper is a brief overview of RCC MSU experience in supercomputers productivity improving by use of smart software and analytical techniques.

Interactive visual analysis tools bring the ability of the real-time discovery of knowledge in large and complex datasets using visual analytics. It involves multiple iterations of data processing using various data handling approaches and the efficiency of the whole chain of the analysis process depends on the performance of chosen techniques and related implementations, as well as the quality of applied methods. Stages, where data processing includes intellectual handling (i.e., data mining and machine learning), which are the most resource-intensive, require a distinct attention for evaluation of different approaches. Clustering is one such machine learning technique that is commonly used to discover groups of data objects for further analysis. This work is focused on evaluation of clustering algorithms within the interactive visual analysis toolkit InVEx (Interactive Visual Explorer). InVEx represents a visual analytics approach aimed at cluster analysis and in-depth study of implicit correlations between multidimensional data objects. It is originally designed to enhance the analysis of computing metadata of the ATLAS experiment at the LHC for operational needs, but it also provides the same capabilities for other domains to analyze large amounts of multidimensional data. The experiments and evaluation processes are carried out using operational data from the supercomputer at the Lomonosov Moscow State University. These processes include benchmark tests to assess the relative performance between chosen clustering algorithms and corresponding metrics to assess the quality of produced clusters. Obtained results will be used as guidelines in assisting users in a process of visual analysis using InVEx.

Running any computing center is a complex task. With the growth of scales and costs such tasks become challenges. So the top supercomputer sites, being big in everything, have always required special approaches to manage, to control, and to take care of them. At present, large HPC centers can have a variety of totally diverse systems containing up to millions of components, having thousands of users worldwide with the full range of complicated applications. Obviously, tons of data have to be managed in a concerted way to allow such an informational factory functioning. This paper shares the design principles, some implementation details and the roadmap vision regarding the Octoshell HPC center management system, which has been developed and is currently being used in the everyday practice of Moscow State University supercomputer center. This open source system manages Lomonosov and Lomonosov-2 systems with a total of over 5 PFlops peak performance complexes at present, providing multiple tools aimed to tackle most typical workflow tasks both for regular users and system administrators in a single shell.

Efficiency is a major weakness in modern supercomputers. Low efficiency of user applications is one of the main reasons for that. There are many software tools for analyzing and improving the performance of parallel applications. However, supercomputer users often do not have sufficient knowledge and skills to apply these tools correctly in their specific case. Moreover, users often do not know that their applications work inefficiently. The main goal of our project is to help any HPC user to detect performance flaws in their applications and find out how to deal with them. To this end, we plan to develop an open-source software solution that performs automatic massive analysis of all jobs running on a supercomputer to identify those with efficiency issues and helps users to conduct a detailed analysis of an individual program (using existing software tools) to identify and eliminate the root causes of the loss of efficiency.

These days the number of supercomputer users and the jobs they execute is rapidly growing, especially for supercomputers, providing computing time to external users. Supercomputers and their computing time are highly expensive, so their efficiency is crucial for both users and owners. There are several ways to increase operational efficiency, however, in most cases it involves a trade-off between efficiency metrics. This brings about a need to define “efficiency” in each specific case. We use the historical data from two largest Russian supercomputers to create a number of metrics in order to provide the definition of resource management “efficiency”. The data from both Lomonosov and Lomonosov-2 supercomputers consists of over one year history of job executions. Lomonosov and Lomonosov-2 efficiency in terms of CPU hours utilization is considerably high, nevertheless, our global goal is to offer the way to maintain or improve this metric when maximizing others examined in the paper.

The resource utilization analysis of HPC systems can be performed in different ways. The method of analysis is selected depending primarily on the original focus of research. It can be a particular application and/or a series of application run analyses, a selected partition or a whole supercomputer system utilization study, a research on peculiarities of workgroup collaboration, and so on. The larger an HPC center is, the more diverse are the scenarios and user roles that arise. In this paper, we share the results of our research on possible roles and scenarios, as well as typical methods of resource utilization analysis for each role and scenario. The results obtained in this research have served as the basis for the development of appropriate modules in the Octoshell management system, which is used by all users of the largest HPC center in Russia, at Lomonosov Moscow State University.

One of the main problems of modern supercomputers is the low efficiency of their usage, which leads to the significant idle time of computational resources, and, in turn, to the decrease in speed of scientific research. This paper presents three approaches to study the efficiency of supercomputer resource usage based on monitoring data analysis. The first approach performs an analysis of computing resource utilization statistics, which allows to identify different typical classes of programs, to explore the structure of the supercomputer job flow and to track overall trends in the supercomputer behavior. The second approach is aimed specifically at analyzing off-the-shelf software packages and libraries installed on the supercomputer, since efficiency of their usage is becoming an increasingly important factor for the efficient functioning of the entire supercomputer. Within the third approach, abnormal jobs - jobs with abnormally inefficient behavior that differs significantly from the standard behavior of the overall supercomputer job flow - are being detected. For each approach, the results obtained in practice in the Supercomputer Center of Moscow State University are demonstrated.