Most HPC resources that are available to a wide public userbase (for example XSEDE/ACCESS resources) have fairly basic user guides for building and running applications on the systems. Large performance differences, however, can occur if users are not experienced with detailed job configuration flags and their specific application behavior. This performance difference affects job quality-how well the job used its resources to accomplish the user's goals. However, a one-dimensional idea of job quality cannot capture all of the intricacies around how or why users might select a particular configuration. This paper presents four metrics for four different views of job quality, and then presents results from experiments over several applications that show the variability in quality that results from a variety of job configuration tuning parameters. This work can point the way towards guidelines for how users can potentially improve their own quality goals of their applications and thereby use the HPC resources most beneficially. One significant result we saw across applications is that high thread counts with low processes-per-node counts are generally not the best configurations; most applications do better with low thread counts and more processes per node.

Deciding whether or not a production job used its resources well is an ongoing concern in high performance computing (HPC). In this paper we show that low-overhead heartbeat data can be used to classify a job into a job quality level, using both machine learning and artificial intelligence techniques. Five HPC applications (one real, four mini) were used to create heartbeat datasets labeled with a job quality level, and then seven different ML and AI techniques were evaluated by training application-specific models for classifying runs into four different levels of job quality. The results show that, with very high accuracy, heartbeat data can serve as a mechanism for understanding production HPC job quality. Job quality feedback in production will help both users and administrators to better use and administer expensive HPC resources to meet computational needs.

AppEKG is a high performance computing (HPC) oriented application heartbeat framework designed for low-overhead monitoring of HPC applications in production, providing a unified, understandable view of dynamic HPC application behavior by capturing time-varying behavior with little overhead (similar to 1%). In this paper we demonstrate the value of heartbeat data collected with AppEKG, showing that it can be used for anomaly detection and run classification. This paper uses statistical modeling and time-series shapelet analyses as initial examples of heartbeat data analysis. The results of these analyses show that application heartbeats can be useful to help better understand how HPC applications are using the costly time and resources they consume, and that this can be applied in production.

Benchmarks play an important role in the development of machine learning algorithms, with reinforcement learning (RL) research having been heavily influenced by the available environments. However, RL environments are traditionally run on the CPU, limiting their scalability with typical academic compute. Recent advancements in JAX have enabled the wider use of hardware acceleration to overcome these computational hurdles, enabling massively parallel RL training pipelines and environments. This is particularly useful for multi-agent reinforcement learning (MARL) research. First of all, multiple agents must be considered at each environment step, adding computational burden, and secondly, the sample complexity is increased due to non-stationarity, decentralised partial observability, or other MARL challenges. In this paper, we present JaxMARL, the first open-source code base that combines ease-of-use with GPU enabled efficiency, and supports a large number of commonly used MARL environments as well as popular baseline algorithms. When considering wall clock time, our experiments show that per-run our JAX-based training pipeline is up to 12500x faster than existing approaches. %This enables efficient and thorough evaluations, with the potential to alleviate the evaluation crisis of the field. We also introduce and benchmark SMAX, a vectorised, simplified version of the popular StarCraft Multi-Agent Challenge, which removes the need to run the StarCraft II game engine. This not only enables GPU acceleration, but also provides a more flexible MARL environment, unlocking the potential for self-play, meta-learning, and other future applications in MARL. We provide code at https://github.com/flairox/jaxmarl.

Local Memory Store (LMStore) is a novel scratchpad memory (SPM) design, with recent research evaluation showing its capability for improving program performance. However, the performance of LMStore depends on its memory layout decided by its allocation scheme. In this paper, we evaluate the impact of SPM allocation on LMStore performance. Our experimental results, using benchmarks from the Malardalen WCET benchmark suite executing on LMStore architecture modeled in the PyCacheSim simulator, demonstrate that LMStore with a stack distance-based SPM allocation scheme significantly improves data movement by an average of 44.46% compared to a Cache-only architecture, and by an average of 23.89% compared to LMStore with a frequency-based SPM allocation scheme.

Local Memory Store (LMStore) is a novel hardware-controlled, compiler-managed Scratchpad memory (SPM) design [1], with an initial research evaluation that showed its possibility for improving program performance. This initial evaluation was performed over memory traces prior to the development of compiler support for LMStore. In this paper, we present compiler support for the LMStore design, and present experimental results that better evaluate LMStore performance. Experimental results on benchmarks from Malardalen benchmark suite [2] executing on the LMStore architecture modeled in Multi2Sim demonstrate that a hybrid LMStore-Cache architecture improves execution time by an average of 19.8 %, compared to a conventional cache-only architecture.

While many good development-oriented tools exist for analyzing and improving the performance of HPC applications, capability for capturing and analyzing the dynamic behavior of application in real production runs is lacking. Many heavily-used applications do keep some internal metrics of their performance, but there is no unified way of using these. In this paper we present the initial idea of AppEKG, both a concept of and a prototype tool for providing a unified, understandable view of HPC application behavior in production. Our prototype AppEKG framework can achieve less than 1% overhead, thus usable in production, and still provide dynamic data collection that captures time-varying runtime behavior.

Large-scale parallel file systems (PFSs) play an essential role in high-performance computing (HPC). However, despite their importance, their reliability is much less studied or understood compared with that of local storage systems or cloud storage systems. Recent failure incidents at real HPC centers have exposed the latent defects in PFS clusters as well as the urgent need for a systematic analysis. To address the challenge, we perform a study of the failure recovery and logging mechanisms of PFSs in this article. First, to trigger the failure recovery and logging operations of the target PFS, we introduce a black-box fault injection tool called PFault, which is transparent to PFSs and easy to deploy in practice. PFault emulates the failure state of individual storage nodes in the PFS based on a set of pre-defined fault models and enables examining the PFS behavior under fault systematically. Next, we apply PFault to study two widely used PFSs: Lustre and BeeGFS. Our analysis reveals the unique failure recovery and logging patterns of the target PFSs and identifies multiple cases where the PFSs are imperfect in terms of failure handling. For example, Lustre includes a recovery component called LFSCK to detect and fix PFS-level inconsistencies, but we find that LFSCK itself may hang or trigger kernel panics when scanning a corrupted Lustre. Even after the recovery attempt of LFSCK, the subsequent workloads applied to Lustre may still behave abnormally (e.g., hang or report I/O errors). Similar issues have also been observed in BeeGFS and its recovery component BeeGFS-FSCK. We analyze the root causes of the abnormal symptoms observed in depth, which has led to a new patch set to be merged into the coming Lustre release. In addition, we characterize the extensive logs generated in the experiments in detail and identify the unique patterns and limitations of PFSs in terms of failure logging. We hope this study and the resulting tool and dataset can facilitate follow-up research in the communities and help improve PFSs for reliable high-performance computing.

Long running applications often have varying behaviors, here called phases. While considerable work in computer architecture has been done in identifying application phases based on how the hardware is being exercised, comparatively less work has been focused on identifying application phases based on regions of source code being executed. In this paper we introduce a new methodology and an efficient tool framework, IncProf, for observing and capturing the time-varying source execution behavior of applications, and for then deducing application phases from the resulting data. Uses of this capability include simply better understanding the varying behavior of long running applications, and for efficiently tracking deployed application performance in the future by providing information to identify good instrumentation points.

High rates of discipline citations predict adverse life outcomes, a harm disproportionately borne by Black and Latino boys. We hypothesized that these citations arise in part from negative cycles of interaction between students and teachers, which unfold in contexts of social stereotypes. Can targeted interventions to facilitate identity safety-a sense of belonging, inclusion, and growth-for students help? Experiment 1 combined social-belonging, values-affirmation, and growth-mindset interventions delivered in several class sessions in 2 middle schools with a large Latino population (N = 669). This treatment reduced citations among negatively stereotyped boys in 7th and 8th grades by 57% as compared with a randomized control condition, 95% CI [-77%, -20%]. A growth-mindset only treatment was also effective (70% reduction, 95% CI [-84%, -43%]). Experiment 2 tested the social-belonging intervention alone, a grade earlier, at a third school with a large Black population and more overall citations (N = 137 sixth-grade students). In 2 class sessions, students reflected on stories from previous 7th-grade students, which represented worries about belonging and relationships with teachers early in middle school as normal and as improving with time. This exercise reduced citations among Black boys through the end of high school by 65%, 95% CI [-85%, -15%], closing the disparity with White boys over 7 years by 75%. Suggesting improved interactions with teachers, longitudinal analyses found that the intervention prevented rises in citations involving subjective judgments (e.g., "insubordination") within 6th and 7th grades. It also forestalled the emergence of worries about being seen stereotypically by the end of 7th grade. Identity threat can give rise to cycles of interaction that are maladaptive for both teachers and students in school; targeted exercises can interrupt these cycles to improve disciplinary outcomes over years. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

In this work we investigate the dynamic communication behavior of parent and proxy applications, and investigate whether or not the dynamic communication behavior of the proxy matches that of its respective parent application. The idea of proxy applications is that they should match their parent well, and should exercise the hardware and perform similarly, so that from them lessons can be learned about how the HPC system and the application can best be utilized. We show here that some proxy/parent pairs do not need the extra detail of dynamic behavior analysis, while others can benefit from it, and through this we also identified a parent/proxy mismatch and improved the proxy application.

In this paper, we present a methodology for modeling the expected runtime of a job based on historical application data and data from the job itself. This estimation model is useful for both for HPC users and administrators as a metric to compare the actual job runtime to, thus establishing a measure of performance of the job. We used job data, system data, and hardware performance counters in a near-zero overhead manner to model and assess job performance, in particular whether or not the job runtime was in line with expectations from historical application performance. We show over three proxy applications and three real applications that our estimations are within 5% of actual performance.

Proxy applications, or proxies, are simple applications meant to exercise systems in a way that mimics real applications (their parents). However, characterizing the relationship between the behavior of parent and proxy applications is not an easy task. In prior work [1], we presented a data-driven methodology to characterize the relationship between parent and proxy applications based on collecting runtime data from both and then using data analytics to find their correspondence or divergence. We showed that it worked well for hardware counter data, but our initial attempt using MPI function data was less satisfactory. In this paper, we present an exploratory effort at making an improved quantification of the correspondence of communication behavior for proxies and their respective parent applications. We present experimental evidence of positive results using four proxy applications from the current ECP Proxy Application Suite and their corresponding parent applications (in the ECP application portfolio). Results show that each proxy analyzed is representative of its parent with respect to communication data. In conjunction with our method presented in [1] (correspondence between computation and memory, behavior), we get a strong understanding of how well a proxy predicts the comprehensive performance of its parent.

Proxy applications are a simplified means for stake-holders to evaluate how both hardware and software stacks might perform on the class of real applications that they are meant to model. However, characterizing the relationship between them and their behavior is not an easy task. We present a data-driven methodology for characterizing the relationship between real and proxy applications based on collecting runtime data from both and then using data analytics to find their correspondence and divergence. We use new capabilities for application-level monitoring within LDMS (Lightweight Distributed Monitoring System) to capture hardware performance counter and MPI-related data. To demonstrate the utility of this methodology, we present experimental evidence from two system platforms, using four proxy applications from the current ECP Proxy Application Suite and their corresponding parent applications (in the ECP application portfolio). Results show that each proxy analyzed is representative of its parent with respect to computation and memory behavior. We also analyze communication patterns separately using mpiP data and show that communication for these four proxy/parent pairs is also similar.

The advances in 3D circuit integration have reignited the idea of processing-in-memory (PIM). In this paper, we evaluate 3D mesh-based network on chip (NoC) for 3D-PIM systems with single and multiple network configurations. We study stacked mesh (S-Mesh), which is a mesh-bus hybrid architecture for 3D NoCs that connects vertically stacked 2D meshes through buses. Previous S-Mesh studies have not addressed the problems and modifications needed at the building blocks of the network. We explain in details the internal structure of the S-Mesh, as well as, the problems and possible solutions of connecting 2D meshes using vertical buses. Also, we evaluate the performance of 3D NoCs via two traffic patterns, one of which is a novel traffic pattern that better measures 3D-PIM systems performance. Finally, we use the Rodinia benchmarks to measure the performance under real workloads. We use DSENT to evaluate the power consumption. Our results show ~ 15% performance improvement for the S-Mesh under zero-load packet latency and ~ 11% lower average packet latency for the Rodinia benchmarks. Also, S-Mesh is the low power configuration with the router static power accountable for 90% of the total network power consumption.

Because data collection in HPC systems happens on the nodes and is easily related to the job running on the node, tools presenting the data and subsequent analyses to the user generally present them at the job level. Our position is that this is the wrong level of abstraction and thus limits the value of the analyses, often dissuading users from using any of the offered tools. In this paper we present the position that tools need to present analyses at the level users are interested in, which is their applications.

The research presented here investigates potential psychological and health consequences of concealing a chronic illness. Data were collected from 2,500 individuals living with multiple sclerosis (MS), as part of an ongoing longitudinal research project. Questions on identity concealment and psychosocial reserve (a broad measure of well‐being) were embedded in a semi‐annual national survey. Responses were linked to each participant's concurrent responses to questions about their disability status, and prospectively to the same measure of disability status 1 year later. Just over 16% of respondents indicated that it was mostly true to very true that they actively concealed their MS and most indicated at least some degree of concealment. For people at lower levels of disability, decisions to conceal or disclose were not related to their levels of psychosocial reserve. However, with rising disability, concealment predicted lower levels of psychosocial reserve. Concealment was also associated with improved disability status 1 year later. A mediation analysis suggests that this may be in part because people who concealed were more likely to be employed. Taken together, the current research adds to the evidence that consequences of concealment often may be multifaceted and depend on a variety of moderators, including degree of disability.

The advances on 3D circuit integration have reignited the idea of processing-in-memory (PIM). In this paper, we evaluate 3D mesh-based NoC design for 3D-PIM systems. We study the stacked mesh (S-Mesh) which is a mesh-bus hybrid architecture for 3D NoCs that connects vertically stacked 2D meshes through buses. Previous S-Mesh studies have not addressed the problems and modifications needed at the building blocks of the network. We explain in details the internal structure of the S-Mesh, as well as, the problems and solutions of connecting 2D meshes using vertical buses. Also, we evaluate the performance of 3D NoC designs via two traffic patterns, one of which is a novel traffic pattern that better measures 3D-PIM systems performance. Our results show 15% performance improvement for the S-Mesh for zero-load packet latency while having a negligible decrease in saturation throughput.

Little is known about social identity threat from religion or religiosity. We collected data from a diverse sample of Protestants, Catholics, Jews, and Muslims across the United States ( N = 970) to test whether, and for whom, religion and religiosity, like other social identities, can be consequential sources of identity threat. Results suggest that religious threat is highest among religious minority groups (Muslims and Jews) and highly religious Protestants. Threat predicted (1) lower belonging, (2) a greater propensity to conceal one’s religion, and (3) more intergroup bias, although these patterns varied somewhat by religion. Results illuminate how a broader social climate in which religion and specific religious groups are often the subject of heated rhetoric may trigger identity threat and exacerbate intergroup hostilities.