With increasing data production from scientific checkpoint-restart (CR) workloads, data management libraries utilize asynchronous I/O operations to hide the cost of accessing global file systems in high-performance computing (HPC) systems. However, the characteristics of asynchronous operation, such as software and hardware architecture and workload features, dictate the impact of asynchronous I/O operations on checkpoint-restart application resources and asynchronous I/O performance. In this work, we aim to study the impact of these design choices on CR performance, such as CPU, memory, and checkpointing, while also considering the performance of asynchronous I/O operation and the performance variability introduced in the workload. We observe the following three main findings. First, the architecture used for asynchronous I/O must be flexible to manage its resources to reduce application interference while maximizing asynchronous I/O performance. Second, modern MPI-aware schedulers apply an implicit core affinity to each workload, which is helpful for compute-bound applications but can significantly reduce checkpointing performance. Finally, asynchronous I/O designs need to consider the cost of communication between the service and application to reduce the overhead of checkpointing operations. Through this study, we aim to pave the path for future middleware libraries to adopt different designs based on their target workload and HPC system architecture.

We introduce UnifyFS, a user-level file system that aggregates node-local storage tiers available on high performance computing (HPC) systems and makes them available to HPC applications under a unified namespace. UnifyFS employs transparent I/O interception, so it does not require changes to application code and is compatible with commonly used HPC I/O libraries. The design of UnifyFS supports the predominant HPC I/O workloads and is optimized for bulk-synchronous I/O patterns. Furthermore, UnifyFS provides customizable file system semantics to flexibly adapt its behavior for diverse I/O workloads and storage devices. In this paper, we discuss the unique design goals and architecture of UnifyFS and evaluate its performance on a leadership-class HPC system. In our experimental results, we demonstrate that UnifyFS exhibits excellent scaling performance for write operations and can improve the performance of application checkpoint operations by as much as 3× versus a tuned configuration.

Scientific research and development campaigns are materialized by workflows of applications executing on high-performance computing (HPC) systems. These applications con-sist of tasks that can have inter- or intra-application flows of data to achieve the research goals successfully. These dataflows create dependencies among the tasks and cause resource con-tention on shared storage systems, thus limiting the aggregated I/O bandwidth achieved by the workflow. However, these I/O performance issues are often solved by tedious and manual efforts that demand holistic knowledge about the data dependencies in the workflow and the information about the infrastructure being utilized. Taking this into consideration, we design DFMan, a graph-based dataflow management and optimization framework for maximizing I/O bandwidth by leveraging the powerful storage stack on HPC systems to manage data sharing optimally among the tasks in the workflows. In particular, we devise a graph-based optimization algorithm that can leverage an intuitive graph representation of dataflow- and system-related information, and automatically carry out co-scheduling of task and data placement. According to our experiments, DFMan optimizes a wide variety of scientific workflows such as Hurricane 3D on Cloud Model 1 (CM1), Montage Carina Nebula (NGC3372), and an emulated dataflow kernel of the Multiscale Machine-learned Modeling Infrastructure (MuMMI I/O) on the Lassen supercomputer, and improves their aggregated I/O bandwidth by up to 5.42 x, 2.12 x and 1.29 x, respectively, compared to the baseline bandwidth.

Checkpointing large amounts of related data concurrently to stable storage is a common I/O pattern of many HPC applications. However, such a pattern frequently leads to I/O bottlenecks that lead to poor scalability and performance. As modern HPC infrastructures continue to evolve, there is a growing gap between compute capacity vs. I/O capabilities. Furthermore, the storage hierarchy is becoming increasingly heterogeneous: in addition to parallel file systems, it comprises burst buffers, key-value stores, deep memory hierarchies at node level, etc. In this context, state of art is insufficient to deal with the diversity of vendor APIs, performance and persistency characteristics. This extended abstract presents an overview of VeloC (Very Low Overhead Checkpointing System), a checkpointing runtime specifically design to address these challenges for the next generation Exascale HPC applications and systems. VeloC offers a simple API at user level, while employing an advanced multi-level resilience strategy that transparently optimizes the performance and scalability of checkpointing by leveraging heterogeneous storage.

Water quality is a growing area of research, with more and more focus in the UK and globally on environmental issues and water quality. Current methods of monitoring environmental data such as air quality have continued to develop, spurred on by the growth of the Internet of Things. However, water quality monitoring mainly still depends on manual sample collection. This research presents the first implementation of a multi-hop underwater radio frequency sensor network using bowtie antennas combined with the 433 MHz frequency and a controlled flooding routing approach. The experimental work was conducted in the water reservoir and demonstrates the potential of multi-hop routing in underwater sensor networks to extend range to 19 meters as well as improvements on communication distances from 7 meters previously to 17 meters using radio frequency communications in an underwater environment. Simulated results show that the experimental platform used could enable the long-term deployment of an underwater wireless sensor network that used RF for periods of over a year with support for a 100 sensor node network broadcasting twice daily remaining active for 418 days or a 100 sensor node network broadcasting hourly remaining active for 406 days before any node deaths.

Scientific application workflows leverage the capabilities of cutting-edge high-performance computing (HPC) facilities to enable complex applications for academia, research, and industry communities. Data transfer and I/O dependency among different modules of modern HPC workflows can increase the complexity and hamper the overall performance of workflows. Understanding this complexity due to data-dependency and dataflow is an essential prerequisite for developing optimization strategies to improve I/O performance and, eventually, the entire workflow. In this paper, we discuss dataflow patterns for workflow applications on HPC systems. As existing I/O benchmarking tools lack in identifying and representing the dataflow in modern HPC workflows, we have implemented Wemul, an open-source workflow I/O emulation framework, to mimic different types of I/O behavior demonstrated by common and complex HPC application workflows for deeper analysis. We elaborate on the features and usage of Wemul, demonstrate its application to HPC workflows, and discuss the insights from the performance analysis results on Lassen supercomputing cluster at Lawrence Livermore National Laboratory (LLNL).

The processor performance of high performance computing (HPC) systems is increasing at a much higher rate than storage performance. This imbalance leads to I/O performance bottlenecks in massively parallel HPC applications. Therefore, there is a need for improvements in storage and file system designs to meet the ever-growing I/O needs of HPC applications. Storage and file system designers require a deep understanding of how HPC application I/O behavior affects current storage system installations in order to improve them. In this work, we contribute to this understanding using application-agnostic file system statistics gathered on compute nodes as well as metadata and object storage file system servers. We analyze file system statistics of more than 4 million jobs over a period of three years on two systems at Lawrence Livermore National Laboratory that include a 15 PiB Lustre file system for storage. The results of our study add to the state-of-the-art in I/O understanding by providing insight into how general HPC workloads affect the performance of large-scale storage systems. Some key observations in our study show that reads and writes are evenly distributed across the storage system; applications which perform I/O, spread that I/O across ∼78% of the minutes of their runtime on average; less than 22% of HPC users who submit write-intensive jobs perform efficient writes to the file system; and I/O contention seriously impacts I/O performance.

Data analysis, and particularly big data analysis and cluster determination is becoming an increasing challenge in the field of biomolecular simulation. We present our ongoing efforts on identification of orientational states of RAS proteins and associated lipids using Machine Learning-based models. With the advent of deep learning based approaches, it has become possible to use the raw coordinates of molecules to identify their states and their corresponding stability. We employ variational autoencoders using deep neural networks to encode the molecular coordinates of the RAS dimer and the interacting lipid density fields into a meaningful latent space. Our training data comes from a massive simulation campaign of 120,000 coarse-grained MD simulations each over 1 microsecond long, run using our MuMMI framework that on up to 4000 nodes of the Sierra supercomputer at LLNL. We train the neural network using NVIDIA Tesla V100 GPUs to reduce the spatial coordinates into a low-dimensional latent space. We perform spectral clustering in the latent space to determine, in an unsupervised manner, the number of distinct clusters—each corresponding to a distinct state of the RAS dimer. If distinct clusters are successfully identified then they are analyzed further for their geometrical information and lifetime to understand their stability. We believe that meaningful state identification in RAS dimers and associated lipids using ML-based unsupervised methods can provide key insights for experiments facilitating therapeutic strategies. This work has been supported in part by the Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) program established by the U.S. Department of Energy (DOE) and the National Cancer Institute (NCI) of the National Institutes of Health. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC5207NA27344

On large-scale high performance computing (HPC) systems, applications are provisioned with aggregated resources to meet their peak demands for brief periods. This results in resource underutilization because application requirements vary a lot during execution. This problem is particularly pronounced for deep learning applications that are running on leadership HPC systems with a large pool of burst buffers in the form of flash or non-volatile memory (NVM) devices. In this paper, we examine the I/O patterns of deep neural networks and reveal their critical need of loading many small samples randomly for successful training. We have designed a specialized Deep Learning File System (DLFS) that provides a thin set of APIs. Particularly, we design the metadata management of DLFS through an in-memory tree-based sample directory and its file services through the user-level SPDK protocol that can disaggregate the capabilities of NVM Express (NVMe) devices to parallel training tasks. Our experimental results show that DLFS can dramatically improve the throughput of training for deep neural networks on NVMe over Fabric, compared with the kernel-based Ext4 file system. Furthermore, DLFS achieves efficient user-level storage disaggregation with very little CPU utilization.

In this work, we perform analysis of detection and counting of cars using a low-power IBM TrueNorth Neurosynaptic System. For our evaluation we looked at a publicly-available dataset that has overhead imagery of cars with context present in the image. The trained neural network for image analysis was deployed on the NS16e system using IBM's EEDN training framework. Through multiple experiments we identify the architectural bottlenecks present in TrueNorth system that does not let us deploy large neural network structures. Following these experiments we propose changes to CNN model to circumvent these architectural bottlenecks. The results of these evaluations have been compared with caffe-based implementations of standard neural networks that were deployed on a Titan-X GPU. Results showed that TrueNorth can detect cars from the dataset with 97.60% accuracy and can be used to accurately count the number of cars in the image with 69.04% accuracy. The car detection accuracy and car count (-/+ 2 error margin) accuracy are comparable to high-precision neural networks like AlexNet, GoogLeNet, and ResCeption, but show a manifold improvement in power consumption.

In 2016, there were 317 serious water pollution incidents in the UK, with 78,000 locations where businesses discharge controlled quantities of pollutants into rivers; therefore, continuous monitoring is vital. Since 1998, the environment agency has taken over 50 million water samples for water quality monitoring. The Internet of Things has grown phenomenally in recent years, reaching all aspects of our lives, many of these connected devices use wireless sensor networks to relay data to internet-connected nodes, where data can be processed, analyzed and consumed. However, Underwater wireless communications rely mainly on alternative communication methods such as optical and acoustic, with radio frequencies being an under-exploited method. This research presents real world results conducted in the Leeds and Liverpool Canal for the novel use of the 433 MHz radio frequency combined with a bowtie antenna in underwater communications in raw water, achieving distances of 7 m at 1.2 kbps and 5 m at 25 kbps.

Global checkpointing to external storage (e.g., a parallel file system) is a common I/O pattern of many HPC applications. However, given the limited I/O throughput of external storage, global checkpointing can often lead to I/O bottlenecks. To address this issue, a shift from synchronous checkpointing (i.e., blocking until writes have finished) to asynchronous checkpointing (i.e., writing to faster local storage and flushing to external storage in the background) is increasingly being adopted. However, with rising core count per node and heterogeneity of both local and external storage, it is non-trivial to design efficient asynchronous checkpointing mechanisms due to the complex interplay between high concurrency and I/O performance variability at both the node-local and global levels. This problem is not well understood but highly important for modern supercomputing infrastructures. This paper proposes a versatile asynchronous checkpointing solution that addresses this problem. To this end, we introduce a concurrency-optimized technique that combines performance modeling with lightweight monitoring to make informed decisions about what local storage devices to use in order to dynamically adapt to background flushes and reduce the checkpointing overhead. We illustrate this technique using the VeloC prototype. Extensive experiments on a pre-Exascale supercomputing system show significant benefits.

Parallel File Systems (PFSs) are frequently deployed on leadership High Performance Computing (HPC) systems to ensure efficient I/O, persistent storage and scalable performance. Emerging Deep Learning (DL) applications incur new I/O and storage requirements to HPC systems with batched input of small random files. This mandates PFSs to have commensurate features that can meet the needs of DL applications. BeeGFS is a recently emerging PFS that has grabbed the attention of the research and industry world because of its performance, scalability and ease of use. While emphasizing a systematic performance analysis of BeeGFS, in this paper, we present the architectural and system features of BeeGFS, and perform an experimental evaluation using cutting-edge I/O, Metadata and DL application benchmarks. Particularly, we have utilized AlexNet and ResNet-50 models for the classification of ImageNet dataset using the Livermore Big Artificial Neural Network Toolkit (LBANN), and ImageNet data reader pipeline atop TensorFlow and Horovod. Through extensive performance characterization of BeeGFS, our study provides a useful documentation on how to leverage BeeGFS for the emerging DL applications.

Geosmin contamination in water is a leading cause of odor related complaints to water companies in UK, tainting water with an earthy smell that is detectable by humans in quantities as low as 4 nanograms per liter. Current Geosmin detection methods depend on lab-based equipment, requiring samples to be collected and transported before Geosmin can be tested. This research presents a novel method for the detection of Geosmin in water using Microwave spectroscopy capable of detecting differentiating between four levels of Geosmin contamination: 5 ng/L, 10 ng/L, 0.5 mg/L and 1 mg/L as well as control samples. Frequencies within the 5.4 GHz to 5.9, 6.4 GHz to 6.5 GHz and 7.2 GHz to 7.5 GHz ranges showed significant separation between the sample classes.

Developing a file system is a challenging task, especially a kernel-level file system. User-level file systems alleviate the burden and development complexity associated with kernel-level implementations. The Filesystem in Userspace (FUSE) is a widely used tool that allows non-privileged users to develop file systems in user space. When a FUSE file system is mounted, it runs as a user-level process. Application programs and FUSE file system processes are bridged through FUSE kernel module. However, as the FUSE kernel module transfers requests between an application program and a file system process, the overheads in a FUSE file system call from crossing the user-kernel boundary is non-trivial. The overheads contain user-kernel mode switches, context switches, and additional memory copies. In this paper, we describe our Direct-FUSE framework that supports multiple FUSE file systems as well as other, custom user-level file systems in user space without the need to cross the user/kernel boundary into the FUSE kernel module. All layers of Direct-FUSE are in user space, and applications can directly use pre-defined unified file system calls to interact with different user-defined file systems. Our performance results show that Direct-FUSE can outperform some native FUSE file systems by 11.9% on average and does not add significant overhead over backend file systems.

Deep learning, and in particular Recurrent Neural Networks (RNN) have shown superior accuracy in a large variety of tasks including machine translation, language understanding, and movie frame generation. However, these deep learning approaches are very expensive in terms of computation. In most cases, Graphic Processing Units (GPUs) are in used for large scale implementations. Meanwhile, energy efficient RNN approaches are proposed for deploying solutions on special purpose hardware including Field Programming Gate Arrays (FPGAs) and mobile platforms. In this paper, we propose an effective quantization approach for Recurrent Neural Networks (RNN) techniques including Long Short Term Memory (LSTM), Gated Recurrent Units (GRU), and Convolutional Long Short Term Memory (ConvLSTM). We have implemented different quantization methods including Binary Connect {-1, 1}, Ternary Connect {-1, 0, 1}, and Quaternary Connect {-1, -0.5, 0.5, 1}. These proposed approaches are evaluated on different datasets for sentiment analysis on IMDB and video frame predictions on the moving MNIST dataset. The experimental results are compared against the full precision versions of the LSTM, GRU, and ConvLSTM. They show promising results for both sentiment analysis and video frame prediction.

Wireless sensor networks have thrived over recent years, with conventional wireless sensor networks becoming an ever-growing area of research. However, underwater wireless sensor networks are underdeveloped in comparison with wireless sensor networks. This paper looks at several important considerations for wireless sensor networks and underwater sensor networking platform targeted at environmental monitoring. Many requirements are transferable from conventional wireless sensor networks and this paper looks at how these challenges have been tackled. However, considerations such as energy efficiency hold a greater weight where sensor nodes are more difficult to retrieve, consideration must also be given to other factors such as the impact of communication technologies, communication distances, data rates and ease of deployment.