MRI is the most effective method for screening high-risk breast cancer patients. While current exams rely on the qualitative evaluation of morphological features before and after contrast administration and less on contrast kinetic information, recent developments in fast acquisition methods aim to combine both. However, balancing spatial resolution, temporal resolution and scan time poses a considerable challenge in dynamic MRI. Here, we introduce a radial MRI reconstruction framework for Dynamic Contrast Enhanced (DCE) imaging, termed Enhanced Locally low-rank Imaging for Tissue contrast Enhancement (ELITE), to address these limitations. ELITE combines locally low-rank subspace modeling to capture spatially localized tissue dynamics with deep learning. We evaluate its effectiveness using the publicly available fastMRI breast initiative, demonstrating substantial improvements in CNR and noise reduction while enabling flexible temporal resolution down to 1 second. ELITE also shows benefits in neck and brain imaging, making it a viable alternative for other DCE-MRI applications.

We present our work on developing and training scalable, trustworthy, and energy-efficient predictive graph foundation models (GFMs) using HydraGNN, a multi-headed graph convolutional neural network architecture. HydraGNN expands the boundaries of graph neural network (GNN) computations in both training scale and data diversity. It abstracts over message passing algorithms, allowing both reproduction of and comparison across algorithmic innovations that define nearest-neighbor convolution in GNNs. This work discusses a series of optimizations that have allowed scaling up the GFMs training to tens of thousands of GPUs on datasets consisting of hundreds of millions of graphs. Our GFMs use multitask learning (MTL) to simultaneously learn graph-level and node-level properties of atomistic structures, such as energy and atomic forces. Using over 154 million atomistic structures for training, we illustrate the performance of our approach along with the lessons learned on two state-of-the-art US Department of Energy (US-DOE) supercomputers, namely the Perlmutter petascale system at the National Energy Research Scientific Computing Center and the Frontier exascale system at Oak Ridge Leadership Computing Facility. The HydraGNN architecture enables the GFM to achieve near-linear strong scaling performance using more than 2000 GPUs on Perlmutter and 16,000 GPUs on Frontier.

Mainstream key-value stores adopt the write-friendly log-structured merge (LSM) tree as the underlying storage engine. However, the LSM tree is suboptimal for read operations as key-value pairs are not allowed to be promoted from a lower level to an upper level in the storage hierarchy. This leads to drops in read QPS when key-value pairs at a low level are frequently accessed due to an increase in file I/Os and CPU cycles for multi-level search operations. To address this problem of slow reads, we propose SkipLSM, an LSM tree-based key-value store augmented with a caching layer for hot keys on a high-performance NVMe SSD. SkipLSM enables fast retrieval of hot key-value pairs by leveraging a DRAM-SSD cache hierarchy. To reduce the I/O overhead of caching, SkipLSM introduces a lightweight storage manager which builds on a user-level NVMe driver. We implement SkipLSM on RocksDB, a widely used LSM tree-based storage engine, as a lightweight plug-in. Our evaluation of SkipLSM with four read-intensive workloads shows that SkipLSM increases the QPS by 42.0%, 81.1%, and 46.4% for YCSB B, C, and D, respectively, and 94.0% for the PreDist workload of Facebook MixGraph benchmark over the vanilla RocksDB.

MRI is the most effective method for screening high-risk breast cancer patients. While current exams primarily rely on the qualitative evaluation of morphological features before and after contrast administration and less on contrast kinetic information, the latest developments in acquisition protocols aim to combine both. However, balancing between spatial and temporal resolution poses a significant challenge in dynamic MRI. Here, we propose a radial MRI reconstruction framework for Dynamic Contrast Enhanced (DCE) imaging, which offers a joint solution to existing spatial and temporal MRI limitations. It leverages a locally low-rank (LLR) subspace model to represent spatially localized dynamics based on tissue information. Our framework demonstrated substantial improvement in CNR, noise reduction and enables a flexible temporal resolution, ranging from a few seconds to 1-second, aided by a neural network, resulting in images with reduced undersampling penalties. Finally, our reconstruction framework also shows potential benefits for head and neck, and brain MRI applications, making it a viable alternative for a range of DCE-MRI exams.

In this work, we propose MDLoader, a hybrid in-memory data loader for distributed deep neural networks. MDLoader introduces a model-driven performance estimator to automatically switch between one-sided and collective communication at runtime.

Background and Purpose The purpose of this study is to evaluate the feasibility of using 3-dimensional (3D) ultra-short echo time (UTE) radial imaging method for measurement of the permeability of the blood-brain barrier (BBB) to gadolinium-based contrast agent. In this study, we propose to use the golden-angle radial sparse parallel (GRASP) method with 3D center-out trajectories for UTE, hence named as 3D UTE-GRASP. We first examined the feasibility of using 3D UTE-GRASP dynamic contrast-enhanced (DCE)-magnetic resonance imaging (MRI) for differentiating subtle BBB disruptions induced by focused ultrasound (FUS). Then, we examined the BBB permeability changes in Alzheimer's disease (AD) pathology using Alzheimer's disease transgenic mice (5xFAD) at different ages. Methods For FUS experiments, we used four Sprague Dawley rats at similar ages where we compared BBB permeability of each rat receiving the FUS sonication with different acoustic power (0.4-1.0 MPa). For AD transgenic mice experiments, we included three 5xFAD mice (6, 12, and 16 months old) and three wild-type mice (4, 8, and 12 months old). Results The result from FUS experiments showed a progressive increase in BBB permeability with increase of acoustic power (p < .05), demonstrating the sensitivity of DCE-MRI method for detecting subtle changes in BBB disruption. Our AD transgenic mice experiments suggest an early BBB disruption in 5xFAD mice, which is further impaired with aging. Conclusion The results in this study substantiate the feasibility of using the proposed 3D UTE-GRASP method for detecting subtle BBB permeability changes expected in neurodegenerative diseases, such as AD.

Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83x and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.

Today's deep neural network (DNN) training pipeline utilizes hardware resources holistically, including host CPUs and storage devices for preprocessing the input data and accelerators like GPUs for computing gradients. As the performance of the accelerator scales rapidly, the frontend data preparation stages are becoming a new performance bottleneck to yield suboptimal training throughput. Since the bottleneck in the pipeline may vary depending on hardware configurations, DNN models, and datasets, overprovisioning hardware resources for data preparation such as CPU cores and disk bandwidth is not a cost-effective solution. Instead, we make a case for leveraging multiple data formats, possibly with opposing characteristics in resource utilization, to balance the training pipeline. This idea is realized by Liquid, a new system for building an efficient training pipeline with multi-format datasets. Our evaluation on three distinct execution environments demonstrates that Liquid achieves up to 3.05x and 1.54x higher data preparation throughput on Cityscapes/CityPersons (PNG) and ImageNet (JPEG) datasets, respectively, over the baseline single-format pipeline. This leads up to 2.02x and 1.25x higher end-to-end geomean training throughput with no accuracy drop.

Today's deep neural network (DNN) training pipeline utilizes hardware resources holistically, including host CPUs and storage devices for preprocessing the input data and accelerators like GPUs for computing gradients. As the performance of the accelerator scales rapidly, the frontend data preparation stages are becoming a new performance bottleneck to yield suboptimal training throughput. Since the bottleneck in the pipeline may vary depending on hardware configurations, DNN models, and datasets, overprovisioning hardware resources for data preparation such as CPU cores and disk bandwidth is not a cost-effective solution. Instead, we make a case for leveraging multiple data formats, possibly with opposing characteristics in resource utilization, to balance the training pipeline. This idea is realized by Liquid, a new system for building an efficient training pipeline with multi-format datasets. Our evaluation on three distinct execution environments demonstrates that Liquid achieves up to 3.05x and 1.54x higher data preparation throughput on Cityscapes/CityPersons (PNG) and ImageNet (JPEG) datasets, respectively, over the baseline single-format pipeline. This leads up to 2.02x and 1.25x higher end-to-end geomean training throughput with no accuracy drop.

Graph neural networks (GNNs) are a class of Deep Learning models used in designing atomistic materials for effective screening of large chemical spaces. To ensure robust prediction, GNN models must be trained on large volumes of atomistic data on leadership class supercomputers. Even with the advent of modern architectures that consist of multiple storage layers that include node-local NVMe devices in addition to device memory for caching large datasets, extreme-scale model training faces I/O challenges at scale. We present DDStore, an in-memory distributed data store designed for GNN training on large-scale graph data. DDStore provides a hierarchical, distributed, data caching technique that combines data chunking, replication, low-latency random access, and high throughput communication. DDStore achieves near-linear scaling for training a GNN model using up to 1000 GPUs on the Summit and Perlmutter supercomputers, and reaches up to a 6.15x reduction in GNN training time compared to state-of-the-art methodologies.

This study explores two different approachs of measuring subtle BBB disruption. To induce different levels of BBB disruption, we used a focused ultrasound (FUS) sonication with intravenously injected microbubbles with an animal model. Each animal underwent FUS procedure with different acoustic power levels. We compared the changes measured using DCE-MRI with Gadolinium-based contrast agent for volume transfer rate constant and Ferumoxytol-based ACE-MRI to measure transendotheliel water exchange rate. Our results suggest that both the water exchange rate and the contrast exchange rate show sensitive detection of subtle BBB disruption, which could shed light on understanding different permeability changes in BBB.

While the globally low-rank (GLR) model has been demonstrated to be effective in representing global contrast change, it is expected to be less effective for spatially localized signal dynamics. In this work, we propose an improved reconstruction framework, which extends GRASP-Pro using a locally low-rank (LLR) model to represent spatially localized dynamics based on clusters or patches information. This approach has been tested in multiple DCE applications including both cancer and healthy subjects. In addition, we propose an anatomical cluster-based reconstruction approach for brain DCE-MRI.

The training process of deep neural networks (DNNs) is usually pipelined with stages for data preparation on CPUs followed by gradient computation on accelerators like GPUs. In an ideal pipeline, the end-to-end training throughput is eventually limited by the throughput of the accelerator, not by that of data preparation. In the past, the DNN training pipeline achieved a near-optimal throughput by utilizing datasets encoded with a lightweight, lossy image format like JPEG. However, as high-resolution, losslessly-encoded datasets become more popular for applications requiring high accuracy, a performance problem arises in the data preparation stage due to low-throughput image decoding on the CPU. Thus, we propose L3, a custom lightweight, lossless image format for high-resolution, high-throughput DNN training. The decoding process of L3 is effectively parallelized on the accelerator, thus minimizing CPU intervention for data preparation during DNN training. L3 achieves a 9.29x higher data preparation throughput than PNG, the most popular lossless image format, for the Cityscapes dataset on NVIDIA A100 GPU, which leads to 1.71x higher end-to-end training throughput. Compared to JPEG and WebP, two popular lossy image formats, L3 provides up to 1.77x and 2.87x higher end-to-end training throughput for ImageNet, respectively, at equivalent metric performance.

Dynamic contrast enhancement (DCE) MRI has been increasingly utilized in clinical practice. While machine learning (ML) applications are gaining momentum in MRI reconstruction, the dynamic nature of image acquisition for DCE-MRI limits access to a simultaneously high spatial and temporal resolution ground truth image for supervised ML applications. In this study, we introduced a pipeline to simulate the ground truth DCE-MRI k-space data from real breast perfusion images. Based on physical model and the clinical images, we estimate the perfusion parameters. Treating those as ground truth, we simulated the signal. Using our simulated images, we trained ML reconstruction models. We demonstrate the utility of our simulation pipeline using two ML models and one conventional reconstruction method. Our results suggest that, even though the image quality of the ML reconstructions seem to be very close to the simulated ground truth, the temporal pattern and its kinetic parameters may not be close to the ground truth data.

To meet surging demands for deep learning inference services, many cloud computing vendors employ high-performance specialized accelerators, called neural processing units (NPUs). One important challenge for effective use of NPUs is to achieve high resource utilization over a wide spectrum of deep neural network (DNN) models with diverse arithmetic intensities. There is often an intrinsic mismatch between the compute-to-memory bandwidth ratio of an NPU and the arithmetic intensity of the model it executes, leading to under-utilization of either compute resources or memory bandwidth. Ideally, we want to saturate both compute TOP/s and DRAM bandwidth to achieve high system throughput. Thus, we propose Layerweaver, an inference serving system with a novel multi-model time-multiplexing scheduler for NPUs. Layerweaver reduces the temporal waste of computation resources by interweaving layer execution of multiple different models with opposing characteristics: compute-intensive and memory-intensive. Layerweaver hides the memory time of a memory-intensive model by overlapping it with the relatively long computation time of a compute-intensive model, thereby minimizing the idle time of the computation units waiting for off-chip data transfers. For a two-model serving scenario of batch 1 with 16 different pairs of compute- and memory-intensive models, Layerweaver improves the temporal utilization of computation units and memory channels by 44.0% and 28.7%, respectively, to increase the system throughput by 60.1% on average, over the baseline executing one model at a time.

Data augmentation tuned to datasets and tasks has had great success in various AI applications, such as computer vision, natural language processing, autonomous driving, and bioinformatics. However, most of the specific parameter-based augmentation strategies are inefficient in finding suitable augmentation parameters to improve the model performance whenever the dataset changes. We introduce a dynamic data augmentation strategy called Faster and Smarter AutoAugment (FSAA) that separates the data augmentation method through the initial policy search result. Based on our policy branching principle, the augmentation policy dynamically clusters data points within the dataset according to the degree of performance change at the data split stage. With extensive experimentation on various datasets for image recognition task, we show that FSAA dramatically reduces the GPU computation cost, a problem in the existing automatic data augmentation strategies, by up to over 90%. It also ensures diversity and generalization of the dataset augmentation by searching for more diverse policies than existing autonomous methods in less time, resulting in consistent performance gains and robustness across multiple models.

The explosive expansion of Deep Neural Networks (DNN) model size expedites the need for larger memory capacity. This movement is particularly true for models in natural language processing (NLP), a dominant application of AI along with computer vision. For example, a recent extreme-scale language model GPT-3 from OpenAI has over 175 billion parameters. Furthermore, such a model mostly consists of FC layers with huge dimensions, and thus has a relatively high arithmetic intensity. In that sense, an extreme-scale language model does not suit well to the conventional HBM DRAM-based memory system that lacks capacity and offers extremely high bandwidth. For this reason, we propose to pair the neural network training accelerator with the flash-based memory system instead of the HBM DRAM-based memory system. To design the effective flash-based memory system, we optimize the existing SSD design to improve the SSD bandwidth as well as endurance. Finally, we evaluate our proposed platform, and show that Behemoth achieves 3.65× cost saving over TPU v3 and 2.05× training throughput improvement over the accelerator attached to a commercial SSD.

With mobile applications' ever-increasing demands for memory capacity, along with a steady increase in the number of applications running concurrently, memory capacity is becoming a scarce resource on mobile devices. When the memory pressure is high, current mobile OSes often kill application processes that have not been used recently to reclaim memory space. This leads to a long delay when a user relaunches the killed application, which degrades the user experience. Even if this mechanism is disabled to utilize a compression-based in-memory swap mechanism, relaunching the application still incurs a substantial latency penalty as it requires the decompression of compressed anonymous pages and a stream of I/O accesses to retrieve file-backed pages into memory. This paper identifies conventional demand paging as the primary source of this inefficiency and proposes ASAP, a mechanism for fast application switch via adaptive prepaging on mobile devices. ASAP performs prepaging by combining i) high-precision switch footprint estimators for both file-backed and anonymous pages, and ii) efficient implementation of the prepaging mechanism to minimize resource waste for CPU cycles and disk bandwidth during an application switch. Our evaluation using eight real-world applications on Google Pixel 4 and Pixel 3a demonstrates that ASAP can reduce the switch time by 22.2% and 28.3% on average, respectively (with a maximum of 33.3% and 35.7%, respectively), over the vanilla Android 10.

Deep neural networks (DNNs) are widely used in various AI application domains such as computer vision, natural language processing, autonomous driving, and bioinformatics. As DNNs continue to get wider and deeper to improve accuracy, the limited DRAM capacity of a training platform like GPU often becomes the limiting factor on the size of DNNs and batch size-called memory capacity wall. Since increasing the batch size is a popular technique to improve hardware utilization, this can yield a suboptimal training throughput. Recent proposals address this problem by offloading some of the intermediate data (e.g., feature maps) to the host memory. However, they fail to provide robust performance as the training process on a GPU contends with applications running on a CPU for memory bandwidth and capacity. Thus, we propose FlashNeuron, the first DNN training system using an NVMe SSD as a backing store. To fully utilize the limited SSD write bandwidth, FlashNeuron introduces an offloading scheduler, which selectively offloads a set of intermediate data to the SSD in a compressed format without increasing DNN evaluation time. FlashNeuron causes minimal interference to CPU processes as the GPU and the SSD directly communicate for data transfers. Our evaluation of FlashNeuron with four state-of-the-art DNNs shows that FlashNeuron can increase the batch size by a factor of 12.4x to 14.0x over the maximum allowable batch size on NVIDIA Tesla V100 GPU with 16GB DRAM. By employing a larger batch size, FlashNeuron also improves the training throughput by up to 37.8% (with an average of 30.3%) over the baseline using GPU memory only, while minimally disturbing applications running on CPU.