In-situ analytics have recently emerged as a promising approach to reduce I/O, network, and storage congestion for scientific data analysis. At the same time, Mean Time To Failure (MTTF) has also continuously decreased for HPC clusters, making failures during the execution of a simulation more likely. It is difficult to apply Checkpointing and Restart (C/R) to a simulation executed with in-situ analytics, because we not only need to store analytic and simulation states, but also need to ensure consistency between them.To solve the above problem, we present a novel solution to apply C/R approach. The solution exploits a common property of analytics applications, which is that they fit the reduction-style processing pattern. Based on the observation that the global reduction result is correct if all local reduction objects have been processed, only global reduction objects and progress of the reduction procedure is checkpointed, together with an application-level checkpoint of the simulation. We also discuss the specific details for handling time-sharing and space sharing modes, which are suitable for multi-core and many-core environments, respectively.We have evaluated our approach in the context of Smart middleware for developing in-situ analytics, which provides a Map-Reduce like API. We demonstrate the low overheads of our approach with different scientific simulations and analytics tasks.
A key advantage of infrastructure-as-a-service (IaaS) clouds is providing users on-demand access to resources. To provide on-demand access, however, cloud providers must either significantly overprovision their infrastructure (and pay a high price for operating resources with low utilization) or reject a large proportion of user requests (in which case the access is no longer on-demand). At the same time, not all users require truly on-demand access to resources. Many applications and workflows are designed for recoverable systems where interruptions in service are expected. For instance, many scientists utilize high-throughput computing (HTC)-enabled resources, such as Condor, where jobs are dispatched to available resources and terminated when the resource is no longer available. We propose a cloud infrastructure that combines on-demand allocation of resources with opportunistic provisioning of cycles from idle cloud nodes to other processes by deploying backfill virtual machines (VMs). For demonstration and experimental evaluation, we extend the Nimbus cloud computing toolkit to deploy backfill VMs on idle cloud nodes for processing an HTC workload. Initial tests show an increase in IaaS cloud utilization from 37.5% to 100% during a portion of the evaluation trace but only 6.39% overhead cost for processing the HTC workload. We demonstrate that a shared infrastructure between IaaS cloud providers and an HTC job management system can be highly beneficial to both the IaaS cloud provider and HTC users by increasing the utilization of the cloud infrastructure (thereby decreasing the overall cost) and contributing cycles that would otherwise be idle to processing HTC jobs.
Large scale data processing is increasingly common in cloud computing systems like MapReduce, Hadoop, and Dryad in recent years. In these systems, files are split into many small blocks and all blocks are replicated over several servers. To process files efficiently, each job is divided into many tasks and each task is allocated to a server to deals with a file block. Because network bandwidth is a scarce resource in these systems, enhancing task data locality(placing tasks on servers that contain their input blocks) is crucial for the job completion time. Although there have been many approaches on improving data locality, most of them either are greedy and ignore global optimization, or suffer from high computation complexity. To address these problems, we propose a heuristic task scheduling algorithm called Balance-Reduce(BAR), in which an initial task allocation will be produced at first, then the job completion time can be reduced gradually by tuning the initial task allocation. By taking a global view, BAR can adjust data locality dynamically according to network state and cluster workload. The simulation results show that BAR is able to deal with large problem instances in a few seconds and outperforms previous related algorithms in term of the job completion time.
This paper describes the management of files distributed in heterogeneous Data Grids by using RNS (Resource Namespace Service). RNS provides hierarchical namespace management for name-to-resource mapping as a key technology to use Grid resources for different kinds of middleware. RNS directory entries and junction entries can contain their own XML messages as metadata. We define attribute expressions in XML for the RNS entries and give an algorithm to access distributed files stored within different kinds of Data Grids. The example in this paper shows how our Grid application can retrieve the actual locations of files from the RNS server. An application can also access the distributed files as though they were files in the local file system without worrying about the underlying Data Grids. This approach can be used in a Grid computing system to handle distributed Grid resources.
Distributed Shared Memory (DSM) machines are a wide class of multi-processor computing systems where a large virtually-shared address space is mapped on a network of physically distributed memories. High memory latency and network contention are two of the main factors that limit performance scaling of such architectures. Modern high-performance computing DSM systems have evolved towards the exploitation of massive hardware multi-threading and fine-grained memory hashing to tolerate irregular latencies, avoiding network hot-spots and improving scalability. Parallel simulation is a promising approach, which has been extensively used to model the performance of such large-scale machines. One of the most critical factors in coping with the simulation speed-accuracy trade-off is network modeling. The Cray XMT is a massively multi-threaded supercomputing architecture that belongs to the DSM class. In this paper, we discuss the development of a network contention model for a full-system XMT simulator. We start by measuring the effects of network contention on a 128-processorXMT machine, we then investigate the trade-off that exists between simulation accuracy and speed, comparing three network models which operate at different levels of accuracy. The comparison and model validation is performed by executing a string-matching algorithm on the full-system simulator and on the actual machine, using three datasets that generate noticeably different contention patterns. Results prove that simulator accuracy in execution time remains within 10% of the real machine. We also show that the slowdown due to contention modeling is limited to 20%, when simulating a small number of processors, and becomes negligible for simulations with higher processor counts.
