Doing Quality Assurance work on Debian, a Linux distribution with more than 12000 packages, requires an impressive amount of computing power, which is usually not available for its developers. In this article, we report on the development of an infrastructure to run quality-assurance tasks on Debian using the Grid'5000 experimental platform. In particular, we focus on the problem of rebuilding all packages in Debian from source. We describe the details of this task, and the infrastructure we developed, with scalability and robustness in mind. The results we obtained are then presented, and we discuss possible improvements and lessons we learnt in the process, which might be useful in the context of other large-scale experiments.
Nowadays, scientists are interested in using computer grids to run their specific domain applications. Nevertheless the difficulty for non expert in computer science to run those applications on a high number of nodes discourages them. This paper deals with the deployment and the management of an electromagnetic simulator on a computer grid, by using the TUNe autonomic middleware. First, we describe the simulator and how to make it work with TUNe. Then, we measure TUNe's performance during specific phases: deployment, starting and undeployment. Moreover, some capabilities like automatic repair of applications in case of failure are shown.
As the number of services and the size of data involved in workflows increases, centralised orchestration techniques are reaching the limits of scalability. In the classic orchestration model, all data pass through a centralised engine, which results in unnecessary data transfer, wasted bandwidth and the engine to become a bottleneck to the execution of a workflow. Choreography techniques, although more complex to model offer a decentralised alternative and are the optimal architecture for data-centric workflows; data are passed directly to where they are required, at the next service in the workflow. While orchestration is the dominant architectural approach, there are relatively few choreography languages and even fewer concrete implementations. This papers contributions are twofold. Firstly we argue the case for choreography in data-intensive computing, and demonstrate through workflow patterns the advantages in terms of scalability when a choreography architecture is adopted. Secondly we introduce the Light Weight Coordination Calculus (LCC), a type of process calculus used to formally define choreographies, and the OpenKnowledge framework, a choreography-based architecture, providing the functionality for peers to coordinate in an open peer-to-peer system. Through LCC and the OpenKnowledge framework we practically demonstrate how choreography can be achieved in a lightweight manner with a comparatively simple process language.
This paper proposes an efficient and transparent transaction management on HVEM DataGrid. HVEM DataGrid has its data workflow which defines the precedence of the data to be saved. The precedence drives many states in a transaction for data management on HVEM DataGrid. We divide the transaction into several sub-transactions according to the states and classify the sub-transactions. Through the transaction hierarchy and parallelism, large image data can be uploaded to HVEM DataGrid efficiently and safely. The transaction management guarantees effective garbage collection as well. The garbage collection is important in HVEM DataGrid because it allows incomplete data insertion and update; it is not easy to decide whether the experimental results and their metadata are stored completely or not. Also, as shared storage system, HVEM DataGrid is composed of heterogeneous storages such as DBs and file storages. But, the transaction to save data to the datagrid requires to access those storages at the same time. And, the access should be transparent to users. The transaction hierarchy can solve the simultaneous access and satisfy these requirements. We will show the proposed scheme to be sound through analysis and to be effective by performance evaluation and overhead estimation. This paper shows that HVEM DataGrid can be a good model of e-Science datagrid through the efficient and transparent transaction scheme across many heterogeneous storages.
Scientific codes are all subject to variation in performance depending on the runtime platform and/or configuration, the output writing API employed, and the file system for output. Since changing the IO routines to match the optimal or desired configuration for a given system can be costly in terms of human time and machine resources, the Adaptable IO System provides an API nearly as simple as POSIX IO that also provides developers with the flexibility of selection the optimal IO routines for a given platform, without recompilation. As a side effect, we also gain the ability to transparently integrate more tightly with workflow systems like Kepler and Pegasus and visualization systems like Visit with no runtime impact. We achieve this through our library of highly tuned IO routines and other transport methods selected and configured in an XML file read only at startup. ADIOS-based IO has demonstrated high levels of performance and scalability. For example, we have achieved 20 GB/sec write performance using GTC on the Jaguar Cray XT4 system at Oak Ridge National Labs (about 50\% of peak performance). We can change GTC output among MPI-IO synchronous, MPI-IO collective, POSIX IO, no IO (for baseline testing), asynchronous IO using the Georgia Tech DataTap system, and Visit directly for in situ visualization with no changes to the source code. We designed this initial version of ADIOS based on the data requirements of 7 major scientific codes (GTC, Chimera, GTS, XGC1, XGC0, FLASH, and S3D) and have successfully adapted all of them to use ADIOS for all of their IO needs.
Reservoir uncertainty analysis is significant for petroleum engineers for predictions of reservoir performance. However, analysis of reservoir performance uncertainty is challenging because large amounts of data must be transferred efficiently, reliably and securely between sites, and thousands of simulations are executed across different resources. There are several steps in conducting reservoir performance prediction, including: (a) transferring input files to remote resources, (b) running thousands of simulations in different scheduler systems, (c) monitoring the jobs, (d) transferring output files from remote sites to the local system, and (e) post-processing to determine whether simulations have resolved uncertainties adequately. This whole process may have to be repeated as new data are obtained, or if uncertainty thresholds change. Therefore, it is essential to automate end-to-end processing for this complex, composite application with many tasks that are executed in a specific order. We implemented an end-to-end automated system for reservoir uncertainty analysis using Grid technologies such as Condor-G, DAGMan, and Stork. This paper describes the requirements, design and implementation of such a system.
Internet scale desktop grids can aggregate large numbers of globally distributed resources to support large computations, and can potentially be used to address large applications in science, engineering and business with significant computational requirements. However current practices are largely limited to embarrassingly parallel applications, where the individual tasks are independent and do not require inter-task communications, which seriously limits the class of application that can benefit from these environments. This talk will describe an autonomic computing engine that extends traditional desktop grids to support more general application workflows and formulations, which require synchronization and inter-task communications. The engine provides the abstractions and mechanisms required to support Internet scale parallel applications, including mechanisms for dynamic resource discovery, dynamic and anonymous task distribution, task coordination and execution, decoupled communication and data exchange, task/resource failures, etc. Deployment of the engine and its evaluations using the PlanetLab wide-area testbed will be presented.
High-performance computers such as the petascale systems being installed at DOE and NSF centers in the US are conventionally focused on "heroic" computations in which many processors are applied to a single task. Yet a growing number of science applications are equally concerned with "logistical" issues: that is, with the high-performance and reliable execution of many tasks that operate on large shared data and/or are linked by communication-intensive producer-consumer relations. Such applications may require the extreme computational capacity and specialized communication fabrics of petascale computers, but are not easily expressed using conventional parallel programming models such as MPI. To enable the use of high-performance computers for these applications, we need new methods for the efficient dispatch, coupling, and management of large numbers of communication-intensive tasks. I discuss how work on scripting languages, high-throughput computing, and parallel I/O can be combined to build new tools that enable the efficient and reliable execution of applications involving from hundreds to millions of uniprocessor and multiprocessor tasks, with aggregate communication requirements of tens of gigabytes per second. I illustrate my presentation by referring to our experiences adapting the Swift parallel programming system (www.ci.uchicago.edu/swift) for efficient execution in both large-scale grid and petascale cluster environments.
Matlab is one of the most commonly used languages for scientific computing with approximately one million users worldwide. Many of the programs written in matlab can benefit from the increased performance offered by multicore processors and parallel computing clusters. The Lincoln pMatlab library (http://www.ll.mit.edu/pMatlab) allows high performance parallel programs to be written quickly using the distributed arrays programming paradigm. This talk provides an introduction to distributed arrays programming and will describe the best programming practices for using distributed arrays to produce programs that perform well on multicore processors and parallel computing clusters. These practices include understanding the concepts of parallel concurrency vs. parallel data locality, using Amdahl's Law, and a well defined design-code-debug-test process for parallel codes.
Now in its sixth year, Challenges of Large Applications in Distributed Environments (CLADE) has sought to encourage innovation in addressing the complex issues that arise in large-scale applications of distributed computation. Focus areas have included scheduling and mapping, data management and workflows as well as applications and infrastructure. This talk will highlight developments presented at CLADE through the years.
Large scale scientific applications generally experience different execution phases at runtime and each phase has different computational and communication requirements. An optimal solution or numerical scheme for one execution phase might not be appropriate for the next phase of the application execution. In this paper we present Physics Aware Programming (PAP) paradigm that supports dynamic changes of the application solution if it optimizes the application performance at runtime. In the PAP approach, the application execution state is periodically monitored and analyzed to identify its current execution phase (state). For each change in the application execution phase, we will exploit the spatial and temporal attributes of the application physics to select the numerical algorithms/solvers that optimize its performance. We have applied our approach to a Ten-Tusscher's model of human ventricular epicardia myocyte paced at a varied cycle length (1000 to 50 ms). At runtime, we recognize the current phase of the heart simulation and based on the detected phase, we adopt the simulation Δt that maximizes the performance and maintains the required accuracy. Our experimental results show that we can achieve a speedup of three orders of magnitude while maintaining the required accuracy.
The scientific computing community, especially academia is clearly in need of technology to handle and organize the 1-100+ Terabyte datasets coming from computer simulations and scientific instrumentation. In this paper we briefly describe GrayWulf, an exemplar cluster for data intensive applications using SQL Server and HPC Clusters. One of the key software components of GrayWulf is Trident, a scientific workflow workbench that performs automatic scheduling of workflows across the cluster. We examine the challenges of scheduling workflows on GrayWulf, algorithms to improve performance, and present early results from applying Trident to schedule data loading workflows on GrayWulf for an actual e-Science project
An open distributed system provides a best-effort guarantee on the quality of service provided to applications. This has worked well for throughput-based applications of the kind typically executed in Condor or BOINCstyle environments. For other applications, the absence of timeliness of correctness guarantees limit the utility or appeal of this environment. Computational results that are too late or erroneous are not usable to the application. We present techniques designed to efficiently promote usable computing in open distributed systems.
With the exponential growth of complete genome sequences, the analysis of these sequences is becoming a powerful approach to build genome-scale metabolic models. These models can be used to study individual molecular components and their relationships, and eventually study cells as systems. However, constructing genome-scale metabolic models manually is time-consuming and labor-intensive. This property of manual model-building process causes the fact that much fewer genome-scale metabolic models are available comparing to hundreds of genome sequences available. To tackle this problem, we design SWARM, a scientific workflow that can be utilized to improve genome-scale metabolic models in high-throughput fashion. SWARM deals with a range of issues including the integration of data across distributed resources, data format conversions, data update, and data provenance. Putting altogether, SWARM streamlines the whole modeling process that includes extracting data from various resources, deriving training datasets to train a set of predictors and applying Bayesian techniques to assemble the predictors, inferring on the ensemble of predictors to insert missing data, and eventually improving draft metabolic networks automatically. By the enhancement of metabolic model construction, SWARM enables scientists to generate many genome-scale metabolic models within a short period of time and with less effort.
e-AIRS, an abbreviation of 'e-Science Aerospace Integrated Research System', is a virtual organization designed to support the aerospace engineering processes on the e-Science environment. Through the collaborative work between aerospace researchers and computer scientists, e-AIRS web portal with the full services on aerodynamic research process is devised. Currently, e-AIRS supports CFD simulations, remote experimental service, and collaborative and integrative study between computation and experiment. Of these supports, CFD (Computational Fluid Dynamics) service is composed is of mesh generation, CFD analysis and result visualization, and enables users to conduct the full simulation process on the web. This characteristic makes e-AIRS to be a good system for both education and research. In this paper, details on e-AIRS computing service and, the use of this system to fluid dynamic lectures on universities and integrative launch vehicle simulation are to be described.
Large parameter sweep studies are common in a broad range of scientific disciplines. However, many challenges exist in supporting this type of computation in a widely distributed computing environment. These challenges exist because contributing sites in a federated distributed computing environment usually expose only a very narrow resource-sharing interface. This paper looks at the challenges encountered by parameter sweep studies using two concrete application examples. The paper also shows how a system for building personal clusters on demand has been developed to solve many of these problems.
This paper presents a generalizable architecture for a grid-enabled biomedical imaging application, used for processing of pathological images for computer-aided prognosis. The presented architecture provides the scientists and developers a collaborative environment for management of remote image data and algorithm repositories and job execution over the grid. It is specifically designed for processing large-scale images in a distributed environment. In addition to the architecture, we also present its application to the computer-aided neuroblastoma prognosis as well as performance evaluation of the developed system using digitized pathological images.
For support of the world-wide ITER (International Thermonuclear Experimental Reactor) project [1], large scale numerical simulations will be a necessity. Plasma turbulence simulations play a key role for the design, construction and optimization of the necessary fusion devices. The simulations will be so compute and memory intensive that applications must be able to efficiently use tens of thousands of processors. Highly scalable applications are mandatory. In Europe, within the DEISA supercomputing grid project, the DEISA Extreme Computing Initiative supports application enabling work and usage of Europe's most powerful supercomputers for challenging projects. With all support of the DEISA Extreme Computing Initiative and the DEISA Joint Research Activity in Plasma Physics, leading plasma turbulence simulations have been enabled for hyperscaling and for efficient and portable usage in the heterogeneous DEISA grid. Here we report about the large scale applications GENE and ORB5 with high relevance for ITER. GENE, a so-called Vlasov-code, was parallelized to such a high degree that efficient usage on 32,768 processors could be demonstrated. The PIC code ORB5 was enabled for hyperscaling especially through application of the domain cloning concept. Efficient usage of ORB5 code could be demonstrated on 8192 processors.
Virtual observatories will give astronomers easy access to an unprecedented amount of data. Extracting scientific knowledge from these data will increasingly demand both efficient algorithms as well as the power of parallel computers. Nearly all efficient analyses of large astronomical datasets use trees as their fundamental data structure. Writing efficient tree-based techniques, a task that is time-consuming even on single-processor computers, is exceedingly cumbersome on massively parallel platforms (MPPs). Most applications that run on MPPs are simulation codes, since the expense of developing them is offset by the fact that they will be used for many years by many researchers. In contrast, data analysis codes change far more rapidly, are often unique to individual researchers, and therefore accommodate little reuse. Consequently, the economics of the current high-performance computing development paradigm for MPPs does not favor data analysis applications. We have therefore built a library, called Ntropy, that provides a flexible, extensible, and easy-to-use way of developing tree-based data analysis algorithms for both serial and parallel platforms. Our experience has shown that not only does our library save development time, it can also deliver excellent serial performance and parallel scalability. Furthermore, Ntropy makes it easy for an astronomer with little or no parallel programming experience to quickly scale their application to a distributed multiprocessor environment. By minimizing development time for efficient and scalable data analysis, we enable wide-scale knowledge discovery on massive datasets.
After having deployed a first data challenge on malaria and a second one on avian flu, respectively in summer 2005 and spring 2006, we are demonstrating here again how efficiently the computational grids can be used to produce massive docking data at a high-throughput. During more than 2 months and a half, we have achieved at least 140 million dockings, representing an average throughput of almost 80,000 dockings per hour. This was made possible by the availability of thousands of CPUs through different infrastructures worldwide. Through the acquired experience, the WISDOM production environment is evolving to enable an easy and fault-tolerant deployment of biological tools; in this case it is the FlexX commercial docking software which is used to dock the whole ZINC database against 4 different targets.
