Scheduling of large-scale, distributed topology-aware applications requires that not only the properties of the requested machines be considered, but also the properties of the machines' interconnections. This requirement severely complicates the scheduling process, as even a matching between a single multi-processor task and available machines in a single time slot becomes an NP-complete problem with no polynomial approximation. In this paper we propose a complete scheduling framework for multi-cluster, heterogeneous environments that provides, in practice, an efficient solution for the scheduling of topology-aware applications. The proposed framework is very flexible as it is composed of pluggable components and can be easily configured to support a variety of scheduling policies. We also describe three novel scheduling and coallocation algorithms that were developed and plugged into the framework. The proposed scheduling framework was integrated into the QosCosGrid system, where it is used as the main decision-making module.

The concept of topology-aware grid applications is derived from parallelized computational models of complex systems that are executed on heterogeneous resources, either because they require specialized hardware for certain calculations, or because their parallelization is flexible enough to exploit such resources. Here we describe two such applications, a multi-body simulation of stellar evolution, and an evolutionary algorithm that is used for reverse-engineering gene regulatory networks. We then describe the topology-aware middleware we have developed to facilitate the "modeling-implementing-executing" cycle of complex systems applications. The developed middleware allows topology-aware simulations to run on geographically distributed clusters with or without firewalls between them. Additionally, we describe advanced coallocation and scheduling techniques that take into account the applications topologies. Results are given based on running the topology-aware applications on the Grid'5000 infrastructure.

In this work we present a simple, yet powerful, methodology for application-agnostic diagnostic and remediation of performance hot spots in elastic multi-tiered client/server applications, deployed as collections of black box Virtual Machines (VM). Our novel out-of-band black-box performance management system, Network Analysis for Remediating Performance Bottlenecks (NAP), listens to the TCP/IP traffic on the virtual network interfaces of the VMs comprising an application and analyzes statistical properties of this traffic. From this analysis, which is application independent and transparent to the VMs, NAP identifies performance bottlenecks that might effect application performance and derives remediation decisions that are most likely to alleviate the application performance degradation. We prototyped our solution for the Xen hypervisor and evaluated it using the popular Trade6 benchmark that simulates a typical e-commerce application. Our results show that NAP successfully identifies performance bottlenecks in a complex multi-tier application setting, while incurring negligible performance overhead.

As modern data mining applications increase in complexity, so too do their demands for resources. Grid computing is one of several emerging networked computing paradigms promising to meet the requirements of heterogeneous, large-scale, and distributed data mining applications. Despite this promise, there are still too many issues to be resolved before grid technology is commonly applied to large-scale data mining tasks. To address some of these issues, the authors developed the DataMiningGrid system. It integrates a diverse set of programs and application scenarios within a single framework, and features scalability, flexible extensibility, sophisticated support for relevant standards and different users.

The DataMiningGrid system has been designed to meet the requirements of modern and distributed data mining scenarios. Based on the Globus Toolkit and other open technology and standards, the DataMiningGrid system provides tools and services facilitating the grid-enabling of data mining applications without any intervention on the application side. Critical features of the system include flexibility, extensibility, scalability, efficiency, conceptual simplicity and ease of use. The system has been developed and evaluated on the basis of a diverse set of use cases from different sectors in science and technology. The DataMiningGrid software is freely available under Apache License 2.0.

Grids are becoming mission-critical components in research and industry, offering sophisticated solutions in leveraging large-scale computing and storage resources. Grid resources are usually shared among multiple organizations in an opportunistic manner. However, an opportunistic or best effort quality-of-service scheme may be inadequate in situations where a large number of resources need to be allocated and applications which rely on static, stable execution environments. The goal of this work is to implement what we refer to as quasi-opportunistic supercomputing. A quasi-opportunistic supercomputer facilitates demanding parallel computing applications on the basis of massive, non-dedicated resources in grid computing environments. Within the EU-supported project QosCosGrid we are developing a quasi-opportunistic supercomputer. In this work we present the results obtained from studying and identifying the requirements a grid needs to meet in order to facilitate quasi-opportunistic supercomputing. Based on these requirements we have designed architecture for a quasi-opportunistic supercomputer. The paper presents and discusses this architecture.

Modern distributed applications require coallocation of massive amounts of resources. Grid level allocation systems must efficiently decide where these applications can be executed. To this end, the resource requests are described as labeled graphs, which must be matched with equivalent labeled graphs of available resources. The coallocation problem described in the paper has real-world requirements and inputs that differ from those of a classical graph matching problem. We propose a new algorithm to solve the coallocation problem. The algorithm is especially tailored for medium to large grid systems, and is currently being integrated into the QosCosGrid system's allocation module.

The ultimate vision of grid computing are virtual supercomputers of unprecedented power, through utilization of geographically dispersed distributively owned resources. Despite the overwhelming success of grids there still exist many demanding applications considered the exclusive prerogative of real supercomputers (i.e. tightly coupled parallel applications like complex systems simulations). These rely on a static execution environment with predictable performance, provided through efficient co-allocation of a large number of reliable interconnected resources. In this paper, we describe a novel quasi-opportunistic supercomputersystem that enables execution of demanding parallel applications in grids through identification and implementation of the set of key technologies required to realize the vision of grids as (virtual) supercomputers. These technologies include an incentive-based framework basic on ideas from economics; a co-allocation subsystem that is enhanced by communication topology-aware allocation mechanisms; a fault tolerant message passing library that hides the failures of the underlying resources; and data pre-staging orchestration.

Emerging data mining applications in science, engineering and other sectors increasingly exploit large and distributed data sources as well as computationally intensive algorithms. Adapting such applications to grid computing environments has implications for grid resource brokering. A grid resource broker supporting such applications needs to provide eective and ecient job scheduling, execution and monitoring. Furthermore, to be useable by domain-oriented end users and to be able to evolve gracefully with emerging grid technology, it should hide the underlying complexity of the grid from such users and be compliant with important grid standards and technology. The DataMiningGrid Resource Broker was designed to meet these requirements. This paper presents the DataMiningGrid Resource Broker and the results from evaluating it in a European-wide test bed.

The shift towards intrinsically distributed complex problem solving environments is prompting a need for new systems, which utilize the virtually unlimited data and computational resources of the Grid and at the same time hide all the related complexity from the user. Currently, there is no coherent framework, which offers data miners, who are usually not Grid experts, the ability to easily construct data mining tasks and execute them on the Grid. Therefore, there is a need to assemble a complete system that includes: a) a user-friendly environment for defining complex data mining tasks and b) a Grid middleware that supports execution of such tasks, while utilizing mechanisms for managing data and computational resources as well as having sophisticated job-monitoring capabilities. This paper will focus on the high-level design of such a system, which currently is being developed in the DataMiningGrid project with emphasis on the design and implementation of the resource broker service. We show how different resources from various domains can be exploited, in order to give the data mining researchers the ability to access and utilize resources needed for modern, distributed and computationally intensive data mining algorithms.

Grids are becoming mission-critical components in research and industry, offering quite sophisticated solutions for the exploitation of virtually unlimited computing and storage resources. The grid resources are usually shared among multiple organizations and typically managed in a "best effort" manner. However, many real-world supercomputing applications, such as computational fluid dynamics (CFD), weather forecasting, and complex system simulations, rely on co-allocation of large numbers of reliable resources as well as on a static and stable execution environment, making such "best effort" Quality of Service inadequate. In our work we aim at implementing a new concept of quasi- opportunistic supercomputing that will enable execution of demanding parallel applications on massive non-dedicated resources in grid environments. In this paper we identify a set of requirements that must be satisfied by a grid resource management system in order for grids to become an affordable alternative to supercomputing. The European-funded project QosCosGrid is currently developing such a system. Usability and efficiency of our implementation will be tested using a set of benchmark applications from the Complex Systems domain. 1. The challenges of supercomputing on top of a grid