We developed a set of HTTP REST APIs on top of iRODS to support users of different communities to automate both ingestion and retrieval data workflows. We built a common REST APIs layer by implementing basic functionalities, including the interaction with iRODS, within an extensible framework (RAPyDO: Rest Apis with Python on Docker) that we developed and adopted to build communities-specific REST APIs. More in details, we are collaborating with the EUropean DATa infrastructure EUDAT Collaborative Data Infrastructure (CDI); European projects like EOSC-hub and SeaDataCloud; national initiatives in collaboration with Telethon Foundation (a non-profit organization for genetic diseases research) and SIGU (Italian Society for Human Genomics). All endpoints are written by using the Python language on the Flask framework. APIs are served through an uWSGI web server deployed within a Docker container. We created a wrapper of the python irods client (PRC) to let both the core framework and communities specific APIs for easily interact with iRODS by supporting all main authentication protocols like native passwords, Pluggable Authentication Modules (PAM), Grid Security Infrastructure (GSI). To be able to support all required authentication methods we contributed to the PRC development by implementing authentication modules for both GSI and PAM. Most of iRODS-based functions that we developed can be mapped against corresponding icommands like ils, iget, iput, imv, icp, imeta, irule, iticket but also more complex functionalities have been realized, for instance streamed read/write operations from/to network sockets. To be able to execute data intensive and complex workflows, we also introduced an asynchronous layer implemented on Celery, a task management queue based on distributed message passing.

The amount of genomic data produced by DNA-sequencing is growing at an unprecedented rate due to the ever greater throughput provided by the new generation of genome sequencing (NGS) platforms. To understand and interpret biological data a huge number of metadata and annotations are required. Genomic metadata include very heterogeneous biological and clinical attributes gathered at different levels of details such as diseases, genes, proteins, interactions, pathways but also phenotypic characterization of patients and clinical evidence or diagnosis. All these information are required to draw a comprehensive picture of many underlying phenomenons, thus contributing to scientifically understand the observed data. The complexity of genomic data arises due to the number of involved entities (from millions to billions) and the complex relationships between them; biological information is typically highly connected, not uniform, semi-structured and unpredictable. Relationships and connections may be stored in a relational database and data can be extracted adopting traversal-type queries implying joins. Nevertheless joins with large tables easily become too cumbersome and computationally expensive to design, execute and maintain. This critical aspect makes relational databases non-suitable for this kind of data operations. Studies already suggested that graph databases are among the best choices to explore linked data, due to the design of the core engine which optimizes performance in exploring connections [1]. They also provide a flexible solution for the integration and exploration of multiple levels of biological information [2] and some specific sets of biological data are already structured as graphs [3].

EUDAT is a Consortium of European Data Centers and Scientific Communities whose focus is the development and realisation of the Collaborative Data Infrastructure (CDI), a common model for managing data spanning all European research data centres and data repositories and providing an interoperable layer of common data services. The EUDAT Service Suite is a set of a) implementations of the CDI model and b) standards, developed and offered by members of the EUDAT Consortium. These EUDAT Services include a baseline of CDI-compliant interface and API services – a “CDI Gateway” – plus a number of web-based GUIs and command-line client tools. On the other hand,the EPOS initiative aims at creating a pan-European infrastructure for the solid Earth science to support a safe and sustainable society. In accordance with this scientific vision, the mission of EPOS is to integrate the diverse and advanced European Research Infrastructures for solid Earth Science relying on new e-science opportunities to monitor and unravel the dynamic and complex Earth System. EPOS will enable innovative multidisciplinary research for a better understanding of the Earth’s physical and chemical processes that control earthquakes, volcanic eruptions, ground instability and tsunami as well as the processes driving tectonics and Earth’s surface dynamics. Through the integration of data, models and facilities EPOS will allow the Earth Science community to make a step change in developing new concepts and tools for key answers to scientific and socio-economic questions concerning geo-hazards and geo-resources as well as Earth sciences applications to the environment and to human welfare. To achieve this integration challenge and the interoperability among all involved communities, EPOS has designed an architecture capable to organize and manage distributed discipline-oriented centers (called Thematic Core Services TCS). Such design envisage the creation of an integrating e-Infrastructure called Integrated Core Service (ICS), whose aim is to collect and integrate Data, Data Products, Software and Services, and provide homogeneous access to them to the end user, hiding all the complexity of the underlying network of TCS and National data centers. Therefore, EPOS can take advantage of EUDAT CDI at different levels: at the TCS level, providing technologies, knowledge and B2* services to discipline-oriented communities, and at the ICS level, by facilitating the integration and interoperability of different communities with different level of maturity in terms of technology expertise. EUDAT services are particularly suitable to facilitate this process as they can be deployed across the community centers to complement or augment existing services of more mature communities as well as be used by less mature communities as a gateway towards the EPOS integration.

The wide variety of scientific user communities work with data since many years and thus have already a wide variety of data infrastructures in production today. The aim of this paper is thus not to create one new general data architecture that would fail to be adopted by each and any individual user community. Instead this contribution aims to design a reference model with abstract entities that is able to federate existing concrete infrastructures under one umbrella. A reference model is an abstract framework for understanding significant entities and relationships between them and thus helps to understand existing data infrastructures when comparing them in terms of functionality, services, and boundary conditions. A derived architecture from such a reference model then can be used to create a federated architecture that builds on the existing infrastructures that could align to a major common vision. This common vision is named as 'ScienceTube' as part of this contribution that determines the high-level goal that the reference model aims to support. This paper will describe how a well-focused use case around data replication and its related activities in the EUDAT project aim to provide a first step towards this vision. Concrete stakeholder requirements arising from scientific end users such as those of the European Strategy Forum on Research Infrastructure (ESFRI) projects underpin this contribution with clear evidence that the EUDAT activities are bottom-up thus providing real solutions towards the so often only described 'high-level big data challenges'. The followed federated approach taking advantage of community and data centers (with large computational resources) further describes how data replication services enable data-intensive computing of terabytes or even petabytes of data emerging from ESFRI projects.

The steadily increasing amounts of scientific data and the analysis of 'big data' is a fundamental characteristic in the context of computational simulations that are based on numerical methods or known physical laws. This represents both an opportunity and challenge on different levels for traditional distributed computing approaches, architectures, and infrastructures. On the lowest level data-intensive computing is a challenge since CPU speed has surpassed IO capabilities of HPC resources and on the higher levels complex cross-disciplinary data sharing is envisioned via data infrastructures in order to engage in the fragmented answers to societal challenges. This paper highlights how these levels share the demand for 'high productivity processing' of 'big data' including the sharing and analysis of 'large-scale science data-sets'. The paper will describe approaches such as the high-level European data infrastructure EUDAT as well as low-level requirements arising from HPC simulations used in distributed computing. The paper aims to address the fact that big data analysis methods such as computational steering and visualization, map-reduce, R, and others are around, but a lot of research and evaluations still need to be done to achieve scientific insights with them in the context of traditional distributed computing infrastructures.

Current praxis of software licensing has been identified as major obstacle for Grid computing a couple of years ago already. Recent surveys of Clouds indicate that the same holds true for Cloud computing. As a consequence, using commercial applications that require access to a license server for authorisation at run-time has been quite limited until recently in distributed computing environments. Due to the mandatory centralised control of license usage during application run-time traditional software licensing practices are not suitable. In this paper we present a novel approach for managing software licenses as web service resources in distributed service oriented environments. Licenses become mobile objects, which may move to the environment where required to authorise the execution of a license protected application. The SmartLM solution, which has been recently implemented as a prototype decouples authorisation for license usage from authorisation for application execution.

A recent survey of the 451group on Cloud usage highlights software licensing as one of the top five obstacles for Cloud computing, quite similar to what has been observed in the Grid already a couple of years. The reasons are the same: the current praxis of software licensing, both in terms of business models and licensing technology. As a consequence, using commercial applications that require access to a license server for authorisation at run-time has been quite limited until recently in distributed computing environments, especially when the environment stretches across administrative domains like it is the case for public Clouds. In this paper we present a novel approach for managing software licenses as web service resources in distributed service oriented environments. Licenses become mobile objects, which may move to the environment where required to authorise the execution of a license protected application.

One of the major obstacles to using commercial applications in Distributed Computing Infrastructures like Grids or Clouds is the current technology that relies on controlling the use of these applications with software licenses. "Software licensing practices are limiting the acceleration of grid adoption" was one of the results of a survey of the 451group in 2005. Just recently the 451group published a similar report on obstacles to the broad adoption of Cloud Computing - and again licensing practices were listed among the top five obstacles. elasticLM overcomes the limitations of existing licensing technologies allowing seamless running of license protected applications in computing environments ranging from local infrastructures to external Grids and Clouds.

Until recently the use of applications requiring a software license for execution was quite limited in distributed environments. Due to the mandatory centralised control of license usage at application run-time, e.g. heartbeat control by the license server running at the home site of a user, traditional software licensing practices are not suitable especially when the distributed environment stretches across administrative domains. In this paper we present a novel approach for managing software licenses as web service resources in distributed service oriented environments. Licenses become mobile objects, which may move to the environment where required to authorise the execution of a license protected application. A first implementation has been realised for dynamic Grid environments in the European SmartLM project co-funded by the European Commission. The SmartLM solution decouples authorisation for license usage from authorisation for application execution. All authorisations are expressed as and guaranteed by Service Level Agreements. We will present the core technology, discuss various security aspects and how they are addressed in SmartLM, and present a number of usage scenarios leveraged by the SmartLM technology. Finally, we will give an outlook on specific issues and current work extending the solution to Clouds and service based systems in general.

This paper is a description of the European A-WARE project. For each user, a deployed instance of the A-WARE technology presents a facade over the underlying fabric of grid resources a bridge between users and resources. The primary target in grid fabric is the Web services based UNICORE 6 software. For the user, A-WARE provides a single point of entry to their distributed resources, and importantly it shields them from the complexity of direct engagement. The primary user interface is Web browser based and built with two well known portal solutions, EnginFrame and GridSphere. The mediation layer is based on an enterprise service bus, which offers excellent integration prospects. This is used for tasks scoped to a single resource, e.g. the management and monitoring of individual resources, as well as tasks covering multiple resources, e.g. workflow enactment, and search functionality. In this paper, we highlight the benefits of such an approach and describe how the adoption of key business integration standards enable the transfer of grid technology to a broader range of fields, from HPC to the business domain.

The wide adoption of Service Oriented Architecture by the Grid community has made available several software tools that allow exposing hardware resources and scientific data to remote peers by means of well standardized protocols. Hence the necessity for scientists to easily design a simulation that leverages distributed applications. In this paper, we present the implementation details of A-WARE, a workflow framework that adopts recognised standards, especially by the enterprise community, like BPEL. In this way our product can boast a higher level of interoperability with other similar systems. The workflow graphical notation is also based on a standard: BPMN. BPMN provides a unique, standardized and comprehensive modeling format understandable by both business people (involved in the area of business process management) and IT people, resulting also portable across different departments or companies.

In order to foster uptake by scientific and business users we need an easy way to access Grid resources. This is the motivation for the A-WARE project. We build upon a fabric layer of Grid and other resources, by providing a higher-layer service for managing the interaction with these resources - A One-Stop, Fire-and-(almost)Forget, Dropping-off and Rendezvous Point. Work assignments can be formulated using domain specific dialects, allowing users to express themselves in their domain of expertise. Both Web service and REST bindings are provided, as well as allowing the component to be embedded into other presentation technologies (such as portals). In addition common desktop notification mechanisms such as Email, RSS/Atom feeds and instant messaging keep users informed and in control. We propose using the Java Business Integration specification as the framework for building such a higher-level component, delivering unprecedented opportunities for the integration of Grid technologies with the enterprise computing infrastructures commonly found in businesses.

The UNICORE grid system provides a seamless, secure and intuitive access to distributed grid resources. In recent years, UNICORE 5 is used as a well-tested grid middleware system in production grids (e.g. DEISA, D-Grid) and at many supercomputer centers world-wide. Beyond this production usage, UNICORE serves as a solid basis in many European and International research projects and business scenarios from T-Systems, Philips Research, Intel, Fujitsu and others. To foster ongoing developments in multiple projects, UNICORE is open source under BSD license at SourceForge. More recently, the new Web services-based UNICORE 6 has become available that is based on open standards such as the Web services addressing (WS-A) and the Web services resource framework (WS-RF) and thus conforms to the open grid services architecture (OGSA) of the open grid forum (OGF). In this paper we present the evolution from production UNICORE 5 to the open standards-based UNICORE 6 and its various Web services-based interfaces. It describes the interface integration of emerging open standards such as OGSA-BES and OGSA-RUS and thus provides an overview of UNICORE 6.

Large-scale scientific research often relies on the collaborative use of Grid and e-Science infrastructures that provide computational or storage related resources. One of the ideas of these modern infrastructures is to facilitate the routine interaction of scientists and their workflows with advanced problem solving tools and computational resources. While many production Grid projects and e-Science infrastructures have begun to offer services for the usage of resources to end-users during the past several years, the corresponding emerging standards defined by GGF and OASIS still appear to be in flux. In this paper, we present the GridBean technology that bridges the gap between the constantly changing basic Grid or e-Science infrastructures and the need of stable application development environments for the Grid users.