Data management is one of the cornerstones in the distributed production computing environment that the EGEE project aims to provide for a e-Science infrastructure.We have designed and implemented a set of services and client components, addressing the diverse requirements of all user communities. LHC experiments as main users will generate and distribute approximately 15 PB of data per year worldwide using this infrastructure. Another key user community, biomedical projects, have strict security requirements with less emphasis on the volume of data.We maintain three service groups for grid data management: The Disk Pool Manager (DPM) Storage Element (with more than 100 instances deployed world-wide), the LCG File Catalogue (LFC) and the File Transfer Service (FTS) which sustains an aggregated transfer rate of 1.5GB/sec. They are complemented by individual client components and also tools which help coordinating more complex uses cases with multiple services (GFAL-client, lcg_util, eds-cli).In this paper we show how these services, keeping clean and standard interfaces among each other, can work together to cover the data flow and how they can be used as individual components to cover diverse requirements. We will also describe are as that we consider for further improvements, both for performance and functionality.

The world's largest scientific machine - the large hadron collider (LHC), situated outside Geneva, Switzerland - will generate some 15PB of data at rates up to 1.5 GB/s (in the case of the heavy-ion experiment, ALICE) to tape per year of operation. The processing of this data will be performed using a world-wide grid, the (worldwide) LHC computing grid built on top of the enabled grid for e-science and open science grid infrastructures. The LHC computing grid, which has offered a service for over two years now, is based upon a tier model comprising some 150 sites in tens of countries. In this paper, we describe the data management middleware stack - one of the key services provided by data grids. We give an overview of the different services implemented, a disk-based storage system which can support encryption, tools to manage the storage system and access files, the LCG file catalogue, and the file transfer service. We also review the relationship between these services.

Storage management is one of the most important enabling technologies for large-scale scientific investigations. Having to deal with multiple heterogeneous storage and file systems is one of the major bottlenecks in managing, replicating, and accessing files in distributed environments. Storage resource managers (SRMs), named after their Web services control protocol, provide the technology needed to manage the rapidly growing distributed data volumes, as a result of faster and larger computational facilities. SRMs are grid storage services providing interfaces to storage resources, as well as advanced functionality such as dynamic space allocation and file management on shared storage systems. They call on transport services to bring files into their space transparently and provide effective sharing of files. SRMs are based on a common specification that emerged over time and evolved into an international collaboration. This approach of an open specification that can be used by various institutions to adapt to their own storage systems has proven to be a remarkable success - the challenge has been to provide a consistent homogeneous interface to the grid, while allowing sites to have diverse infrastructures. In particular, supporting optional features while preserving interoperability is one of the main challenges we describe in this paper. We also describe using SRM in a large international high energy physics collaboration, called WLCG, to prepare to handle the large volume of data expected when the Large Hadron Collider (LHC) goes online at CERN. This intense collaboration led to refinements and additional functionality in the SRM specification, and the development of multiple interoperating implementations of SRM for various complex multi- component storage systems.

The Large Hadron Collider (LHC) at CERN, the European Organisation for Nuclear Research, will produce unprecedented volumes of data when it starts operation in 2007. To provide for its computational needs, the LHC Computing Grid (LCG) is being deployed as a worldwide computational grid service, providing the middleware upon which the physics analysis for the LHC will be carried out. Data management middleware will be a key component of the LCG, enabling users to analyse their data without reference to the complex details of the computing environment. In this paper we review the performance tests of the LCG File Catalog (LFC) and make comparisons with other data management catalogs. We also survey the deployment status of the LFC within the LCG.