Decentralized Online Social Networks (DOSNs) offer an alternative to the usual centralized solutions, that promises to preserve more control of a user's data to the user herself. An important aspect in the design of DOSNs is the selection of the Peer-to-Peer overlay network. We propose the use of an augmented Pastry-based overlay network design, so as to produce a topology that incorporates the social network. Our augmentation aims to reduce the number of overlay hops required for the communication between two users in the social network. We experimentally show that our approach reduces the number of overlay hops and the total latency between two socially connected users versus the standard Pastry overlay. Additionally, we compare our approach with Sprout, a Chord alternative, and show that in many cases we achieve similarly low latency, while maintaining a bounded number of links.

Peer-to-peer (P2P) technology has long been considered a natural complement to standard CDN infrastructure for video distribution since it greatly reduces costs and improves quality of user experience. However, P2P solutions have traditionally required the installation of additional software or plugins to be deployed, which significantly hinders adoption. In this paper, we present Hive.js, a browser-based plugin-less distributed caching platform for video streaming. Hive.js is layered over WebRTC, a new set of HTML5 APIs for direct browser-to-browser communication, and it is designed to transport adaptive HTTP streaming protocols, specifically MPEG-DASH. Initial results obtained by evaluating Hive.js in a controlled test environment show that our approach significantly reduces the load on CDN infrastructure and does not sacrifice quality of user experience.

Key-based routing schemes, where a message is forwarded towards a server responsible for a partition in a large name space, does not provide strong delivery guarantees when the network is reconfigured with servers joining and leaving. This best-effort behavior is sufficient for eventually consistent data services such as key-value stores, content distribution networks or publish/subscribe systems. However, such schemes are not able to provide stronger consistency guarantees as required by, for example, metadata services and databases.We present RECODE, a framework for reconfigurable, consistent and decentralized data services. RECODE simplifies the implementation of strongly consistent data services, and continues to provide strong guarantees even during reconfiguration. More specifically, we introduce the routecast primitive which delivers messages for a key in the same total order, independent of the servers responsible for the key. We demonstrate the expressiveness and practical usability of RECODE by presenting three applications: a map of atomic registers, a set of distributed counters, and a lease management system. We evaluate the performance and elasticity of RECODE executing in a cluster.

In this work, we present a distributed caching solution which addresses the problem of efficient delivery of HTTP live streams in large private networks. With our system, we have conducted tests on a number of pilot deployments. The largest of them, with 3000 concurrent viewers, consistently showed that our system saves more than 90% of traffic towards the source of the stream while providing the same quality of user experience of a CDN. Another result is that our solution was able to reduce the load on the bottlenecks in the network by an average of 91.6%.

Scalable key-based routing in distributed systems, where a message is forwarded towards a machine responsible for a partition in a large key space, has been used in many services such as key-value stores, content distribution networks and media streaming. This success can mainly be attributed to the simplicity of the route abstraction, a developer does not need to care about the mechanisms for membership management, load balancing or data replication. A limitation, however, is that most key-based routing solutions are best-effort, which means that only eventually consistent data access is possible. This thesis presents a system (Recode) with a key-based routing primitive called routecast which provides strong delivery semantics. More specifically, routecast guarantees that a message for a key is delivered in the same total order at a set of replicas. With stronger guarantees, applications such as coordination and metadata services as used in large storage systems or consistent key-value stores can use key-based routing. Additionally, Recode aims to be both reconfigurable, to handle changes to the machines running the service and updates to the workload, and fully decentralized which means there is no single point of failure or bottleneck. We make three main contributions in this thesis: 1) a group communication abstraction using primary/backup with leases for primary fail-over, 2) the design and algorithms of the routecast-primitive and, 3) mechanisms for atomic reconfiguration of a decentralized key space. Each part of the system is broken up into modules and presented with a specification and a set of algorithms. To validate the simplicity claim, we describe how to implement three different applications on top of Recode. Finally, we evaluate Recode in a cluster environment and show that the performance is competitive.

Time series analysis, as an application for high dimensional data mining, is a common task in biochemistry, meteorology, climate research, bio-medicine or marketing. Similarity search in data with increasing dimensionality results in an exponential growth of the search space, referred to as Curse of Dimensionality. A common approach to postpone this effect is to apply approximation to reduce the dimensionality of the original data prior to indexing. However, approximation involves loss of information, which also leads to an exponential growth of the search space. Therefore, indexing an approximation with a high dimensionality, i. e. high quality, is desirable. We introduce Symbolic Fourier Approximation (SFA) and the SFA trie which allows for indexing of not only large datasets but also high dimensional approximations. This is done by exploiting the trade-off between the quality of the approximation and the degeneration of the index by using a variable number of dimensions to represent each approximation. Our experiments show that SFA combined with the SFA trie can scale up to a factor of 5--10 more indexed dimensions than previous approaches. Thus, it provides lower page accesses and CPU costs by a factor of 2--25 respectively 2--11 for exact similarity search using real world and synthetic data.

Large-scale distributed systems often require scalable and fault-tolerant mechanisms to coordinate exclusive access to shared resources such as files, replicas or the primary role. The best known algorithms to implement distributed mutual exclusion with leases, such as Multipaxos, are complex, difficult to implement, and rely on stable storage to persist lease information. In this paper we present {\bf F}LEASE, an algorithm for fault-tolerant lease coordination in distributed systems that is simpler than Multipaxos and does not rely on stable storage. The evaluation shows that {\bf F}LEASE can be used to implement scalable, decentralized lease coordination that outperforms a central lock service implementation by an order of magnitude.

Distributed hash tables (DHT) often rely on uniform hashing for balancing the load among their nodes. However, the most overloaded node may still have a load up to O(log N) times higher than the average load. DHTs with support for range queries cannot rely on hashing to fairly balance the system's load since hashing destroys the order of the stored items. Ensuring a fair load distribution is vital to avoid individual nodes becoming overloaded, potentially leading to node crashes or an incentive not to participate in the system. In both scenarios explicit load balancing schemes can help to spread the load more evenly. In this paper, we improve on existing algorithms for item-based active load balancing by relying on approximations of global properties. We show that the algorithms can be made more efficient by incorporating estimates of properties such as the average load and the standard deviation. Our algorithms reduce the network traffic induced by load balancing while achieving a better load balance than standard algorithms. We also show that these improvements can be applied to passive load balancing algorithms. Compared to DHTs without explicit load balancing, both variants are able to reduce the total maintenance traffic, i.e. item movements due to churn and load balancing, by up to 18%. Simultaneously the system is being balanced achieving a better load distribution.

Infrastructure federation is becoming an increasingly important issue for modern Distributed Computing Infrastructures (DCIs): Dynamic elasticity of quasi-static Grid environments, incorporation of special-purpose resources into commoditized Cloud infrastructures, cross-community collaboration for increasingly diverging areas of modern e-Science, and Cloud Bursting pose major challenges on the technical level for many resource and middleware providers. Especially with respect to increasing costs of operating data centers, the intelligent yet automated and secure sharing of resources is a key factor for success. With the D-Grid Scheduler Interoperability (DGSI) project within the German D-Grid Initiative, we provide a strategic technology for the automatically negotiated, SLA-secured, dynamically provisioned federation of resources and services for Grid-and Cloud-type infrastructures. This goal is achieved by complementing current DCI schedulers with the ability to federate infrastructure for the temporary leasing of resources and rechanneling of workloads. In this work, we describe the overall architecture and SLA-secured negotiation protocols within DGSI and depict an advanced mechanism for resource delegation through means of dynamically provisioned, virtualized middleware. Through this methodology, we provide the technological foundation for intelligent capacity planning and workload management in a cross-infrastructure fashion.

We present status and results of AstroGrid-D, a joint effort of astrophysicists and computer scientists to employ grid technology for scientific applications AstroGrid-D provides access to a network of distributed machines with a set of commands as well as software interfaces. It allows simple use of computer and storage facilities and to schedule or monitor compute tasks and data management. Iris based on the Globus Toolkit middleware (GT4).Chapter 1 describes the context which led to the demand for advanced software solutions in Astrophysics. and we state the goals of the projectWe then present characteristic astrophysical applications that have been implemented on AstroGrid-D in chapter 2 We describe simulations of different complexity, compute-intensive calculations running on multiple sites (Section 2 1), and advanced applications for specific scientific purposes (Section 2 2), such as a connection to robotic telescopes (Section 2.2.3) We can show from these examples how grid execution improves e g the scientific workflowChapter 3 explains the software tools and services that we adapted or newly developed Section 3 I is focused on the administrative aspects of the infrastructure. to manage users and monitor activity Section 3.2 characterises the central components of our architecture The AstroGrid-D information service to collect and store metadata. a file management system, the data management system, and a job manager for automatic submission of compute tasksWe summarise the successfully established infrastructure in chapter 4, concluding with our future plans to establish AstroGrid-D as a platform of modern e-Astronomy (C) 2010 Elsevier B V. All rights reserved.

A file system snapshot is a stable image of all files and directories in a well-defined state. Local file systems offer point-in-time consistency of snapshots, which guarantees that all files are frozen in a state in which they were at the same point in time. However, this cannot be achieved in a distributed file system without global clocks or synchronous snapshot operations. We present an algorithm for distributed file system snapshots that overcomes this problem by relaxing the point-intime consistency of local file system snapshots to a time span-based consistency. Built on loosely synchronized server clocks, it makes snapshots available within milliseconds, without any kind of locking or synchronization. Our evaluation demonstrates that enabling and accessing snapshots involves a read/write throughput penalty of no more than 1% under normal conditions.

There are only few data mining algorithms that work in a massively parallel and yet online (i.e. incremental) fashion. A combination of both features is essential for mining of large data streams and adds scalability to the concept of Online Aggregation introduced by J. M. Hellerstein et al. in 1997. We show how an online version of the Map-Reduce programming model can be used to implement such algorithms, and propose a solution for the "hardest" class of these algorithms - those requiring multiple Map-Reduce phases. An experimental evaluation confirms that the proposed methods can substantially accelerate interactive analysis of large data sets and facilitate scalable stream mining.

Distributed key/value-stores are a key component of many large-scale applications. Traditionally they have been designed using Distributed Hash Tables (DHTs). DHTs, however, setup a tight coupling between the naming of nodes and assignment of keys to nodes which limits application control over data placement. We propose using small amounts of shared state in a semi-centralized architecture for more flexible data placement by introducing explicit mapping between keys and nodes via an indirection layer (blockspace). Our design is based on a membership layer that provides O(1) routing thereby targeting interactive applications. We evaluate a centralized and decentralized approach showing that both have relatively low overhead and provide efficient load balancing.

A lease is a token which grants its owner exclusive access to a resource for a defined span of time. In order to be able to tolerate failures, leases need to be coordinated by distributed processes. We present FaTLease, an algorithm for fault-tolerant lease negotiation in distributed systems. It is built on the Paxos algorithm for distributed consensus, but avoids Paxos' main performance bottleneck of requiring persistent state. This property makes our algorithm particularly useful for applications that can not dispense any disk bandwidth. Our experiments show that FaTLease scales up to tens of thousands of concurrent leases and can negotiate thousands of leases per second in both LAN and WAN environments.

Distributed key/value stores are a basic building block for large-scale Internet services. Support for range queries introduces new challenges to load balancing since both the key and workload distribution can be non-uniform.We build on previous work based on the power of choice to present algorithms suitable for active and passive load balancing that adapt to both the key and workload distribution. The algorithms are evaluated in a simulated environment, focusing on the impact of load balancing on scalability under normal conditions and in an overloaded system.

We present a framework for transactional data access on data stored in a DHT. It allows to atomically read and write items and to run distributed transactions consisting of a sequence of read and write operations on the items. Items are symmetrically replicated in order to achieve durability of data stored in the SON. To provide availability of items despite the unavailability of some replicas, operations on items are quorum-based. They make progress as long as a majority of replicas can be accessed. Our framework processes transactions optimistically with an atomic commit protocol that is based on Paxos atomic commit. We present algorithms for the whole framework with an event based notation. Additionally we discuss the problem of lookup inconsistencies and its implications on the one-copy serializability property of the transaction processing in our framework.