Existing architectures designed to host large-scale virtual environments (VEs) use a variety of approaches, but they often limit the interaction range with other users or with the VE. How densely users can populate a given region is also limited by the hosting machine's CPU or bandwidth resources. We are motivated to remove such restrictions and present SPEX, an infrastructure that supports scalable spatial publish/subscribe for VE applications. SPEX is scalable and fault-tolerant, with adaptive load balancing and low latency as its key features. It is designed for the state and overlay management in VEs with many concurrent users. We evaluate a practical SPEX implementation within Amazon's EC2 Cloud and present a feasible approach to supporting 750 users across a continent with low latency, opening the possibility for hosting fast-paced games (e.g., first-person shooters) or applications on a large-scale.

Key/value data storage systems serve as the fundamental component of scalable cloud-based services. However, the scalability of existing key/value datastores comes at the cost of a narrow data access API with relaxed data consistency. We present Innesto, a distributed key/value datastore that provides search as part of its API. Search allows data items to be retrieved based on constraints on multiple different attributes. Innesto’s strong consistency data model and its transactional interface bring much of the power of traditional relational databases to cloud-scale performance. Isolation between transactions can be performed using either traditional locks or using lock-free synchronisation based on clock vectors. Our evaluation of Innesto shows that it offers these extra features with competitive performance compared to an industrial key/value datastore such as Cassandra which offers an inferior feature set.

Key/value data storage systems serve as the fundamental component of scalable cloud-based services. However, the scalability of existing key/value data stores comes at the cost of a narrow data access API. We present Innesto, a distributed key/value data store that provides search as part of its API. Search allows data items to be retrieved based on constraints on multiple different attributes. Innesto's strong consistency data model and its transactional interface bring much of the power of traditional relational databases to cloud-scale performance. Our evaluation of Innesto shows it offers these extra features with competitive performance compared to an industrial key/value data store such as Cassandra which offers an inferior feature set.

Browsers have become mature execution platforms enabling web applications to rival their desktop counterparts. An important class of such applications is interactive multimedia: games, animations, and interactive visualizations. Unlike many early web applications, these applications are latency sensitive and processing (CPU and graphics) intensive. When demands exceed available resources, application quality (e.g., frame rate) diminishes because it is hard to balance timeliness and utilization. The quality of ambitious web applications is also limited by single-threaded execution prevalent in the Web. Applications need to scale their quality, and thereby scale processing load, based on the resources that are available. We refer to this as scalable quality. DOHA is an execution layer written entirely in JavaScript to enable scalable quality in web applications. DOHA favors important computations with more influence over quality based on hints from application-specific adaptation policies. To utilize widely available multi-core resources, DOHA augments HTML5 web workers with mechanisms to facilitate state management and load-balancing. We evaluate DOHA with an award-winning web-based game. When resources are limited, the modified game has better timing and overall quality. More importantly, quality scales linearly with a small number of cores and the game is playable in challenging scenarios that are beyond the scope of the original game.

Operating systems represent large pieces of complex software that are carefully tested and broadly deployed. Despite this, developers frequently have little more than their source code to understand how they behave. This static representation of a system results in limited insight into execution dynamics, such as what code is important, how data flows through a system, or how threads interact with one another. We describe Tralfamadore, a system that preserves complete traces of machine execution as an artifact that can be queried and analyzed with a library of simple, reusable operators, making it easy to develop and run new dynamic analyses. We demonstrate the benefits of this approach with several example applications, including a novel unified source and execution browser.

Shared storage underlies most enterprise VM deployments because it is an established technology that administrators are familiar with and because it good job of protecting data. However, shared storage is also very expensive to scale. This paper describes Capo, a transparent and persistent block request proxy for virtual machine disk images. Capo reduces the load on shared storage by using local disks as persistent caches, using multicast-based preloading to broadcast read results across a cluster, and by imposing differential durability - dividing a VM's file system into regions of varying writeback frequency. We motivate the system's design through the analysis of a week-long trace of 55 production virtual desktops and then describe and evaluate our implementation. Capo is particularly well suited for virtual desktop deployments, in which large numbers of VMs boot from a small number of gold master images and are refreshed on a periodic basis.

As the Web has grown in popularity, web applications have started to rival their desktop counterparts. An important class of such applications is interactive multimedia: games, animations, and interactive visualizations. Unlike many early web applications, these applications are latency sensitive and processing (CPU and graphics) intensive. As web applications execute on a wide range of hardware from phones to powerful desktops and dynamically share these platforms with other applications, resolving the conflict between timeliness, application quality, and CPU utilization is a challenge. We propose scalable quality, which consists of application-defined policies and an adaptive execution layer that includes HTML5 worker threads. This paper describes our adaptive approach to maximize quality according to available resources. We also explore the challenges of using web workers in a real game and share our initial design decisions for concurrent web-based game development.

Even as virtualization has promised to ease cluster scale and management, it presents system administrators and storage system designers with opaque blobs of data that represent entire virtual volumes . In these environments, application and file-level semantics are abandoned long before data reaches the disk . Our research borrows from past work and is creating virtual storage interfaces that preserve file-level information in order to improve the management and efficiency of storage .

When changing a storage system, the stakes are high. Any modification can undermine stability, causing temporary downtime, a permanent loss of data, and still worse - a loss of user confidence. This results in a cautious conservatism among storage developers. On one hand, the risks do justify taking great care with storage system changes. On the other hand, this slow and cautious deployment attitude is a poor match for cloud services tied closely to web-based frontends that follow an "always beta" mantra. Unlike traditional enterprise servers, cloud-based systems are still exploring what facilities should be provided by the storage layer, requiring that storage services be able to evolve as quickly as the applications that consume them. In this paper, we argue that by building support for evolution into the basic structure of a storage system, new features (and fixes) can be deployed in a fast and cautious manner. We summarize our experiences in developing such a system and detail its requirements and design. We also share some initial experience in deploying it on a rapidly evolving, but production, cloud hosting service that we have been building at UBC.

Program source is an intermediate representation of software; it lies between a developer's intention and the hardware's execution. Despite advances in languages and development tools, source itself and the applications we use to view it remain an essentially static representation of software, from which developers can spend considerable energy postulating actual behavior.Emerging techniques in execution logging promise to provide large shared repositories containing high-fidelity recordings of deployed, production software. Tralfamadore(1) is a system that combines source and execution trace analysis to capitalize on these recordings, and to expose information from the "experience" of real execution within the software development environment, allowing developers to inform their understanding of source based on how it behaves during real execution.

Parallax is a distributed storage system that uses virtualization to provide storage facilities specifically for virtual environments. The system employs a novel architecture in which storage features that have traditionally been implemented directly on high-end storage arrays and switches are relocated into a federation of storage VMs, sharing the same physical hosts as the VMs that they serve. This architecture retains the single administrative domain and OS agnosticism achieved by array- and switch-based approaches, while lowering the bar on hardware requirements and facilitating the development of new features. Parallax offers a comprehensive set of storage features including frequent, low-overhead snapshot of virtual disks, the 'gold-mastering' of template images, and the ability to use local disks as a persistent cache to dampen burst demand on networked storage.

One of the challenges of large-scale network simulations is the lack of scalable and realistic Internet topology generators. Previous topology generators are either not scalable to millions of nodes, or not able to capture characteristics of the Internet topology. In this work, we propose a topology generator which can generate accurate large-scale models of the Internet. We extract the AS (autonomous system) level and router level topology of the Internet with various data sources such as BGP routing tables and traceroute records. With the real Internet topology, we infer the AS topology and the commercial relationship among ASes. We also group the routers into clusters according to their positions in the Internet. A compact routing core is built with the AS topology and router cluster topology. Each generated topology consists of the routing core and a set of end-hosts connected to router clusters. The generated topology is realistic since its routing core is extracted from Internet. We make the assumption of uniform routing policy within an AS. Therefore, the routing path calculation of any source/destination pair consists of finding the AS path for the source/destination ASes and finding the router level path within each AS in the AS path. Since the routing pate depends only on the routing core, its size is independent of the number of end-hosts in the generated topology.

From this model, we draw four phyla of application: point- to-point, multiplexed, fragmented, and peer to peer. This last phylum defines our target domain and we apply lessons learned from the other three groups to it. Through our tax- onomy, we describe a set of architectural systems problems that must be addressed. The second aspect of our examination has been to step back and examine the implications involved with the adop- tion of large-scale distributed operating environments. In this section, we are less concerned with classical systems issues (performance, robustness, and scale) and more con- cerned with pragmatic factors involved in building a good system. We present a broad set of pertinent problems that will need to be addressed for these systems to be successful outside of the research laboratory.

Parallax is a distributed storage system that uses virtualization to provide storage facilities specifically for virtual environments. The system employs a novel architecture in which storage features that have traditionally been implemented directly on high-end storage arrays and switches are relocated into a federation of storage VMs, sharing the same physical hosts as the VMs that they serve. This architecture retains the single administrative domain and OS agnosticism achieved by array- and switch-based approaches, while lowering the bar on hardware requirements and facilitating the development of new features. Parallax offers a comprehensive set of storage features including frequent, low-overhead snapshot of virtual disks, the 'gold-mastering' of template images, and the ability to use local disks as a persistent cache to dampen burst demand on networked storage.

Hardware virtualization gives administrators the flexibility to rapidly create, destroy and relocate virtual machines across physical hosts. Unfortunately, the storage systems upon which these systems depend are not nearly as agile. To facilitate the rapid, safe development of block devices that can meet the needs of virtual machines, we present the Block Mason virtual block device framework. Although the block device interface is simple and intuitive, block devices themselves must generally be implemented in the operating system kernel, an environment which is neither simple nor portable. Block Mason allows users to build small, reusable block processing elements in user space, and to connect them together into powerful composite modules using a simple declarative graph language. Although the environment emphasizes simplicity for developers and end users, it includes built-in support for powerful operations like live reconfiguration and dependency tracking.

Allowing applications to survive hardware failure is an expensive undertaking, which generally involves reengineering software to include complicated recovery logic as well as deploying special-purpose hardware; this represents a severe barrier to improving the dependability of large or legacy applications. We describe the construction of a general and transparent high availability service that allows existing, unmodified software to be protected from the failure of the physical machine on which it runs. Remus provides an extremely high degree of fault tolerance, to the point that a running system can transparently continue execution on an alternate physical host in the face of failure with only seconds of downtime, while completely preserving host state such as active network connections. Our approach encapsulates protected software in a virtual machine, asynchronously propagates changed state to a backup host at frequencies as high as forty times a second, and uses speculative execution to concurrently run the active VM slightly ahead of the replicated system state.

Simulation of P2P systems at the scale of millions of nodes is important because some problems with the protocols or their implementations might not appear at smaller scales. In this work, we propose a parallel message-level simulator, P2PNet, which can simulate P2P systems with up to millions of nodes. P2PNet applies the technique of time expansion and uses real time to synchronize the processing of events among the participating processors. Simulation results show that P2PNet has small overhead compared with a single-processor event-driven simulator, a large speedup when multiple computers are used and no late events.

The current routing algorithms of DHT-based P2P systems have a large end-to-end delay and inconsistent routing performance because of their random selection of identifiers (IDs). In this paper, an Internet topology based overlay construction method is proposed for tree-based DHTs. The node ID is divided into three parts and assigned according to the autonomous system (AS), IP network prefix, and IP address of the node. This algorithm assigns the AS ID prefix based on the AS-level Internet topology. Proximity Neighbor Selection (PNS) is used with topology based ID assignment so that the overlay routing can match the underlying IP routing path. The assignment of AS ID prefixes also takes into account the node densities of ASes to alleviate the ID space load imbalance. Simulation results show that this method can reduce the routing stretch and the standard deviation of the routing stretch without introducing any single points of failure.

This paper presents a proposal for incorporating a group communication paradigm into an object-based environment. Group communication is an interaction involving an arbitrary number of participants. Its application simpli es speci cation and implementation of complex systems. Object-oriented programming can signi cantly pro t from the addition of a group mechanism. Our proposed mechanism, Multiple Object Invocation, provides group communication among objects. The paper rst presents a model of general structure of group services and the use of group communication in this model. Multiple Object Invocation is presented and its characteristics and application in the general model is discussed. Implementation of Multiple Object Invocation in the Emerald object-oriented language and distributed operating system is presented.