With the recent advances in vision transformers and large language models (LLMs), finetuning costly large models on downstream learning tasks poses significant challenges under limited computational resources. This paper presents a REsource and ComputAtion-efficient Pruning framework (RECAP) for the finetuning of transformer-based large models. RECAP by design bridges the gap between efficiency and performance through an iterative process cycling between pruning, finetuning, and updating stages to explore different chunks of the given large-scale model. At each iteration, we first prune the model with Taylor-approximation-based importance estimation and then only update a subset of the pruned model weights based on the Fisher-information criterion. In this way, RECAP achieves two synergistic and yet conflicting goals: reducing the GPU memory footprint while maintaining model performance, unlike most existing pruning methods that require the model to be finetuned beforehand for better preservation of model performance. We perform extensive experiments with a wide range of large transformer-based architectures on various computer vision and natural language understanding tasks. Compared to recent pruning techniques, we demonstrate that RECAP offers significant improvements in GPU memory efficiency, capable of reducing the footprint by up to 65%.

Federated learning (FL) is a distributed learning framework that inherently provides data privacy and parallel computation capability over a set of participating devices (clients). In real-life applications, these clients can have a great variety in terms of resources (storage, RAM, CPU/GPU speed, network speed, etc.). However, most previous FL studies do not consider this scenario with system heterogeneity and assume that all clients can operate on the same full-size deep neural network (DNN) model. In this work, we demonstrate a scalable FL approach, ScaleFL, which tackles system heterogeneity through hierarchically downscaling the DNN model for clients with limited resources. ScaleFL utilizes early exits to form multi-exit DNN models by injecting early exit networks into the given DNN. During FL, the model is adaptively split along depth (exits) and width (hidden dimensions) based on the resource budget of each participating client. A proof-of-concept demonstration is provided with interactive features, demonstrating the system flow on image classification and NLP benchmark workloads.

Federated learning (FL) is an attractive distributed learning paradigm supporting real-time continuous learning and client privacy by default. In most FL approaches, all edge clients are assumed to have sufficient computation capabilities to participate in the learning of a deep neural network (DNN) model. However, in real-life applications, some clients may have severely limited resources and can only train a much smaller local model. This paper presents ScaleFL, a novel FL approach with two distinctive mechanisms to handle resource heterogeneity and provide an equitable FL framework for all clients. First, ScaleFL adaptively scales down the DNN model along width and depth dimensions by leveraging early exits to find the best-fit models for resource-aware local training on distributed clients. In this way, ScaleFL provides an efficient balance of preserving basic and complex features in local model splits with various sizes for joint training while enabling fast inference for model deployment. Second, ScaleFL utilizes self-distillation among exit predictions during training to improve aggregation through knowledge transfer among subnetworks. We conduct extensive experiments on benchmark CV (CIFAR-10/100, ImageNet) and NLP datasets (SST-2, AgNews). We demonstrate that ScaleFL outperforms existing representative heterogeneous FL approaches in terms of global/local model performance and provides inference efficiency, with up to 2x latency and 4x model size reduction with negligible performance drop below 2%.

This paper presents a transparent Container Network Memory storage device, coined as CNetMem, aiming to address the open problem of unpredictable performance degradation of containers when the working set of an application no longer fits in container memory. First, CNetMem will enable application tenants running in a container to park their working set memory/file to a faster network memory storage by organizing a group of remote memory nodes as remote memory donors. This allows CNetMem to take advantage of remote idle memory on a cluster before resorting to a slow local I/O subsystem like local disk without any modification of host OS or application. Second, CNetMem provides a hybrid batching technique to remove or alleviate performance bottlenecks in the I/O performance critical path for remote memory read/write with replication or disk backup for fault tolerance. Third, CNetMem introduces a rank-based node selection algorithm to find the optimal node for placing remote memory blocks across cluster. This helps CNetMem to reduce the performance impact due to remote memory eviction. Extensive experiments are conducted on three big data applications and four machine learning workloads. The results show that CNetMem achieves up to 172× throughput improvements compared to vanilla Linux and up to 5.9× completion time improvements over existing approaches in big data and ML workload.

We present RDMAbox, a set of low level RDMA optimizations that provide better performance than previous approaches. The optimizations are packaged in easy-to-use kernel and user space libraries for applications and systems in data centers. We demonstrate the flexibility and effectiveness of RDMAbox by implementing a kernel remote paging system and a user space file system using RDMAbox. RDMAbox employs two optimization techniques. First, we suggest RDMA request merging and chaining to reduce the total number of I/O operations to the RDMA NIC. The I/O merge queue at the same time functions as a traffic regulator to enforce admission control and avoid overloading the NIC. Second, we propose Adaptive Polling to achieve higher efficiency of polling Work Completion than existing busy polling while maintaining the low CPU overhead of event trigger. Our implementation of a remote paging system with RDMAbox outperforms existing representative solutions with up to 4x throughput improvement and up to 83% decrease in average tail latency in bigdata workloads, and up to 83% reduction in completion time in machine learning workloads. Our implementation of a user space file system based on RDMAbox achieves up to 5.9x higher throughput over existing representative solutions.

An efficient approach to managing big data workloads is to enable applications to work directly with huge pages. This can effectively avoid or reduce the memory fragmentation problem due to high frequent memory allocation and deallocation and significantly minimize the performance degradation of big data applications. This paper presents XPage, a huge page memory management framework, with three novel features. First, XPage by design can provide automated huge page managements with transparency to both OS and applications. Second, Xpage represents a memory management redesign that brings performance and memory saving to memory intensive applications by supporting dynamic huge page memory management without resorting to splitting huge pages for memory fragmentation. Third but not the least, XPage can efficiently minimize the internal fragmentation without impacting performance of applications. We conduct extensive experiments to evaluate the effectiveness of XPage in minimizing internal memory fragmentation in the presence of dynamic memory intensive big data workloads, by comparing XPage with vanilla Linux using 4KB base page and Linux with 2MB huge page.

Federated learning(FL) is an emerging distributed learning paradigm with default client privacy because clients can keep sensitive data on their devices and only share local training parameter updates with the federated server. However, recent studies reveal that gradient leakages in FL may compromise the privacy of client training data. This paper presents a gradient leakage resilient approach to privacy-preserving federated learning with per training example-based client differential privacy, coined as Fed-CDP. It makes three original contributions. First, we identify three types of client gradient leakage threats in federated learning even with encrypted client-server communications. We articulate when and why the conventional server coordinated differential privacy approach, coined as Fed-SDP, is insufficient to protect the privacy of the training data. Second, we introduce Fed-CDP, the per example-based client differential privacy algorithm, and provide a formal analysis of Fed-CDP with the (∊,δ) differential privacy guarantee, and a formal comparison between Fed-CDP and Fed-SDP in terms of privacy accounting. Third, we formally analyze the privacy-utility tradeoff for providing differential privacy guarantee by Fed-CDP and present a dynamic decay noise-injection policy to further improve the accuracy and resiliency of Fed-CDP. We evaluate and compare Fed-CDP and Fed-CDP(decay) with Fed-SDP in terms of differential privacy guarantee and gradient leakage resilience over five benchmark datasets. The results show that the Fed-CDP approach outperforms conventional Fed-SDP in terms of resilience to client gradient leakages while offering competitive accuracy performance in federated learning.

Since very few contributions to the development of an unified memory orchestration framework for efficient management of both host and remote idle memory have been made, we present Valet, an efficient approach to orchestration of host and remote shared memory for improving performance of memory intensive workloads. The paper makes three original contributions. First, we redesign the data flow in the critical path by introducing a host-coordinated memory pool that works as a local cache to reduce the latency in the critical path of the host and remote memory orchestration. Second, Valet utilizes unused local memory across containers by managing local memory via Valet host-coordinated memory pool, which allows containers to dynamically expand and shrink their memory allocations according to the workload demands. Third, Valet provides an efficient remote memory reclaiming technique on remote peers, based on two optimizations: (1) an activity-based victim selection scheme to allow the least-active-block of data to be selected for serving the eviction requests and (2) a migration protocol to move the least-active-block of data to less-memory-pressured remote node. As a result, Valet can effectively reduce the performance impact and migration overhead on local nodes. Our extensive experiments on both NoSQL systems and Machine Learning (ML) workloads show that Valet outperforms existing representative remote paging systems by up to 282 × throughput improvement and up to 98% latency decrease over conventional OS swap facility for big data and ML workloads, and by up to 5.3 × throughput improvement and up to 80% latency decrease over the state-of-the-art remote paging systems. Valet is open sourced at https://github.com/git-disl/Valet.

Virtual machines (VM) and containers are virtualization technologies that partition computing resources and isolate workloads. They are the foundations for resource consolidation which leads to the success of cloud computing. VMs perform partitioning and isolation at the machine device level while containers do so at the operation system level. The pros and cons of VMs and containers are generally well understood. VMs provide better isolation and security while containers are less resource intensive and perform better. There are various attempts to address the shortcomings in both communities to narrow the gap between the two. In this paper, we review these efforts and discuss their strengths and weaknesses. We also present the cVM architecture with ideas to explore ways of reducing VM resource demand and improving VM performance towards being comparable to those of containers. cVMs bootstrap from a “disk template” where Linux Logical Volume Manager snapshot is used to allow cVMs to share read-only files and directories, thus reducing their disk footprint. Similarly, “memory template” allows cVMs to share read-only code and data memory pages, thus reducing their memory footprint. Finally, cVMs leverage device hotplug capability and lightweight Linux distributions to reduce bootup time without sacrificing too much generality. And cVMs also take advantage of device virtualization and utilize device passthrough to achieve near native I/O performance.

The volume of trajectory data has become tremendously huge in recent years. How to effectively and efficiently maintain and compute such trajectory data has become a challenging task. In this paper, we propose a trajectory spatial and temporal compression framework, namely CLEAN. The key of spatial compression is to mine meaningful trajectory frequent patterns on road network. By treating the mined patterns as dictionary items, the long trajectories have the chance to be encoded by shorter paths, thus leading to smaller space cost. And an error-bounded temporal compression is carefully designed on top of the identified spatial patterns for much low space cost. Mean-while, the patterns are also utilized to improve the performance of two trajectory applications, range query and clustering, without decompression overhead. Extensive experiments on real trajectory datasets validate that CLEAN significantly outperforms existing state-of-art approaches in terms of spatial-temporal compression and trajectory applications.

The volume of trajectory data has become tremendously large in recent years. How to effectively and efficiently search similar trajectories has become an important task. Firstly, to measure the similarity between a trajectory and a query, literature works compute spatial similarity and temporal similarity independently, and next sum the two weighted similarities. Thus, two trajectories with high spatial similarity and low temporal similarity will have the same overall similarity with another two trajectories with low spatial similarity and high temporal similarity. To overcome this issue, we propose to measure the similarity by synchronously matching the spatial distance against temporal distance. Secondly, given this new similarity measurement, to overcome the challenge of searching top- k similar trajectories over a huge trajectory database with non-trivial number of query points, we propose to efficiently answer the top- k similarity search by following two techniques: trajectory database grid indexing and query partitioning. The performance of our proposed algorithms is studied in extensive experiments based on two real data sets.

Data exfiltration attacks have led to huge data breaches. Recently, the Equifax attack affected 147M users and a third-party library - Apache Struts - was alleged to be responsible for it. These attacks often exploit the fact that sensitive data are stored unencrypted in process memory and can be accessed by any function executing within the same process, including untrusted third-party library functions. This paper presents StackVault, a kernel-based system to prevent sensitive stack-based data from being accessed in an unauthorized manner by intra-process functions. Stack-based data includes data on stack as well as data pointed to by pointer variables on stack. StackVault consists of three components: (1) a set of programming APIs to allow users to specify which data needs to be protected, (2) a kernel module which uses unforgeable function identities to reliably carry out the sensitive data protection, and (3) an LLVM compiler extension that enables transparent placement of stack protection operations. The StackVault system automatically enforces stack protection through spatial and temporal access monitoring and control over both sensitive stack data and untrusted functions. We implemented StackVault and evaluated it using a number of popular real-world applications, including gRPC. The results show that StackVault is effective and efficient, incurring only up to 2.4% runtime overhead.

Machine Learning and Artificial Intelligence models are created, trained and used by different entities. The entity that curates data used for the model is frequently different from the entity that trains the model, which is different yet again from the end user of the trained model. The end user needs to trust the received AI model, and this requires having the provenance information about how the model was trained, and the data the model was trained on. This chapter describes how blockchain can be used to track the provenance of training models, leading to better trusted Artificial Intelligence.

Ballooning is a popular solution for dynamic memory balancing. However, existing solutions may perform poorly in the presence of heavy guest swapping. Furthermore, when the host has sufficient free memory, guest virtual machines (VMs) under memory pressure is not be able to use it in a timely fashion. Even after the guest VM has been recharged with sufficient memory via ballooning, the applications running on the VM are unable to utilize the free memory in guest VM to quickly recover from the severe performance degradation. To address these problems, we present MemFlex , a shared memory swapper for improving guest swapping performance in virtualized environment with three novel features: (1) MemFlex effectively utilizes host idle memory by redirecting the VM swapping traffic to the host-guest shared memory area. (2) MemFlex provides a hybrid memory swapping model, which treats a fast but small shared memory swap partition as the primary swap area whenever it is possible, and smoothly transits to the conventional disk-based VM swapping on demand. (3) Upon ballooned with sufficient VM memory, MemFlex provides a fast swap-in optimization, which enables the VM to proactively swap in the pages from the shared memory using an efficient batch implementation. Instead of relying on costly page faults, this optimization offers just-in-time performance recovery by enabling the memory intensive applications to quickly regain their runtime momentum. Performance evaluation results are presented to demonstrate the effectiveness of MemFlex when compared with existing swapping approaches.

Database plays an important role in transactional information systems. One significant performance impacting factor is data lock contention in transaction processing. In order to guide better database design, we propose a novel solution to identify contention hotspots displayed in DBMS transaction logs. To analyze the large volume of data collected in the transaction log, our solution employees big data engine Spark for better computation performance and scalability. A novel algorithm is also introduced to optimize computation for analyzing hotspots in the distributed cluster. The experimental results from a benchmark OLTP workload demonstrate the effectiveness and high scalability of our solution.

Dynamic VM memory management via the balloon driver is a common strategy to manage the memory resources of VMs under changing workloads. However, current approaches rely on kernel instrumentation to estimate the VM working set size, which usually result in high run-time overhead. Thus system administrators have to tradeoff between the estimation accuracy and the system performance. This paper presents iBalloon, a light-weight, accurate and transparent prediction based mechanism to enable more customizable and efficient ballooning policies for rebalancing memory resources among VMs. Experiment results from well known benchmarks such as Dacapo and SPECjvm show that iBalloon is able to quickly react to the VM memory demands, provide up to 54% performance speedup for memory intensive applications running in the VMs, while incurring less than 5% CPU overhead on the host machine as well as the VMs.

Nowadays large enterprises maintain a huge amount of data in multiple backend systems including traditional database systems and recently popular big data systems. In an example of telecom providers, the key business data (e.g., billing information) is maintained in database systems whereas the huge amount of log data is on HDFS with Hive. How to provide insightful analytics on such data becomes a challenging task. Traditional enterprise data warehouse systems with careful database design cannot meet the agile requirement of data scientists to arbitrarily access any useful data (such as the log data). In this paper, we propose a unified computation engine for big data analytics, namely Octopus, to effectively and efficiently bridge data scientists and data warehouse. First, Octopus designs a SQL-alike approach to unify both database queries and machine learning algorithms. Next, Octopus optimizes the running time of such big data analytic tasks by scheduling optimal subtasks to backend systems. A proof-of-concept prototype of Octopus successfully verifies that Octopus can achieve much faster running time than Spark. For example, Octopus outperforms the recent Spark 1.4.0 by 4.58× faster running time to process a complex analytic task, and 5.25× to process a simple aggregation query.

In this paper we present our experiences constructing and testing in-memory data structures designed to be disjoint enough for transactional memory to be profitable as a serialization mechanism with no fallback to traditional locking. Our goal was to restrict memory conflicts to actual contention situations so that transactional memory techniques could be used as efficiently as possible. We describe the hardware transactional execution facility in the IBM zEnterprise EC12 server. We present an order preserving hashed structure that permits insertion, deletion, and traversal operations typically supported by a sorted linked list. We also present a concurrent open addressing hash table structure. We measure the performance and scalability for these data structures on the IBM zEnterprise EC12 server. Our results show near linear scalability of the insertion and deletion operations for up to 96 CPUs. We also discuss transaction abort frequency and hardware/software interactions.

Nowadays large enterprises maintain a huge amount of data in multiple backend systems including traditional database systems and recently popular big data systems. In an example of telecom providers, the key business data (e.g., billing information) is maintained in database systems whereas the huge signaling log data is on HDFS with Hive. How to integrate such data and provide a consolidate query and analytic becomes a challenging task. Neither traditional database warehouse nor recent Big Data system (e.g. Apache Spark and Hadoop) can fully leverage the power of each backend system. In this paper, we build a hybrid data processing engine, called Octopus, to fully integrate backend systems. Given the backend systems, data is distributed at multiple locations. Octopus focuses on the optimization of the amount of data movement. To this end, Octopus proposes a technique of query pushdown for such optimization. A proof-of-concept prototype of Octopus successfully verifies that Octopus can achieve much faster running time than Spark. For example, Octopus outperforms the recent Spark version 1.4.0 by 5.25 X faster running time to process an aggregation query.