Edge computing is increasingly proposed as a solution for reducing resource consumption of mobile devices running simultaneous localization and mapping (SLAM) algorithms, with most edge-assisted SLAM systems assuming the communication resources between the mobile device and the edge server to be unlimited, or relying on heuristics to choose the information to be transmitted to the edge. This paper presents AdaptSLAM, an edge-assisted visual (V) and visual-inertial (VI) SLAM system that adapts to the available communication and computation resources, based on a theoretically grounded method we developed to select the subset of keyframes (the representative frames) for constructing the best local and global maps in the mobile device and the edge server under resource constraints. We implemented AdaptSLAM to work with the state-of-the-art open-source V-and VI-SLAM ORB-SLAM3 framework, and demonstrated that, under constrained network bandwidth, AdaptSLAM reduces the tracking error by 62% compared to the best baseline method.

Today’s Internet must support applications with increasingly dynamic and heterogeneous connectivity requirements, such as video streaming and the Internet of Things. Yet current network management practices generally rely on pre-specified network configurations, which may not be able to cope with dynamic application needs. Moreover, even the best-specified policies will find it difficult to cover all possible scenarios, given applications’ increasing heterogeneity and dynamic network conditions, e.g., on volatile wireless links. In this work, we instead propose a model-free learning approach to find the optimal network policies for current network flow requirements. This approach is attractive as comprehensive models do not exist for how different policy choices affect flow performance under changing network conditions. However, it can raise new challenges for online learning algorithms: policy configurations can affect the performance of multiple flows sharing the same network resources, and this performance coupling limits the scalability and optimality of existing online learning algorithms. In this work, we extend multi-armed bandit frameworks to propose new online learning algorithms for protocol selection with provably sublinear regret under certain conditions. We validate the optimality and scalability of our algorithms through data-driven simulations and testbed experiments. (An extended abstract of this work was accepted by IEEE ICNP as a short paper Zhang et al . (2019)).

The performance of simultaneous localization and mapping (SLAM) systems is impacted by the constrained computation capabilities of mobile devices. Given that the advancement of these systems relies on accurate evaluation of SLAM performance, this issue is exacerbated by the difficulty in evaluating SLAM performance in practice, due to the unavailability of ground truth data. In this demo, we present SpacecraftWalk, a resource-efficient SLAM framework that constructs maps (of the environments) with minimal uncertainty under resource budgets. SpacecraftWalk is evaluated within virtual spacecraft environments (meeting NASA lighting standards) in game engine-based emulators that generate ground truth automatically. Demo participants will navigate in virtual environments while creating their own moving trajectories for evaluating SLAM. They will develop an intuition for how uncertainty-based map construction improves resource efficiency. This demonstration accompanies [1].

Federated learning (FL) has attracted increasing attention as a promising technique to drive a vast number of edge devices with artificial intelligence. However, it is very challenging to guarantee the efficiency of a FL system in practice due to the heterogeneous computation resources on different devices. To improve the efficiency of FL systems in the real world, asynchronous FL (AFL) and semi-asynchronous FL (SAFL) methods are proposed such that the server does not need to wait for stragglers. However, existing AFL and SAFL systems suffer from poor accuracy and low efficiency in realistic settings where the data is non-IID distributed across devices and the on-device resources are extremely heterogeneous. In this work, we propose FedSEA - a semi-asynchronous FL framework for extremely heterogeneous devices. We theoretically disclose that the unbalanced aggregation frequency is a root cause of accuracy drop in SAFL. Based on this analysis, we design a training configuration scheduler to balance the aggregation frequency of devices such that the accuracy can be improved. To improve the efficiency of the system in realistic settings where the devices have dynamic on-device resource availability, we design a scheduler that can efficiently predict the arriving time of local updates from devices and adjust the synchronization time point according to the devices' predicted arriving time. We also consider the extremely heterogeneous settings where there exist extremely lagging devices that take hundreds of times as long as the training time of the other devices. In the real world, there might be even some extreme stragglers which are not capable of training the global model. To enable these devices to join in training without impairing the systematic efficiency, Fed-SEA enables these extreme stragglers to conduct local training on much smaller models. Our experiments show that compared with status quo approaches, FedSEA improves the inference accuracy by 44.34% and reduces the systematic time cost and local training time cost by 87.02× and 792.9×. FedSEA also reduces the energy consumption of the devices with extremely limited resources by 752.9×.

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

Meditation, a mental and physical exercise which helps to focus attention and reduce stress has gained more popularity in recent years. However, meditation requires a concerted effort and regular practice. To explore the feasibility of using Augmented Reality(AR) Devices to assist in meditating, we recruited ten subjects to perform a five-minute meditation task integrated into AR devices. Heart Rate, Heart Rate Variability, and skin conductance response(SCR) are analyzed based on an Electrocardiogram(ECG), Electrodermal activity to monitor the physiological changes during and after a meditation session. Additionally, participants filled out surveys containing the Perceived Stress Questionnaire (PSQ), a clinically validated survey designed to evaluate stress levels before and after meditation to analyze the change in stress levels. Finally, we found significant differences in Heart Rate and Mean SCR Recovery Time for participants between the three study procedure periods (before, during, and after guided meditation).

Kinetic energy harvesting technologies have been progressively used to power wearable devices and to sense the context through energy generation patterns. However, detecting human activities with signals from kinetic harvesters still needs improvement due to the use of approaches based on handcrafted features and the overfitting to device location or subjects. Hence, in this article, we present a deep learning architecture that leverages the feature extraction capability of the convolutional neural networks and the construction of the temporal sequences of recurrent neural networks to improve existing classification results. To provide sufficient data for the deep learning classifier, we propose three data augmentation methods to increase intraclass variance simulating new users performing the same activities. The proposed architecture outperforms existing approaches of kinetic harvesting-based human activity recognition by 13% of accuracy when the training data are augmented with the proposed methods. Finally, given the dependency of kinetic harvesting signals on device location and subjects, we employ transfer learning to improve the classification performance when the system is exposed to new subjects and locations. Transfer learning helps to increase classification performance by 30% when the device location is changed and 35% when the data come from a new subject.

Markerless augmented reality (AR) has the potential to provide engaging experiences and improve outcomes across a wide variety of industries; the overlaying of virtual content, or holograms, onto a view of the real world without the need for predefined markers provides great convenience and flexibility. However, unwanted hologram movement frequently occurs in markerless smartphone AR due to challenging visual conditions or device movement, and resulting error in device pose tracking. We develop a method for measuring hologram positional errors on commercial smartphone markerless AR platforms, implement it as an open-source AR app, HoloMeasure, and use the app to conduct systematic quantitative characterizations of hologram stability across 6 different user actions, 3 different smartphone models, and over 200 different environments. Our study demonstrates significant levels of spatial instability in holograms in all but the simplest settings, and underscores the need for further enhancements to pose tracking algorithms for smartphone-based markerless AR.

Plant Spike is an in situ low-cost sensor system that is wireless, miniature, and low powered. It can be seamlessly implanted in subsurface locations across major cities to measure urban soil health. Plant Spike incorporates noncontact soil moisture monitoring, temperature monitoring, light intensity monitoring, advanced power management, and Bluetooth low energy transmit-only communication for transmitting information to a client device. With a novel combination of aggressive power reduction techniques, the system's lifetime is over two years with a 500-mAh battery. By connecting on-board sensors to a single-chip microcontroller, the total component and assembly cost of each module is less than $10. The sensor system has been tested within an urban soil testbed located on Columbia University's Morningside Campus in New York City as well as street tree pits located in Morningside Heights, proving the functionality and robustness of the system. Plant Spike is able to measure temperature and light ranges that are comparable to the fluctuations experienced by soils located within the climate zone of New York City.

Cloud computing continues to provide services related to storage and computation by managing a pool of resources that usually costs more financially as well as in terms of latency. On the other hand, due to the distribution of applications and services closer to the end-users, academics and industry experts are now advocating for going from large, centralized cloud computing infrastructures to a range of computing nodes located at the edge of the network. Principally, fog computing extends the services and resources of the cloud closer to users, which facilitates the leveraging of available services and resources in the edge networks. Fog computing is an attractive solution to delay- and resource-constrained large-scale wireless and real-time applications, with its features such as low latency, location awareness, and its capacity to process large numbers of nodes with wireless access.

The collaboration among mobile devices to form an edge cloud for sharing computation and data can drastically reduce the tasks that need to be transmitted to the cloud. Moreover, reinforcement learning (RL) research has recently begun to intersect with edge computing to reduce the amount of data (and tasks) that needs to be transmitted over the network. For battery-powered Internet of Things (IoT) devices, the energy consumption in collaborating edge devices emerges as an important problem. To address this problem, we propose an RL-based Droplet framework for autonomous energy management. Droplet learns the power-related statistics of the devices and forms a reliable group of resources for providing a computation environment on-the-fly. We compare the energy reductions achieved by two different state-of-the-art RL algorithms. Further, we model a reward strategy for edge devices that participate in the mobile device cloud service. The proposed strategy effectively achieves a 10% gain in the rewards earned compared to state-of-the-art strategies.

Today’s Internet must support applications with increasingly dynamic and heterogeneous connectivity requirements, such as video streaming and the Internet of Things. Yet current network management practices generally rely on pre-specified flow configurations, which cannot cover all possible scenarios. In this work, we instead propose a model-free learning approach to automatically optimize the policies for heterogeneous network flows. This approach is attractive as no existing comprehensive models quantify how different policy choices affect flow performance under dynamically changing network conditions. We extend multi-armed bandit frameworks to propose new online learning algorithms for protocol selection, addressing the challenge of policy configurations affecting the performance of multiple flows sharing the same network resources. This performance coupling limits the scalability and optimality of existing online learning algorithms. We theoretically prove that our algorithm achieves a sublinear regret and demonstrate its optimality and scalability through data-driven simulations.

We study a multi-agent partially observable environment in which autonomous agents aim to coordinate their actions, while also learning the parameters of the unknown environment through repeated interactions. In particular, we focus on the role of communication in a multi-agent reinforcement learning problem. We consider a learning algorithm in which agents make decisions based on their own observations of the environment, as well as the observations of other agents, which are collected through communication between agents. We first identify two potential benefits of this type of information sharing when agents’ observation quality is heterogeneous: (1) it can facilitate coordination among agents, and (2) it can enhance the learning of all participants, including the better informed agents. We show however that these benefits of communication depend in general on its timing, so that delayed information sharing may be preferred in certain scenarios.

Fog computing, which distributes computing resources to multiple locations between the Internet of Things (IoT) devices and the cloud, is attracting considerable attention from academia and industry. Yet, despite the excitement about the potential of fog computing, few comprehensive quantitative characteristics of the properties of fog computing architectures have been conducted. In this paper we examine the properties of task completion latencies in fog computing. First, we present the results of our empirical benchmarking-based study of task completion latencies. The study covered a range of settings, and uniquely considered both traditional and serverless fog computing execution points. It demonstrated the range of execution point characteristics in different locations and the relative stability of latency characteristics for a given location. It also highlighted properties of serverless execution that are not incorporated in existing fog computing algorithms. Second, we present a framework we developed for co-optimizing task completion quality and latency, which was inspired by the insights of our empirical study. We describe fog computing task assignment problems we formulated under this framework, and present the algorithms we developed for solving them.

Fog computing, the distribution of computing resources closer to the end devices along the cloud-to-things continuum, is recently emerging as an architecture for scaling of the Internet of Things (IoT) sensor networking applications. Fog computing requires novel computing program decompositions for heterogeneous hierarchical settings. To evaluate these new decompositions, we designed, developed, and instrumented a fog computing testbed that includes cloud computing and computing gateway execution points collaborating to finish complex data analytics operations. In this interactive demonstration we present one fog-specific algorithmic decomposition we recently examined and adapted for fog computing: a multi-execution point linear regression decomposition that jointly optimizes operation latency, quality, and costs. The demonstration highlights the role fog computing can play in future sensor networking architectures, and highlights some of the challenges of creating computing program decompositions for these architectures. An annotated video of the demonstration is available at [5].

This article focuses on a new type of wireless devices in the domain between RFIDs and sensor networks—Energy-Harvesting Active Networked Tags (EnHANTs). Future EnHANTs will be small, flexible, and self-powered devices that can be attached to objects that are traditionally not networked (e.g., books, furniture, toys, produce, and clothing). Therefore, they will provide the infrastructure for various tracking applications and can serve as one of the enablers for the Internet of Things. We present the design considerations for the EnHANT prototypes, developed over the past 4 years. The prototypes harvest indoor light energy using custom organic solar cells, communicate and form multihop networks using ultra-low-power Ultra-Wideband Impulse Radio (UWB-IR) transceivers, and dynamically adapt their communications and networking patterns to the energy harvesting and battery states. We describe a small-scale testbed that uniquely allows evaluating different algorithms with trace-based light energy inputs. Then, we experimentally evaluate the performance of different energy-harvesting adaptive policies with organic solar cells and UWB-IR transceivers. Finally, we discuss the lessons learned during the prototype and testbed design process.

Numerous energy harvesting wireless devices that will serve as building blocks for the Internet of Things (IoT) are currently under development. However, there is still only limited understanding of the properties of various energy sources and their impact on energy harvesting adaptive algorithms. Hence, we focus on characterizing the kinetic (motion) energy that can be harvested by a wireless node with an IoT form factor and on developing energy allocation algorithms for such nodes. In this paper, we describe methods for estimating harvested energy from acceleration traces. To characterize the energy availability associated with specific human activities (e.g., relaxing, walking, cycling), we analyze a motion dataset with over 40 participants. Based on acceleration measurements that we collected for over 200 hours, we study energy generation processes associated with day-long human routines. We also briefly summarize our experiments with moving objects. We develop energy allocation algorithms that take into account practical IoT node design considerations, and evaluate the algorithms using the collected measurements. Our observations provide insights into the design of motion energy harvesters, IoT nodes, and energy harvesting adaptive algorithms.