Running multiple deep neural networks (DNNs) simultaneously on mobile devices introduces challenges due to constrained computing resources. Previous research has explored the use of heterogeneous processors for accelerating DNN inference but often overlooks thermal issues, which can degrade computing power. In this article, we propose Phoenix, a system specifically designed to enhance the performance of multi-instance DNNs in video applications by maximizing accuracy and ensuring the achievement of a required frame rate. Phoenix allocates DNN tasks to the most suitable hardware processors, understanding complex thermal dynamics through reinforcement learning, and postpones the onset of thermal throttling. Despite optimized task allocation, continuous inference of multiple DNNs can still lead to thermal throttling. To manage performance degradation, Phoenix employs a multi-exit network, adaptively executing inference tasks to ensure consistent frame rates. Phoenix minimizes accuracy loss from early exits by optimally generating and operating multi-exit networks. We evaluated Phoenix using two different benchmarks and Virtual Youtuber streaming application. The results demonstrated that Phoenix effectively enhances device performance by delaying thermal throttling and achieving optimal accuracy while maintaining a consistent frame rate.

Providing fully immersive volumetric videos on mobile devices requires photo-realistic, full-scene rendering with smooth playback experiences. Since traditional 3D representations such as point clouds struggle with visual quality, 3D Gaussian Splatting (3DGS) [4] has emerged as an effective way to represent high-quality 3D scenes. However, existing approaches for 3DGS-based video streaming incur significant rendering overhead, making realtime playback challenging. This article presents Vega [5], a 3DGS-based system that enables fully immersive volumetric video streaming on smartphones. Vega bridges the gap between highquality 3DGS rendering and the constraints of mobile devices—specifically, limited network bandwidth and compute power. The core idea is object-level selective computation, which optimizes both data size and rendering overhead by prioritizing visually significant objects. To realize this idea, Vega introduces a mobile-friendly encoding scheme on the server, in which key frames store full scene data while residual frames encode only dynamic object information to reduce redundancy. On the client side, Vega employs a view-adaptive rendering pipeline that selectively renders visually important objects to meet strict real-time deadlines while effectively utilizing the smartphone's CPU, GPU, and NPU.

Deep neural networks (DNNs) have enabled a wide range of artificial intelligence of things (AIoT) applications, but their increasing complexity poses challenges for deployment on resource-constrained devices. Model compression techniques such as pruning and quantization have been widely adopted to address these challenges; however, they inevitably incur accuracy loss due to information loss. Recently, class-aware pruning has emerged as a promising approach, but existing methods often lack flexibility, as they are typically tailored to fixed target class sets and fail to generalize well to dynamic or broad class distributions. To address this limitation, we propose Mixture of Class-aware Experts (MoCE), a novel framework that combines class-aware pruning with a Mixture of Experts (MoE) architecture. MoCE constructs multiple lightweight experts using class-aware pruning, each specialized for a subset of classes, and employs a shared encoder and a lightweight router to dynamically select the appropriate expert at runtime. Our preliminary results demonstrate the potential of combining class-aware pruning and expert selection to enable accurate and efficient inference on resource-limited AIoT devices.

Sound event detection (SED) based on on-device machine learning (ML) presents considerable energy challenges for ultra-low-power sensor devices. In this article, we propose DASH, a duration-aware SED system designed for energy-constrained sensor devices in domestic environments. As repeated inferences for continuous sound events lead to unnecessary energy consumption, DASH aims to minimize unnecessary inferences by predicting the duration of sound events. However, the variability of sound event durations across different environments and scenarios poses a major challenge in developing a responsive yet energy-efficient duration-aware SED system. To address this, DASH introduces three key solutions: (1) N-probability distribution-based event duration prediction, which identifies checkpoints where new inferences are likely needed; (2) Affinity-guided event classification, which performs low-energy affinity matching at checkpoints to determine whether ML inference is necessary; and (3) Interrupt blocking-enabling cycle-based device state control, which periodically checks for event presence with minimal energy consumption at non-checkpoint times. We implemented DASH on MSP430-based sensor devices deployed in real home environments. Experimental results demonstrate that DASH reduced energy consumption by approximately 97–98% compared to evaluation baselines, with only a 4.7% error rate.

Mobile Augmented Reality (AR) applications demand high-quality, real-time visual prediction, including pixel-level depth and semantics, to enable immersive and context-aware user experiences. Recently, Vision Foundation Models (VFMs) offer strong generalization capabilities on diverse and unseen data, supporting scalable mobile AR experiences. However, deploying VFMs on mobile devices is challenging due to computational limitations, particularly in maintaining both prediction accuracy and real-time performance. In this paper, we present ARIA, the first system that enables on-device inference acceleration of a VFM. ARIA employs the heterogeneity of mobile processors through a parallel and selective inference scheme: full-frame prediction is periodically offloaded to a processor with high parallelism capability like GPU, while low-latency updates on dynamic regions are conducted via a specialized accelerator like NPU. Implemented and evaluated using mobile devices, ARIA achieved significant improvements in accuracy and deadline success rate on diverse real-world mobile AR scenarios.

Although on-device video super-resolution enables high-quality live 360-degree streaming on mobile devices, existing methods often waste energy by overlooking perceived visual quality. In this paper, we present EOS, an energy-efficient on-device super-resolution system for mobile omnidirectional video (ODV) live streaming. EOS reduces energy waste by dynamically adjusting super-resolution complexity based on the predicted visual quality of super-resolved frames. This approach raises two challenges: (1) designing an adaptive inference policy that maximizes energy savings while minimizing degradation in Quality-of-Experience (QoE), and (2) developing a method to predict visual quality under the constraints of mobile ODV live streaming. To tackle these challenges, EOS introduces EOS SR and a No-Reference Upscaling Quality Prediction scheme. EOS SR employs a device-agnostic, scalable deep neural network optimized for mobile devices, with an energy-aware scheduler that jointly selects the optimal super-resolution model and GPU frequency. The No-Reference Upscaling Quality Prediction scheme estimates visual quality across arbitrary viewpoints in real time without requiring high-resolution reference videos. Experiments on commodity smartphones show that EOS reduces average power consumption by 34.6%-49.9% compared to baseline methods, while preserving high visual quality and frame rates.

Mobile Augmented Reality (AR) applications demand high-quality, real-time visual prediction, including pixel-level depth and semantics, to enable immersive and context-aware user experiences. Recently, Vision Foundation Models (VFMs) have offered strong generalization capabilities on diverse and unseen data, supporting scalable mobile AR experiences. However, deploying VFMs on mobile devices is challenging due to computational limitations, particularly in maintaining both prediction accuracy and real-time performance. In this article, we present ARIA [3], the first system that enables on-device inference acceleration of a VFM. ARIA employs the heterogeneity of mobile processors through a parallel and selective inference scheme: full-frame prediction is periodically offloaded to a processor with high parallelism capability like GPU, while lowlatency updates on dynamic regions are conducted via a specialized accelerator like NPU. Implemented and evaluated using mobile devices, ARIA achieved significant improvements in accuracy and deadline success rate on real-world mobile AR scenarios.

For highly immersive mobile volumetric video streaming, it is essential to deliver photo-realistic full-scene content with smooth playback. Unlike traditional representations such as point clouds, 3D Gaussian Splatting (3DGS) has gained attention for its ability to represent high-quality full-scene 3D content. However, our preliminary experiments show that existing methods for 3DGS-based videos fail to achieve smooth playback on mobile devices. In this paper, we propose Vega, a 3DGS-based photo-realistic full-scene volumetric video streaming system that ensures real-time playback on mobile devices. The core idea behind Vega's real-time rendering is object-level selective computation, which allocates computational resources to visually important objects to meet strict rendering deadlines. To enable mobile streaming based on the selective computation, Vega addresses two challenges: (1) designing an encoding scheme that optimizes the data size of videos while being compatible with object-level prioritization, and (2) developing a rendering pipeline that efficiently operates on resource-constrained mobile devices. We implemented an end-to-end Vega system, consisting of a streaming server and an Android application. Experimental results on commodity smartphones show that Vega achieves 30 frames per second (FPS) for full-scene volumetric video streaming while maintaining competitive data size and visual quality compared to existing baselines.

Existing Android systems exhibit energy inefficiency during mobile web browsing due to the lack of awareness of application-level context. Inferring such context from system-level data alone is challenging, but one promising opportunity is using the sequence of task wakeup events, where one task activates another. These sequences show correlation with the type of webpage being used. In this article, we present Ember, a lightweight and responsive power management system for mobile web browsing using only task wakeup sequences. Ember introduces a neural network–based approach to predict optimal CPU clamping values by addressing three key challenges: (1) embedding task names, given as natural-language strings, into meaningful vectors using a Word2Vec-based embedding scheme tailored for task wakeup sequences; (2) minimizing inference overhead with a touch-driven hierarchical inference method that combines lightweight logistic regression with high-accuracy neural networks to balance responsiveness and efficiency; and (3) adapting to within-page interaction dynamics through an interaction-adaptive clamping mechanism that adjusts constraints across different user interaction phases. Implemented on commercial Android smartphones, Ember reduced power consumption by 6.2%–31.2% across a wide range of webpages while maintaining user-perceived quality of experience (QoE).

Reckless usage of electric (e-) scooters causes many injury accidents, raising critical safety concerns. Despite newly introduced regulations, specifically, speed limits and sidewalk driving prohibitions, the number of accidents increases due to the challenges in enforcement. Therefore, a reliable method to detect safety-threatening illegal behaviors of e-scooters is essential to mitigate this growing problem. In this article, we propose SecureRide, a system that accurately detects illegal e-scooter behaviors, i.e., speeding violation and sidewalk riding, at runtime using only battery information, without the need for additional sensors. To this end, we first design a neural network-based illegal behavior predictor that takes sequences of three battery factors, i.e., voltage, current, and capacity, as inputs. The model architecture is optimized based on time constraints, target accuracy, and resource constraints of the target devices. Next, we devise a runtime detection strategy to achieve both high accuracy and low detection time. SecureRide operates in two modes with different predictors–lightweight-quick and complex-accurate models–depending on the driving situation, ensuring both high accuracy and low detection time. We extensively validate SecureRide based on actual driving experiments. Our results show that SecureRide detects illegal behaviors with an accuracy of up to 99.77% within 1.01 seconds.

In e-scooter sharing systems, users randomly select and use e-scooters based on inaccurate battery information. This simple rental policy leads to low profitability on two fronts. First, inaccurate battery information causes unexpected device shutdowns, causing negative user experiences and refunds. Second, randomly selected e-scooters increase operation costs for battery management. In this paper, we propose e-scooter recommendation system, EcoRide, which provides accurate battery estimation and profitable e-scooter selection to maximize profitability of sharing systems. To this end, we propose a battery estimation considering four factors, i.e., battery state, temperature, user weight, and road slope, that affect the available battery energy in e-scooter applications. We define a parameter, dynamic voltage threshold (DVT), to represent dynamically changing battery energy, and use it to estimate battery availability. Next, to achieve cost-effective e-scooter selection, we introduce a multi-agent reinforcement learning (MARL)-based technique to learn policies that minimize operation costs. We define sharing system operation as a MARL problem with an objective function based on battery management costs. To cope with unstable training due to a wide service area and multiple requests, a centralized training technique is adopted. The proposed battery estimation and e-scooter selection technique are validated through actual driving tests and a sharing system simulator, respectively. Additionally, our case study using open data from Washington D.C. demonstrates a profit gain of up to 68% with EcoRide.

Sound event detection (SED) using lightweight sensor device has recently gained attention as a practical means to capture context and activities especially in domestic environments. However, SED applications running on sensor device are severely constrained by device’s energy capacity. One solution is to offload a portion of inference to server for reducing runtime complexity, i.e., energy consumption, of sensor device. Offloading should consider the trade-off between computation and data transmission costs adequately; more computation on sensor device reduces data to be transmitted and vice versa. To address this challenge, we propose SEDAC (Sound Event Detection with Attention-based audio Compression), a novel technique for split learning in SED that compresses data from sensor device to offload less data. SEDAC compresses the input of SED models, or Mel spectrograms, with minimal computation in sensor device. Rather than directly compressing the input, SEDAC achieves data compression by selectively capturing the key parts of sound events using an attention mechanism. The scheme also modifies an existing loss function and employs knowledge distillation to mitigate potential loss of SED accuracy due to data compression. Our evaluation shows that SEDAC outperforms the state-of-the-art data compressive split learning schemes, up to about 30%. Furthermore, our real-world deployment demonstrates that sensor devices with SEDAC successfully operate with minimal energy and memory overhead.

3D object detection with omnidirectional views enables safety-critical applications such as mobile robot navigation. Such applications increasingly operate on resource-constrained edge devices, facilitating reliable processing without privacy concerns or network delays. To enable cost-effective deployment, cameras have been widely adopted as a low-cost alternative to LiDAR sensors. However, the compute-intensive workload to achieve high performance of camera-based solutions remains challenging due to the computational limitations of edge devices. In this paper, we present Panopticus, a carefully designed system for omnidirectional and camera-based 3D detection on edge devices. Panopticus employs an adaptive multi-branch detection scheme that accounts for spatial complexities. To optimize the accuracy within latency limits, Panopticus dynamically adjusts the model's architecture and operations based on available edge resources and spatial characteristics. We implemented Panopticus on three edge devices and conducted experiments across real-world environments based on the public self-driving dataset and our mobile 360 camera dataset. Experiment results showed that Panopticus improves accuracy by 62 average given the strict latency objective of 33ms. Also, Panopticus achieves a 2.1× latency reduction on average compared to baselines.

We propose a cross-device web solution, called Vulture, which distributes graphical user interface (GUI) elements of apps across multiple devices without requiring modifications of web apps or browsers. Several challenges should be resolved to achieve the goals. First, the peer-server configuration should be efficiently established to distribute web resources in cross-device web environments. Vulture exploits an in-browser virtual proxy that runs the web server's functionality in web browsers using a virtual HTTP scheme and a relevant API. Second, the functional consistency of web apps must be ensured in GUI-distributed environments. Vulture solves this challenge by providing a single-browser illusion with a two-tier document object models (DOM) architecture, which handles view state changes and user input seamlessly in cross-device environments. We implemented Vulture and extensively evaluated the system under various combinations of operating platforms, devices, and network capabilities while running 50 real web apps. The experiment results show that the proposed scheme provides functionally consistent cross-device web experiences by allowing fine-grained GUI distribution. We also confirmed that the in-browser virtual proxy reduces the GUI distribution time and the view change reproduction time by averages of 38.47% and 20.46%, respectively.

The capability of energy harvesting application powered by indoor photovoltaic energy is severely affected by dynamic light environments. Accordingly, accurate understanding of the target environment and deploying energy harvesting sensors is practically very hard. In this article, we propose HarvAR, which manages photovoltaic energy harvesting sensors with mobile augmented reality (AR)-empowered techniques. HarvAR utilizes the error-prone RGBD data of mobile device to construct a digital twin (DT), performing depth error compensation and estimating the optical properties of the target space. Using the DT, the proposed system predicts the harvesting capability with low overhead, and recommends adequate locations for installing or relocating harvesting sensors. We implemented the HarvAR system and evaluated its accuracy and efficiency in three indoor environments. Our experiments show that DT configuration and harvesting prediction can be performed in minutes, compared to over 10 h using existing techniques, and harvesting prediction is provided with less than 20% error.

Energy-neutral operation (ENO) aims to provide near-perpetual device operation using energy harvested from ambient environments. Existing ENO techniques, however, have two key problems. The batteries used in harvesting devices have inherently limited lifespans, and the device experiences a long cold-start time when charging the battery. In this paper, we propose a long-lasting and highly responsive batteryless system, called RENO, to solve the problems that occur in energy harvesting devices. Using a supercapacitor to store energy, RENO maximizes the responsiveness in ENO especially running in dynamic harvesting environments such as indoor light energy harvesting. Combining the intermittent characteristics of power-neutral operation (PNO), RENO allows dual-mode operation of PNO and ENO, depending on the current harvesting capability. The device works as a PNO device when charging the energy storage, solving the ENO cold-start issue, while the harvested energy is efficiently managed with ENO. For this dual-mode operation, RENO provides hardware and software that handle the switch between PNO and ENO effectively. Application developers are provided with a well-defined API, which enables energy-efficient development of applications without detailed knowledge of the target hardware. Using the API, developers simply declare a task to be executed as either PNO or ENO, and the rest is handled by the system. The prototype system is implemented, and its functionality is evaluated in controlled environments. We also validate the proposed system with two real-world applications, proving the efficacy of dual-mode batteryless operation.

Detecting a structural anomaly, such as a damaged propeller or motor, is crucial for mission-critical operation of unmanned aerial vehicles (UAVs). The existing solutions often fail to detect structural anomalies because the pre-defined parameters required for the solution are limited in reflecting the flight pattern or the external environment, such as wind conditions. In this paper, we propose a method for detecting structural anomalies in quadcopter UAVs, using only regular data and specifically considering flight patterns and runtime flight conditions. To this end, we employ a long short-term memory (LSTM) autoencoder model to learn complex features of regular flight data. While flying the UAV, the trained model estimates the degree of outlierness of the incoming data and assesses abnormal behavior of UAV by adaptively considering its movement. This way, the proposed method accurately detects structural anomalies in UAVs regardless of the runtime environment or flight mission. Our experiment results with an off-the-shelf UAV show that the proposed approach detects diverse structural anomalies by an average of 98.6% specificity and 90.3% sensitivity.

Optimizing deep neural networks (DNNs) running on resource-constrained devices, such as energy harvesting sensor devices, poses unique challenges due to the limited memory and varying energy conditions. Existing efforts have shown that deploying a multi-exit network mitigates the problem by allowing tradeoffs between accuracy and computational complexity. However, previous works did not fully consider two essential requirements: optimized neural architecture and optimized inference policy. In this paper, we present HarvNet, which comprises two complementary techniques for generating and operating a multi-exit network for energy harvesting devices. First, we provide a neural architecture search scheme, HarvNAS, which configures the best multi-exit architecture while meeting memory and energy constraints. Second, HarvSched learns and constructs the best progressive inference policy with different energy constraints by considering runtime factors, such as the harvesting status and the energy storage level. We implemented HarvNAS using the TensorFlow framework and then implemented and evaluated HarvSched on an MSP430-based sensor device. The evaluation showed that HarvNAS generated a model with up to 2.6%p higher accuracy while saving up to 70% of memory compared to the existing technique, and HarvSched enabled zero-downtime operation of the generated model.

Duty cycle management is critical for the energy-neutral operation of batteryless devices. Many efforts have been made to develop an effective duty cycling method, including machine-learning-based approaches, but existing methods can barely handle the dynamic harvesting environments of batteryless devices. Specifically, most machine-learning-based methods require the harvesting patterns to be collected in advance, as well as manual configuration of the duty-cycle boundaries. In this article, we propose a configuration-free duty cycling scheme for batteryless devices, called CTRL, with which energy harvesting nodes tune the duty cycle themselves adapting to the surrounding environment without user intervention. This approach combines reinforcement learning (RL) with a control system to allow the learning algorithm to explore all possible search space automatically. The learning algorithm sets the target State of Charge (SoC) of the energy storage, instead of explicitly setting the target task frequency at a given time. The control system then satisfies the target SoC by controlling the duty cycle. An evaluation based on the real implementation of the system using publicly available trace data shows that CTRL outperforms state-of-the-art approaches, resulting in 40% less frequent power failures in energy-scarce environments while achieving more than ten times the task frequency in energy-rich environments.