This work proposes LiquidListener, a novel liquid volume sensing method for containers. Specifically, it enables the ubiquitous measurement of liquid volume not available in existing work due to i ) dependencies on dedicated sensing hardware (e.g., capacity sensors) and containers (e.g., transparent containers) and ii ) a high training intensity. A key enabler of LiquidListener is listening to singing sounds . When a user taps a container using solid objects, such as pens and teaspoons, the container vibrates freely and produces a singing sound. As the container is filled with more liquid, the pitch of the sound decreases. Based on this relationship, we develop acoustic-based liquid volume sensing algorithms that support the precise measurement of liquid volume while using only a smartphone and requiring minimal user effort for calibration. The extensive experiments demonstrate that LiquidListener can support high accuracy with an average error ratio of 2.3% in sensing the liquid volume in various containers. In addition, the experimental results indicate that it can still maintain a similar level of accuracy in diverse and dynamically changing environments, even without additional calibration.

The growing trend of multi-device ownerships creates opportunities to use applications across devices. However, the current methods of app development/usage remain in the single-device paradigm, which is far below user expectations. For example, it is currently impossible for users to dynamically partition an existing app across different devices to utilize multiple surfaces. We introduce FLUID, a novel multi-device platform that supports simultaneous operation of multiple devices. FLUID aims to i) distribute the user interfaces (UIs) of a single app across multiple devices, ii) support unmodified legacy apps without extra engineering, and iii) support numerous apps with customized UIs. Previous approaches, like screen mirroring and app migration, do not satisfy those goals altogether. However, FLUID is designed to satisfy the goals. It can efficiently deploy UI objects to different devices by identifying only UI states necessary for accurate rendering. And FLUID can execute the distributed UI objects by supporting cross-device method invocations transparently and synchronizing the replicated UIs across devices. Furthermore, FLUID automatically handles unexpected events that may degrade its usability by efficiently maintaining the distributed UIs up to date. Our evaluation using 20 legacy apps shows that FLUID can transparently support numerous apps and is fast enough for interactive use.

Edge TPU (Tensor Processing Unit) is being widely utilized in various edge computing applications as a high-efficiency, low-power accelerator for deep learning computations. However, temperature rise in Edge TPU can lead to performance degradation, reduced stability, and shortened lifespan, necessitating temperature management through thermal modeling. This paper proposes a task-level thermal modeling technique for predicting Edge TPU temperature. The proposed method estimates power consumption of CPU and Edge TPU based on workloads of various deep learning tasks and predicts the convergence temperature of Edge TPU using a steady temperature model. Through experiments, we confirmed that the proposed method accurately predicts Edge TPU temperature for various workloads. The average prediction error was 0.7 degrees C. This study is expected to serve as a foundation for developing temperature management techniques by presenting an effective temperature prediction model that considers the thermal characteristics of Edge TPU.

Samsung Pay, a widely-used mobile payment service, enables users to pay using just their smartphone thanks to Magnetic Secure Transmission (MST). This technology facilitates communication between smartphones and magnetic card terminals by transmitting payment tokens through magnetic waves. Intriguingly, such magnetic waves inherently produce a distinct sound pattern (called MST sound ) containing payment information, which opens up new opportunities for both potential attackers and payment users. That is, MST sound can serve either as a new side channel for attackers to eavesdrop on MST transactions or as an easily accessible communication channel that enhances the payment experience for users. Inspired by these possibilities, we aim to deeply explore the potential of MST sound across these two dimensions, presenting two frameworks with different objectives: MagSnoop and M2APay. The first is the inference framework, which accurately, robustly, and efficiently infers payment tokens by listening to MST sounds. The second is the payment framework, which helps users establish a secure communication channel between MST-supported smartphones and microphone-equipped smartphones by shielding the vulnerability inherent in MST sound. Our experiments with prototypes of these frameworks achieved high accuracy in token inference and data transmission. Furthermore, both MagSnoop and M2APay are capable of accurately decoding tokens in diverse payment environments, including noisy environments and real-world scenarios.

With the recent advances in IoT, there is a growing interest in multi -surface computing, where a mobile app can cooperatively utilize multiple devices' surfaces. We propose a novel framework that seamlessly augments mobile apps with multi -surface computing capabilities. It enables various apps to employ multiple surfaces with acceptable performance.

Deep neural networks (DNNs) have been deployed in many safety-critical real-time embedded systems. To support DNN tasks in real-time, most previous studies focused on GPU or CPU. However, Edge TPU has not yet been studied for real-time guarantees. This paper presents a real-time DNNs framework for Edge TPU to satisfy multiple DNN inference tasks’ timing requirements. The proposed framework provides 1) SRAM allocation and model partitioning techniques and 2) a MIP-based algorithm that determines the amount of SRAM and the number of segments for each task. The experiment result shows that our framework provides 79% higher schedulability than the existing Edge TPU system.

The increasing complexity and memory demands of Deep Neural Networks (DNNs) for real-time systems pose new significant challenges, one of which is the GPU memory capacity bottleneck, where the limited physical memory inside GPUs impedes the deployment of sophisticated DNN models. This paper presents, to the best of our knowledge, the first study of addressing the GPU memory bottleneck issues, while simultaneously ensuring the timely inference of multiple DNN tasks. We propose RT-Swap, a real-time memory management framework, that enables transparent and efficient swap scheduling of memory objects, employing the relatively larger CPU memory to extend the available GPU memory capacity, without compromising timing guarantees. We have implemented RT-Swap on top of representative machine-learning frameworks, demonstrating its effectiveness in making significantly more DNN task sets schedulable at least 72% over existing approaches even when the task sets demand up to 96.2% more memory than the GPU's physical capacity.

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app – explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7 and is able to adapt them to different contexts with near perfect (98.75 accuracy while reducing both latency and cost by 62.5 compared to the GPT-4 powered baseline.

It is essential to support real-time communication in networked control systems, such as automotive systems or smart factories. Several studies have been conducted to schedule real-time flows by adjusting their priorities using software-defined networking (SDN). However, because its architecture was not initially designed for real-time communication, unexpected challenges may occur if it is not utilized carefully. In particular, because SDN does not provide an atomic method for priority adjustment, some flows may violate their timing constraints by receiving additional interference during priority adjustment. In this letter, we propose novel schemes to update the flow priorities safely in SDN-based real-time systems. To this end, we first analyze such problems through two types of cases. Then, we develop schemes that determine a safe procedure for priority adjustment and synchronize the priority settings of all switches. By network emulations, we demonstrate that the above problem occurs in 149 out of 300 flow sets, but our schemes can effectively prevent all of them.

Samsung Pay, one of the most representative mobile payment services, allows mobile users to make payment transactions almost anywhere using only their smartphone. This is thanks to MST (Magnetic Secure Transmission) that supports communication between smartphones and payment terminals for magnetic cards by transferring payment tokens via magnetic waves. Several attack methods have targeted this new technology by eavesdropping on magnetic fields to intercept the tokens, but with the use of dedicated hardware. This paper raises new security concerns for mobile payment users in a different, yet more effective way; by introducing MagSnoop, a novel framework that infers payment tokens from listening to MST sounds generated during the activation of MST payment transactions. More specifically, we first explore the principle, causing the generation of MST sounds, and the fundamental characteristics of these sounds. We then use these observations to infer payment tokens with a high degree of accuracy, robustness, applicability, and data efficiency. Our experiments with a prototype of MagSnoop demonstrate that it can support high accuracy in token inference (more than 77.8%). In addition, MagSnoop can maintain a reasonable level of accuracy regardless of the payment environments (e.g., 69.2% with a noise level of 50 dBA) and even in the real world (an inference success rate of 68.0% with 15 real-world users).

Mobile apps offer a variety of features that greatly enhance user experience. However, users still often find it difficult to use mobile apps in the way they want. For example, it is not easy to use multiple apps simultaneously on a small screen of a smartphone. In this paper, we present A-Mash, a mobile platform that aims to simplify the way of interacting with multiple apps concurrently to the level of using a single app only. A key feature of A-Mash is that users can mash up the UIs of different existing mobile apps on a single screen according to their preferences. To this end, A-Mash 1) extracts UIs from unmodified existing apps (dynamic UI extraction) and 2) embeds extracted UIs from different apps into a single wrapper app (cross-process UI embedding), while 3) making all these processes hidden from the users (transparent execution environment). To the best of our knowledge, A-Mash is the first work to enable UIs of different unmodified legacy apps to seamlessly integrate and synchronize on a single screen, providing an illusion as if they were developed as a single app. A-Mash offers great potential for a number of useful usage scenarios. For instance, a user can mashup UIs of different IoT administration apps to create an all-in-one IoT device controller or one can mashup today's headlines from different news and magazine apps to craft one's own news headline collection. In addition, A-Mash can be extended to an AR space, in which users can map UI elements of different mobile apps to physical objects inside their AR scenes. Our evaluation of the A-Mash prototype implemented in Android OS demonstrates that A-Mash successfully supports the mashup of various existing mobile apps with little or no performance bottleneck. We also conducted in-depth user studies to assess the effectiveness of the A-Mash in real-world use cases.

Samsung Pay, one of the most representative mobile payment services, allows mobile users to make payment transactions almost anywhere using only their smartphone. This is thanks to MST (Magnetic Secure Transmission) that supports communication between smartphones and payment terminals for magnetic cards by transferring payment tokens via magnetic waves. Several attack methods have targeted this new technology by eavesdropping on magnetic fields to intercept the tokens, but with the use of dedicated hardware. This paper raises new security concerns for mobile payment users in a different, yet more effective way; by introducing MagSnoop, a novel framework that infers payment tokens from listening to MST sounds generated during the activation of MST payment transactions. More specifically, we first explore the principle, causing the generation of MST sounds, and the fundamental characteristics of these sounds. We then use these observations to infer payment tokens with a high degree of accuracy, robustness, applicability, and data efficiency. Our experiments with a prototype of MagSnoop demonstrate that it can support high accuracy in token inference (more than 77.8%). In addition, MagSnoop can maintain a reasonable level of accuracy regardless of the payment environments (e.g., 69.2% with a noise level of 50 dBA) and even in the real world (an inference success rate of 68.0% with 15 real-world users).

Mobile apps offer a variety of features that greatly enhance user experience. However, users still often find it difficult to use mobile apps in the way they want. For example, it is not easy to use multiple apps simultaneously on a small screen of a smartphone. In this paper, we present A-Mash, a mobile platform that aims to simplify the way of interacting with multiple apps concurrently to the level of using a single app only. A key feature of A-Mash is that users can mash up the UIs of different existing mobile apps on a single screen according to their preferences. To this end, A-Mash 1) extracts UIs from unmodified existing apps (dynamic UI extraction) and 2) embeds extracted UIs from different apps into a single wrapper app (cross-process UI embedding), while 3) making all these processes hidden from the users (transparent execution environment). To the best of our knowledge, A-Mash is the first work to enable UIs of different unmodified legacy apps to seamlessly integrate and synchronize on a single screen, providing an illusion as if they were developed as a single app. A-Mash offers great potential for a number of useful usage scenarios. For instance, a user can mashup UIs of different IoT administration apps to create an all-in-one IoT device controller or one can mashup today's headlines from different news and magazine apps to craft one's own news headline collection. In addition, A-Mash can be extended to an AR space, in which users can map UI elements of different mobile apps to physical objects inside their AR scenes. Our evaluation of the A-Mash prototype implemented in Android OS demonstrates that A-Mash successfully supports the mashup of various existing mobile apps with little or no performance bottleneck. We also conducted in-depth user studies to assess the effectiveness of the A-Mash in real-world use cases.

Being able to use a single app across multiple devices can bring novel experiences to the users in various domains including entertainment and productivity. For instance, a user of a video editing app would be able to use a smart pad as a canvas and a smartphone as a remote toolbox so that the toolbox does not occlude the canvas during editing. However, existing approaches do not properly support the single-app multi-device execution due to several limitations, including high development cost, device heterogeneity, and high performance requirement. In this paper, we introduce FLUID-XP, a novel cross-platform multi-device system that enables UIs of a single app to be executed across heterogeneous platforms, while overcoming the limitations of previous approaches. FLUID-XP provides flexible, efficient, and seamless interactions by addressing three main challenges: i) how to transparently enable a single-display app to use multiple displays, ii) how to distribute UIs across heterogeneous devices with minimal network traffic, and iii) how to optimize the UI distribution process when multiple UIs have different distribution requirements. Our experiments with a working prototype of FLUID-XP on Android confirm that FLUID-XP successfully supports a variety of unmodified real-world apps across heterogeneous platforms (Android, iOS, and Linux). We also conduct a lab study with 25 participants to demonstrate the effectiveness of FLUID-XP with real users.

In this article, an adaptive real-time communication system is proposed that leverages the software-defined networking (SDN) paradigm to provide end-to-end deadline guarantees. SDN is an emerging networking paradigm that allows control of the network through directly programmable software controllers, offering the flexibility to manage and optimize network resources dynamically. By utilizing these SDN features, a novel networking system, called RT-SDN, is presented that can effectively support real-time communication through cooperative routing and scheduling. To this end, we present a highly efficient routing algorithm that can adaptively reconfigure the routes of existing flows to find bandwidth-guaranteed routes for all flows. We then introduce a new priority assignment scheme that can achieve better deadline guarantees (i.e., better schedulability). In addition, RT-SDN allows routing and scheduling to operate together by employing a feedback loop between them. We implemented a prototype of RT-SDN as an SDN controller and deployed it on a testbed network of 30 BeagleBone devices. Our experiment results show that RT-SDN can be deployed in real-world commodity networks to provide end-to-end deadline guarantees, and the proposed schemes collectively improve the schedulability to a significant degree.

Deep learning has recently been applied to various research areas of design optimization. This study presents the need and effectiveness of adopting deep learning for generative design (or design exploration) research area. This work proposes an artificial intelligent (AI)-based design automation framework that is capable of generating numerous design options which are not only aesthetic but also optimized for engineering performance. The proposed framework integrates topology optimization and deep generative models (e.g., generative adversarial networks (GANs)) in an iterative manner to explore new design options, thus generating a large number of designs starting from limited previous design data. In addition, anomaly detection can evaluate the novelty of generated designs, thus helping designers choose among design options. The 2D wheel design problem is applied as a case study for validation of the proposed framework. The framework manifests better aesthetics, diversity, and robustness of generated designs than previous generative design methods.

A recent trend in the global mobile/IoT industry is the emergence of next-generation smart devices with various screens, thus mobile/IoT market leaders are highly focused on building a new multi-device computing ecosystem based on such new smart devices. Market leaders are not only simply varying their screen sizes, but also competitively launching new devices equipped with innovative screens like foldable and dual-screen phones. However, the current mobile computing ecosystem is restricted by the single device paradigm that allows a user to interact with only one screen tethered to a single device, limiting the potential that the emerging multi-device computing trend provides.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. We present FLUID, a new multi-device platform that allows users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices. In addition, FLUID aims to require no extra development effort to support a wide range of legacy apps that follow the trend of using custom-made UIs. To this end, FLUID analyzes which UI states are necessary to correctly render UI objects, deploys only those states on different devices, and supports cross-device function calls transparently. In this demo, we demonstrate several interesting use cases supported by our Android-based FLUID prototype.