Photoplethysmography (PPG) has become a cornerstone of physiological sensing in wearable devices, enabling non-invasive monitoring of heart rate and related biomarkers. However, its reliability deteriorates sharply under dynamic, high-intensity, or non-periodic motions such as those in sports, where existing datasets fail to capture realistic wrist dynamics. To address this gap, we introduce PPG-Sport, the first large-scale dataset designed for heart rate monitoring from wrist-worn PPG under real sports conditions. The PPG-Sport dataset includes synchronized PPG, inertial measurement unit (IMU), and electrocardiography (ECG) recordings from both wrists of 30 participants across six representative activities: stationary, walking, running, badminton, table tennis, and basketball, amounting to 48 hours of multimodal data. PPG-Sport uniquely captures three critical properties absent in prior datasets: (1) non-periodicity, reflecting irregular and broadband motion patterns; (2) high intensity, with frequent, large-magnitude accelerations; and (3) bilateral asymmetry, caused by distinct functional roles of the dominant and non-dominant hands during sports. We further establish a deep-learning-based benchmark that combines both temporal and spectral representations of PPG and IMU signals to evaluate heart rate estimation performance. Experimental results show that models trained only on conventional periodic activities fail drastically in sports scenarios. Although incorporating sports data mitigates the degradation, significant errors remain. These findings highlight PPG-Sport as an essential and challenging benchmark for developing motion-robust physiological sensing algorithms during sports activities. Dataset and benchmark code are available at: https://github.com/LaserHu/PPG-Sport.
Prolonged desk work has become a major source of musculoskeletal strain, leading to fatigue, discomfort, and reduced mobility among sedentary professionals. Although prior interventions such as standing desks and exergames promote movement, most overlook cumulative postural imbalance and environmental constraints that shape ergonomic risk. We present Adaptive Stretching Pipeline (ASP), a mixed-reality (MR) system that delivers personalized stretching guidance during work breaks. ASP continuously monitors posture through joint-angle and duration analysis and leverages a vision-language model (VLM) to assess workspace supports such as desks or walls. Based on an integrated need-scoring model, the system generates adaptive MR stretching sessions targeting fatigued body regions during microbreak. A three-phase formative study informed the system design, including expert interviews on clinical stretching practices, workplace observations to identify environmental supports, and stretch video analysis to construct a stretching library. A user study with 15 participants compared ASP to a Bend and randomised condition. Our results provide initial evidence that such adaptive mechanisms can shape perceived effort, engagement, and time perception in short-term interactions.
Smart homes are receiving growing interest in global markets. However, current smart home assistant systems either control appliances solely through users' explicit commands or rely on cloud-based large language models (LLMs) for vague command understanding, which brings drawbacks such as high latency, privacy concern, and high cost. In this work, we propose VCU-LLM, the first system to deploy LLMs on edge devices for local vague command understanding and smart device control plan generation. VCU-LLM introduces a novel vague command knowledge retrieval algorithm that refines device-related information in the input prompt, thereby accelerating the LLM's on-device inference and reducing task complexity. We further construct a dataset for LLM fine-tuning to simulate the use of smart home assistants in controlling devices across different households. During inference, a customized KV-cache technique is applied for further inference acceleration. Our evaluations with both human-based and LLM-based scoring demonstrate that VCU-LLM improves the quality of generated control plans by an average of 43.3% compared with SOTA baselines, while reducing time overhead by an average of 8.44x compared with other on-device baselines. We also implement VCU-LLM through a case study in a real home environment, demonstrating its feasibility in real-world application.
End-users of immersive Augmented Reality (AR) applications struggle to discover and execute unfamiliar mid-air gestures. We introduce ARG esture A id , a novel proof-of-concept conversational assistant that provides adaptive, context-aware assistance tailored to users' tasks and gestural errors. By combining users' voice-based task descriptions with formalized representations of hand gesture data together with LLMs, ARG esture A id dynamically supports users through: (1) feedforward , directing users toward correct pose, position, and placement before initiating gestures, and (2) corrective feedback , offering targeted, adaptive instructions to address user errors during discovery and execution. Our initial feasibility study (N=15), which compared users' real-world interaction data against a manually labeled reference dataset, shows that ARG esture A id achieved an overall accuracy of 81.3%. Our follow-up user study (N=15) evaluated ARG esture A id against a BASELINE LLM-based voice assistant across unimanual and bimanual gestures. Results indicate that ARG esture A id achieved higher task completion rates and was rated more favorably for perceived discoverability and usefulness. These findings suggest that adaptive, in-context feedforward and corrective feedback, grounded in formal gesture representations and observed error patterns, may support users during mid-air gesture discovery and execution, and help them refine their understanding of required gestures. We discuss key considerations for designing future adaptive, context-aware AR systems in AR environments.
Honey bees play a vital role in global agriculture. Understanding the movement of a queen bee within a hive box is essential for advancing both biological research and practical apiculture. In this paper, we present SOPAR, a novel Radio Frequency (RF) Sub-harmonic Oscillating Parametric Resonator designed for non-invasive localization and tracking of a queen bee inside a full-size hive box. The core component of SOPAR is a lightweight RF backscatter tag that can be attached to a bee's thorax without affecting its normal behavior. The tag comprises two passive resonators: (i) an inner spiral inductor bridged by a varactor diode and (ii) an outer circular inductor with a gap bridged by a chip capacitor. The outer inductor harvests energy from the external excitation signal, driving the oscillation of the inner spiral resonator to produce sub-harmonic backscattered signals at half the excitation frequency. The frequency separation between excitation and backscatter signals eliminates self-interference at the RF reader, significantly improving signal-to-noise ratio (SNR) and detection range. The layout of the two resonators is meticulously optimized to maximize magnetic coupling, thereby minimizing the overall tag size. Building on this dual-resonator tag, we design an RF reader with a Bayesian estimation algorithm that localizes the tagged bee by exploiting the spatio-temporal characteristics of the sub-harmonic backscattered signals. We have built a prototype of SOPAR, featuring a tag with a diameter of only 3.7 mm and a weight of less than 10 mg. Extensive experiments demonstrate that SOPAR achieves a median localization error of 3.7 cm when tracking a queen bee in a full-size hive box. Moreover, the results confirm that SOPAR remains robust under diverse environmental conditions.
The rise of digital twins, particularly in smart manufacturing and automated logistics, demands ultra-high-precision indoor localization that existing technologies struggle to provide. Conventional wireless localization methods, which rely on channel state information to estimate a limited set of parameters, neglect critical details of the sensing scene, leading to suboptimal accuracy. This paper presents RayLoc, a novel approach that addresses this limitation by reformulating wireless indoor localization as an inverse problem. By leveraging the digital twin of an environment, our method infers the precise scene parameters that generate the measured CSI, enabling highly accurate localization. At the core of our solution is a fully differentiable ray-tracing simulator that enables backpropagation to all sensing-critical parameters, including target locations. To establish a robust localization context, RayLoc first constructs a high-fidelity sensing scene by refining a coarse-grained background model, often derived from the initial digital twin. Furthermore, to overcome optimization unfriendliness of CSI-based loss landscapes, RayLoc introduces Gaussian kernal smoothing combined with an adaptive convergence strategy to mitigate sparse gradients and local minima. Extensive experiments showcase that RayLoc not only outperforms traditional localization baselines but is also able to generalize to different sensing environments, paving the way for its application in next-generation, high-accuracy-dependent systems.
Sensing-based stress management systems integrate self-report, physiological, and contextual data to assess users' perceived stress and physiological indicators related to stress responses and deliver just-in-time interventions. Prior work has largely focused on improving detection accuracy or evaluating intervention effectiveness, with relatively limited attention to how users interpret and make sense of these data in everyday work contexts. Our study addresses this gap by examining how emotional labor workers construct meaning around alignment and divergence between perceived stress and physiological responses to stress. We present a four-week in-the-wild study with 19 call center workers, combining mobile and wearable sensing, a reflection dashboard, and in-depth interviews. Our findings show that participants interpreted stress indicators through work context, bodily conditions, and prior experiences, and treated divergence as an informative cue rather than a simple error. They also recognized that perceived and physiological responses to stress could change differently before and after micro-interventions. Based on these findings, we discuss design implications for sensing-based stress management systems that support stress data literacy and flexible, context-grounded stress sensemaking in practice.
Personalized services in smart environments based on historical context have become an important demand. A promising direction is to leverage the strong reasoning abilities of Large Language Models (LLMs). However, enabling personalization requires persistent context memory. The key challenge lies in how to efficiently organize long-term device and sensor logs, and how to extract user behavior patterns and preferences from them. To address this challenge, we propose MemAura, a memory management system for smart environments. MemAura consists of three core modules: (1) Memory Graph Manager, which efficiently organizes contextual memory, supports fast retrieval, and provides user patterns and preference information to the LLM; (2) Periodic Pattern Predictor, which predicts periodic behavioral patterns and preferences at a given timestamp; (3) User Profiling Scheme, which transforms commands into explict intent units and incrementally updates and forgets high-level profiles over time. We evaluate MemAura against multiple baselines across two command datasets. With GPT-4o as the backbone, it achieves 100% accuracy and 38%-43% personalization rate. When deployed with lightweight local models, MemAura still achieves up to 95% accuracy, with an average personalization rate of up to 37.5%. However, when using the same lightweight LLM, vanilla only achieves 28%-56% accuracy, with a personalization rate of at most 21%. Vector-RAG achieves 54-83% accuracy and up to 25% personalization with lightweight models. While it outperforms vanilla, it still lags behind MemAura. The results show that MemAura maintains high accuracy and personalization across LLMs of different sizes, with practical token usage and latency. We also conduct a series of experiments to validate the robustness of MemAura. A user study with 8 participants further confirms its usability. The participants comprehensively experienced and evaluated its superior performance and service quality. Our work enables LLMs to better understand user intent and efficiently extract patterns and preferences from contextual memory, thereby making smart living spaces more efficient, context-aware, and user-centric.
The increasing digitalization of mobility turns bicycles into networked information platforms. However, applications such as navigation, music control, and fitness tracking require interactions that can distract attention from traffic while riding. In this work, we investigate four input methods: (1) direct touch on a centrally mounted tablet, (2) an analogue joystick, (3) tactile buttons, and (4) an indirect touch display on the handlebars. We evaluated them in terms of efficiency, accuracy, distraction, and user experience in an outdoor experiment (N = 24), in which participants had to select targets on three layouts (vertical, horizontal, and two-dimensional). The results show that although direct touch provided the fastest task completion time, it required more intentional eye movements and the loss of two-handed steering control. Haptic-based input methods, such as tactile buttons, resulted in slightly longer task completion times but lower error rates, a greater perception of safety, and better user acceptance. The eye-tracking data show a tendency for visual strain to be proportionally lower with haptic inputs, despite longer interaction times.
Runtime adaptation frameworks in mobile applications aim to calibrate model parameters to sustain high performance in evolving deployment scenarios, but label shifts alter label distributions across personalized models, rendering existing adaptation methods deficient. Knowledge sharing offers a promising solution by enabling personalized models to share relevant information, reducing redundant computations during independent fine-tuning. However, a major challenge lies in how to balance the communication cost with adaptation accuracy. This paper presents MACRA, which employs an adaptive, selective model merging mechanism based on arithmetic parameter combinations to facilitate knowledge sharing among mobile devices in response to label shifts. MACRA makes three contributions: First, it identifies and selects the most valuable parameters that reflect the shift, enabling high inference accuracy across different shifts. Second, it automatically identifies two types of label shifts and incorporates a merged model cache and check mechanism to ensure the high-quality updates across devices and minimize unnecessary computations. Third, it dynamically estimates the maximal feasible compression rate per shift, achieving optimal communication efficiency with minimal accuracy loss. Experiments on six human-centered datasets across four modalities show that MACRA compresses the transmitted parameters by up to 1/3000 of the original model, achieving 5x higher transmission compression rate than the best baseline, while improving the overall task accuracy by 13.5% and reducing the latency by 61% across different label shifts.
Modern cities increasingly rely on 3D urban maps as digital infrastructures to support applications such as navigation, autonomous driving, urban development, and cellular network planning. Maintaining these maps with high fidelity and timely updates, however, remains challenging due to the high cost and limited availability of aerial imagery or LiDAR data. In this work, we present SurfNeHF, a lightweight deep learning framework that accurately infers building shapes from noisy crowdsourced GNSS data. By adapting optical inverse rendering to the radio frequency domain, SurfNeHF represents building surfaces as neural height fields and explicitly models signal-surface interactions. This surface-aware formulation enables the framework to effectively disentangle building geometry from surface reflectance and satellite sources, significantly reducing optimization complexity compared to volumetric approaches. Extensive evaluations demonstrate that SurfNeHF consistently outperforms state-of-the-art methods across diverse urban morphologies. Our framework achieves an average reconstruction accuracy of 94.6% while reducing computational overhead by 80% to 90% compared to baselines. With both higher data and system efficiency, SurfNeHF makes periodic large-scale urban map updates more practically feasible.
Self-directed rehabilitation in everyday environments remains difficult due to limited feedback and the absence of contextually grounded guidance. In response, we present RehabGen, a generative vision-language model that generates personalized exercise plans contextualized to a patient's egocentric image and available everyday objects. We followed an expert-in-the-loop (EITL) framework: gathering domain requirements through expert interviews and focus group interviews ( N =8), constructing a 1.66k multimodal preference dataset, and aligning the model using supervised fine-tuning (SFT) and Direct Preference Optimization (DPO). In Expert Validation ( N =23), DPO achieved a 57.6% item-level win rate over the Baseline, with higher scores in Personalized Recommendation (85.7%), Identify Purpose (63.6%), and Recommendation Satisfaction (66.7%). In Patient Experience Evaluation ( N =8), both models outperformed the Baseline. This work contributes an EITL framework and open-source models available at http://rehabgen.hcitech.org/ for clinically grounded, context-aware rehabilitation planning.
Autonomous vehicles (AVs) need to communicate with other road users in uncertain scenarios via external human-machine interfaces (eHMIs). Existing eHMIs are rule-based, relying on manually designed motion and communication patterns for predefined scenarios, which limits adaptability. To address this, we explore LLM-driven eHMIs, a new approach that leverages the human-like reasoning and communication abilities of Large Language Models (LLMs) for generating expressive communicative actions. We investigated two research questions: 1) How does a pre-trained LLM translate intended communicative messages into corresponding eHMI actions? and 2) How communication-efficient are these generated actions to humans? To answer these questions, we first constructed a prompt framework through iterative prototyping, followed by the development of an end-to-end physical pipeline. Finally, we conducted a 26-participant field study using a physical prototype to assess communication effectiveness by comparing human-designed and GPT-o3-mini-generated eHMI actions in terms of participants' correct reactions and message interpretation. Results show that participants' correct reactions do not equate to their correct interpretations. The current pre-trained LLMs can produce communicative actions with similar interpretation comparable to human-designed ones, but cannot enable the correct pedestrian reactions. Our discussion highlights different design patterns between humans and LLMs.
Hand-object interactions are central to everyday activities, yet most intelligent assistants today remain blind to users' physical actions. Existing IMU-based recognition approaches focus on classifying predefined gestures, but they lack the semantic expressiveness required for contextual support in real-world scenarios such as office work and home routines. In this paper, we introduce a semantic tokenization pipeline that bridges continuous inertial signals and large language models (LLMs), enabling assistants to “read” hand movements as naturally as words. We first collected a multimodal dataset of dual-hand activities across office and home environments capturing long-horizon action chains that span multiple interrelated sub-tasks. Using self-supervised representation learning, we discretize IMU embeddings into action tokens that approximate a vocabulary of hand interactions. These tokens are then aligned with natural language through instruction-tuned LLMs, supporting tasks such as action captioning, intent inference, and contextual feedback. Evaluation shows that our tokenization improves semantic consistency with language distributions, and the LLM produces accurate, human-preferred descriptions of actions across diverse activities. We further demonstrate a proof-of-concept assistant prototype that generates contextual reminders. Our findings highlight the potential of transforming raw hand motions into a “language of actions,” paving the way for everyday intelligent assistants that are aware of users' physical interactions. The Project page, source code, and dataset are publicly available at https://scut-hai.github.io/HMotionGPT/.
Underwater operations require divers to access location-specific information, such as inspection protocols and safety guidelines. However, existing technologies face significant limitations: acoustic systems rely on expensive and bulky transceivers, RF communication is limited to centimeter-scale range, and laser systems require precise alignment. In this paper, we present UOTag, a passive, all-angle optical tag system leveraging commercial off-the-shelf (COTS) smartphones as receivers for underwater information retrieval. Traditional optical tags like QR codes provide high information capacity but perform poorly underwater due to narrow viewing angles and a limited range of just 3-6 meters. UOTag adopts the QR code concept for underwater information delivery and leverages a rhombic dodecahedron structure to enable 360° three-dimensional viewing angles, allowing tag access from any direction and eliminating alignment constraints. The system incorporates an underwater-optimized encoding-decoding framework and replaces black-and-white QR codes with a blue-yellow color scheme, effectively extending operational range underwater. Real-world experiments in a pool and at sea show working distances of 11.8 meters and 7.2 meters, respectively, using a compact tag with an edge length of just 15 cm. The system operates with a latency of just 113 ms and costs $8.50 per tag, enabling practical deployment as a convenient information source for divers.
To enhance the speech clarity in earable voice interaction scenarios, dual-microphone speech enhancement (SE) techniques with collaboration of in-ear and out-ear microphones have garnered significant attention from the research community. Nevertheless, existing dual-microphone SE techniques are established on a strong assumption: high-quality in-ear speech (auxiliary modality) could provide efficient complementary information to target airborne speech (primary modality) , which decreases the adaptation in the real world. In our work, we explore a key observation that air pressure imbalance caused by ear canal deformation (ECD) adversely affects the quality of in-ear speech, subsequently leading to a significant degradation in speech enhancement performance. To address this bottleneck issue, we design an efficient quality-aware speech enhancement solution, named QuaSE, which efficiently and dynamically fuses complementary information by assessing the quality variations of in-ear speech. Additionally, based on the analysis of spectral distortion induced by ECD, a training strategy including quality-aware data selection and content-aware augmentation is designed to improve the generalization capability of QuaSE. Extensive experiments demonstrate that QuaSE outperforms state-of-the-art techniques by 6.27%, 4.54%, 14.90%, and 11.93% in terms of PESQ, STOI, SI-SDR, and SegSNR. Moreover, we also validate that the proposed quality-aware fusion strategy can be modularly integrated into other sensing tasks, improving the fusion performance.
Ensuring drinking water safety is critical, yet detecting ppb-level heavy metals remains a challenge, as health-relevant regulatory thresholds (e.g., 10 ppb) are three to seven orders of magnitude below the sensitivity of currently available test strips and wireless sensing systems. This paper presents MetaTap, a low-cost, passive metasurface-based system for high-resolution water contamination detection. Inspired by resonance splitting, MetaTap employs a novel Complementary Asymmetric Split Square (CASS) structure that encodes multiple information-rich resonance dips within the bandwidth of a commercial UWB radar. This design significantly enhances sensing resolution, enabling the detection of contaminants at concentrations as low as 5 ppb. To ensure robustness in complex environments, we develop a physically interpretable model using resonance spectral features and their derivatives to suppress noise from multipath propagation and unknown background constituents. Extensive evaluations in real-world scenarios demonstrate that MetaTap maintains strong robustness against environmental variations and achieves an average detection accuracy of 94.6% for contaminants in real tap water. With a metasurface fabrication cost of only $0.10, MetaTap provides a highly accessible and sensitive platform for real-time water quality monitoring in everyday settings.
mmWave radar has emerged as a promising technology for human sensing. While large bandwidths and multiple antennas enable high-quality point clouds for sensing applications, low-cost 24 GHz radars are restricted to 250 MHz bandwidth by regulations, leading to limited range resolution. In addition, the limited number of antennas, constrained by cost and device size, results in low angular resolution. Together, these limitations produce sparse point clouds that impede fine-grained human sensing. In this work, we present RaPoint, the first system to produce high-density point clouds from 24 GHz radars under regulatory bandwidth limits. RaPoint introduces a joint super-resolution modeling framework that leverages the MUSIC algorithm to jointly exploit super-resolution range and angle dimensions, together with the Doppler dimension, for distinguishing reflection points. To overcome the high computational complexity problem of the MUSIC algorithm, we convert the eigenvalue decomposition from the noise subspace to the sparse signal subspace, drastically reducing computation. We implement RaPoint on three representative mmWave radars. Extensive experiments demonstrate that RaPoint improves point cloud density by 9.45× and 24.10× over the state-of-the-art research and industrial baselines, respectively, and substantially boosts downstream sensing performance. We believe RaPoint takes a significant step toward fine-grained human sensing with low-cost, bandwidth-limited 24 GHz radars, enabling affordable and ubiquitous radar sensing.
Human memory is inherently prone to fragmentation, especially when dealing with multimodal digital traces. While existing AI systems struggle to infer personal significance in meaning-making, manual organization often imposes substantial cognitive burden on users. Therefore, our study investigates: how human-AI collaborative systems can support meaning-making of personal memories and how such meaning-making enhances performance on downstream memory tasks. Through co-design workshops, we identified critical challenges in current meaning-making practices and derived actionable design guidelines. We then developed MemoryLib, a mixed-initiative interactive system that enables dynamic organization and interpretation of multimodal memories augmented through human-AI collaborative practices. Findings from a controlled user experiment demonstrated that MemoryLib significantly improved users' abilities to recall, establish relationships of, and reuse memory content. The study contributes a novel human-AI collaboration approach for everyday memory meaning-making and empirical evidence supporting the system's effectiveness in enhancing memory-related tasks.
Wireless sensing has gained significant research interest in recent years. However, the focus has primarily been on improving sensing performance, such as enhancing accuracy, and little attention has been paid to the security aspects of wireless sensing. In this paper, we demonstrate that acoustic signal-based sensing, widely regarded as the safest wireless sensing modality due to its physical characteristics, can be stealthily attacked and compromised. The core of this stealthy injection attack lies in exploiting the “photoacoustic effect” to convert the energy of laser light into vibrations, inducing acoustic signals, which are then used to compromise acoustic sensing systems. It is important to note that the laser can be invisible to the human eye, and the acoustic signals it generates are inaudible to human ears. To make the attack even stealthier, we propose the concept of “Smartphone-Defined Laser” to use commodity smartphones to control low-cost laser hardware ($0.60), eliminating the need for bulky and expensive signal generators. Through hardware and software co-design, we successfully demonstrate the attack with smartphones and cheap laser hardware. Comprehensive experiments show that the proposed attack can compromise the state-of-the-art acoustic sensing techniques (both chirp-based and continuous wave-based), achieving a high average success rate of 96.1% across different tasks even when the target is 50 m away. We hope our findings raise awareness of the security risks associated with acoustic sensing and encourage further research into enhancing its security.
