While audio data shows promise in addressing various health challenges, there is a lack of research on on-device audio processing for smartwatches. Privacy concerns make storing raw audio and performing post-hoc analysis undesirable for many users. Additionally, current on-device audio processing systems for smartwatches are limited in their feature extraction capabilities, restricting their potential for understanding user behavior and health. We developed a real-time system for on-device audio processing on smartwatches, which takes an average of 1.78 minutes (SD = 0.07 min) to extract 22 spectral and rhythmic features from a 1-minute audio sample, using a small window size of 25 milliseconds. Using these extracted audio features on a public dataset, we developed and incorporated models into a watch to classify foreground and background speech in real-time. Our Random Forest-based model classifies speech with a balanced accuracy of 80.3%.
Image super-resolution (SR) is widely used on mobile devices to enhance user experience. However, neural networks used for SR are computationally expensive, posing challenges for mobile devices with limited computing power. A viable solution is to use heterogeneous processors on mobile devices, especially the specialized hardware AI accelerators, for SR computations, but the reduced arithmetic precision on AI accelerators can lead to degraded perceptual quality in upscaled images. To address this limitation, in this paper we present SR For Your Eyes (FYE-SR), a novel image SR technique that enhances the perceptual quality of upscaled images when using heterogeneous processors for SR computations. FYE-SR strategically splits the SR model and dispatches different layers to heterogeneous processors, to meet the time constraint of SR computations while minimizing the impact of AI accelerators on image quality. Experiment results show that FYE-SR outperforms the best baselines, improving perceptual image quality by up to 2×, or reducing SR computing latency by up to 5.6× with on-par image quality.
Maintaining independence in daily activities and mobility is critical for healthy aging. Older adults who are losing the ability to care for themselves or ambulate are at a high risk of adverse health outcomes and decreased quality of life. It is essential to monitor daily activities and mobility routinely and capture early decline before a clinical symptom arises. Existing solutions use self-reports, or technology-based solutions that depend on cameras or wearables to track daily activities; however, these solutions have different issues (e.g., bias, privacy, burden to carry/recharge them) and do not fit well for seniors. In this study, we discuss a non-invasive, and low-cost wireless sensing-based solution to track the daily activities of low-income older adults. The proposed sensing solution relies on a deep learning-based fine-grained analysis of ambient WiFi signals and it is non-invasive compared to video or wearable-based existing solutions. We deployed this system in real senior housing settings for a week and evaluated its performance. Our initial results show that we can detect a variety of daily activities of the participants with this low-cost system with an accuracy of up to 76.90%.
Acoustic sensing on smartphones has gained extensive attention from both industry and research communities. Prior studies suffer from one fundamental limit, i.e., audio sampling rates on smartphones are constrained at 48 kHz. In this work, we present PowerPhone, a software reconfiguration to support higher sampling rates on both microphones and speakers of smartphones. We reverse-engineered more than 100 smartphones and found that their sampling rates can be reconfigured to 192 kHz. We conducted benchmark experiments and showcased field studies to demonstrate the unleashed sensing capability using our reconfigured smart-phones. First, we improve the sensing resolution from 7 cm to 1cm and enable multi-finger gesture recognition on smart-phones. Second, we push the sensing granularity of subtle movements to 2 μm and show the feasibility of turning the smartphone into a micrometer-level machine vibration meter. Third, we increase the sensing range to 6 m and showcase room-scale human presence detection using a smartphone. Finally, we demonstrate that PowerPhone can enable new applications that were previously infeasible. Specifically, we can detect the home appliance status by analyzing ultrasonic leakages above 24 kHz from the wireless charger while charging a smartphone. Our open-source artifacts can be found at: https://powerphone.github.io.
Acoustic sensing shows great potential to transform billions of consumer-grade electronic devices that people interact with on a daily basis into ubiquitous sensing platforms. In this paper, we share our experience and findings during the process of developing and deploying acoustic sensing systems for real-world usage. We identify multiple practical problems that were not paid attention to in the research community, and propose the corresponding solutions. The challenges include: (i) there exists annoying audible sound leakage caused by acoustic sensing; (ii) acoustic sensing actually affects music play and voice call; (iii) acoustic sensing consumes a significant amount of power, degrading the battery life; (iv) real-world device mobility can fail acoustic sensing. We hope the shared experience can benefit not only the future development of sensing algorithms but also the hardware design, pushing acoustic sensing one step further towards real-life adoption.
Wrist-worn devices hold great potential as a platform for mobile health (mHealth) applications because they comprise a familiar, convenient form factor and can embed sensors in proximity to the human body. Despite this potential, however, they are severely limited in battery life, storage, bandwidth, computing power, and screen size. In this paper, we describe the experience of the research and development team designing, implementing and evaluating Amulet? an open-hardware, open-software wrist-worn computing device? and its experience using Amulet to deploy mHealth apps in the field. In the past five years the team conducted 11 studies in the lab and in the field, involving 204 participants and collecting over 77,780 hours of sensor data. We describe the technical issues the team encountered and the lessons they learned, and conclude with a set of recommendations. We anticipate the experience described herein will be useful for the development of other research-oriented computing platforms. It should also be useful for researchers interested in developing and deploying mHealth applications, whether with the Amulet system or with other wearable platforms.
The human eye offers a fascinating window into an individual's health, cognitive attention, and decision making, but we lack the ability to continually measure these parameters in the natural environment. The challenges lie in: a) handling the complexity of continuous high-rate sensing from a camera and processing the image stream to estimate eye parameters, and b) dealing with the wide variability in illumination conditions in the natural environment. This paper explores the power--robustness tradeoffs inherent in the design of a wearable eye tracker, and proposes a novel staged architecture that enables graceful adaptation across the spectrum of real-world illumination. We propose CIDER, a system that operates in a highly optimized low-power mode under indoor settings by using a fast Search-Refine controller to track the eye, but detects when the environment switches to more challenging outdoor sunlight and switches models to operate robustly under this condition. Our design is holistic and tackles a) power consumption in digitizing pixels, estimating pupillary parameters, and illuminating the eye via near-infrared, b) error in estimating pupil center and pupil dilation, and c) model training procedures that involve zero effort from a user. We demonstrate that CIDER can estimate pupil center with error less than two pixels (0.6 O ), and pupil diameter with error of one pixel (0.22mm). Our end-to-end results show that we can operate at power levels of roughly 7mW at a 4Hz eye tracking rate, or roughly 32mW at rates upwards of 250Hz.
