This autoethnographic study by a Deaf researcher with a multicultural background examined the use of automatic speech recognition and text-to-speech (TTS) on mobile/computerdevices and augmented reality (AR) smart glasses to support communication for Deaf and Hard of Hearing individuals in educational and everyday contexts. Mobile and computer-based tools enable active participation through larger displays but require sustained screen attention, reducing access to facial cues. AR smart glasses support hands-free transcription and visual attention but lack TTS and bidirectional interaction. The findings suggest that improving comfort, reliability, and adaptive input methods is essential, and that combining AR smart glasses with mobile and computer tools offers complementary support for accessibility and social inclusion rather than replacing human communication.
Gesture-based input is a key natural modality for interaction in ubiquitous computing environments. One distinctive use is seated interaction, which introduces physical and mobility constraints that influence hand gesture articulation and reachable interaction space, particularly for wheelchair users with specific motor and mobility abilities. In this article, we examine gesture interaction for seated users, and report empirical insights into their gesture articulation characteristics and preferences through a comparative analysis of 306 hand gestures elicited from 29 participants, of whom eleven were wheelchair users. Our findings reveal a predominant preference for mid-air gestures among users with mobility impairments compared to on-wheelchair input (81.9% versus 18.1%) and, in contrast, a balanced distribution of mid-air and on-chair gestures (51.7% and 48.3%) for users without mobility impairments when seated in conventional chairs. Based on these findings, we compile two representative gesture sets for effecting 13 common system functions, such as content navigation and remote device control, and we derive design implications for inclusive ubiquitous computing environments that incorporate hand-chair gestures.
Although the expansion and generalization of human presence through ubiquitous sensing (a.k.a. "Cross Reality") have had a close association with virtual and augmented reality systems, we have arrived at a crossroads as it transitions into a broader and even more exciting future through multimodality, where this information arrives at our perception via a dynamically optimal and personalized m & eacute;lange of sensation.
This article examines when a mobile system should view a resource as portable (and hence carried by its user), and when it can be viewed as pervasive (and hence easily found whenever and wherever needed). It shows that the anticipated worst case scenario while mobile is the critical determinant. If the resource is easily and reliably available at all visited locations, then it can be treated as a pervasive resource. If there is reasonable doubt about its availability or attributes at even one visited location, then it has to be treated as portable. Using this resource-centric viewpoint, this article highlights the unique role of the cloud in mobile computing, as well as its limitations. It discusses how some legacy systems can be rearchitected using modern technology, and how emerging real-time mobile AI systems will have to straddle the “carry” versus“find” divide.
Xin Liu: My research focuses on three interconnected directions aimed at making personalized health intelligence accessible to everyone. First, I build personal health agents that reason over wearable sensor data. Our work published in Nature Medicine showed that large language models (LLMs) grounded in wearable data can generate expert-quality sleep and fitness coaching, and our agentic system demonstrated how LLM agents can use tools and multistep reasoning to transform more complex health queries into actionable health insights. Most recently, we designed a multiagent personal health system that orchestrates data analysis, domain expert reasoning, and behavioral coaching into a unified framework. Second, I develop wearable foundation models trained on large-scale sensor data to perform diverse health tasks with a single model. Through projects like large sensor model (LSM) and sensor-language model (SensorLM), we created models that learn general representations from wearable time-series, enabling few-shot or zero-shot health understanding without task-specific training. Third, I contribute to advancing LLM reasoning capabilities as a core contributor to Google’s Gemini 2.5, focusing on tabular and data science reasoning, work that directly strengthens the analytical backbone of health agents.
Recent advancements in brain-computer interfaces (BCIs), e.g., Neuralink, have enabled seamless interaction and control across various applications, from assistive technologies to virtual environments. Traditional BCI applications treat users as control units that issue commands to interact with systems. In this work, we propose the novel concept of Human-as-a-Sensor (HaaS), where users function as seamless intelligent multimodal sensors and BCIs extract contextual sensing data from brain signals as they engage naturally with their environment. By tapping directly into neural activity and relaying information through neuronal feedback, HaaS reduces reliance on handheld sensors/screens, with fusion-first perception and feedback. We explore various HaaS-enabled opportunities for sensing applications and demonstrate how HaaS fosters self-determined technology engagement by supporting opt-in local monitoring and management. Moreover, we discuss critical challenges in realizing HaaS, including accuracy, real-world deployment, and privacy issues. Finally, we present a proof-of-concept evaluation that demonstrates the promise of HaaS for enabling natural interaction with sensing systems.
This article presents a case study of everyday work in a highly automated, energy-efficient office building. Through interviews and site visits with 12 participants, including office users, technical support staff, and institutional decision-makers, we investigate the reality of ubiquitous smart building systems in professional settings. Our findings highlight the tension between system automation and user agency, particularly when technical and organizational constraints limit the user’s ability to shape the system’s operation. We describe users’ lived experiences stemming from over two years of everyday interaction with smart systems, revealing key challenges in the long-term use of ubiquitous building technologies. These include issues of algorithmic transparency, perceived reliability, and user strategies to force desired system responses. Based on our analysis, we indicate promising strategies for facilitating the discoverability and adoption of smart building systems in the workplace. By learning from early-stage deployments, our findings inform the design of more usable smart work environments.
As our population ages, care homes play an essential role in providing long-term support for older adults, prompting many to explore technology-based interventions to help meet these growing care needs. While much of the technology in this sector has focused on health monitoring and operational efficiency, there is increasing interest in technologies that support residents’ emotional, cognitive, and social well-being. The ubiquity of pervasive displays makes them promising candidates for a wide range of applications in care environments. However, despite widespread use in other sectors, digital signage systems have seen relatively little exploration in residential care settings. Drawing on insights gained from a long-term collaboration with a local care home and two decades of experience in managing the world’s largest research digital signage deployment, this article identifies key design considerations for signage systems in residential care settings.
Time is the one currency we can never mint again. No laboratory can synthesize it, no algorithm can extend it, and no machine can reverse its flow. We live in a world where every second matters—yet we squander vast amounts of it in hesitation, in searching, in translating complexity into choices. The most profound challenge of our digital age is not the abundance of information but the scarcity of decision bandwidth.
Anandghan Waghmare: My research focuses on extending the sensing capabilities of ubiquitous devices, such as smartphones and smartwatches, beyond their original design. This strategy involves exploring the untapped potential within these increasingly powerful devices to enable functionalities, such as blood glucose measurement, using smartphones or UV light intensity detection through smartwatches. I investigate the integration of advanced signal processing, embedded systems, and machine learning to unlock the hidden sensing capabilities of modern devices. My work primarily pursues two complementary approaches to enhance device functionality while maintaining practicality. The first approach involves developing cost-effective, low-power hardware add-ons that seamlessly integrate with existing devices. These add-ons are designed for affordability, broad compatibility, and minimal energy consumption, ensuring widespread accessibility and ease of use. By leveraging the processing power of the smart device, these add-ons remain simple yet effective, expanding sensing capabilities without necessitating the acquisition of new devices. The second approach focuses on hardware modifications that enhance sensor performance without requiring a complete device redesign. Modifying existing components offers advantages, such as reduced costs and shortened development cycles, and preserves user familiarity. This strategy allows for the unlocking of new functionalities while maintaining the device’s original form, making advanced sensing more commercially viable. Through these strategies, my research aims to develop practical and scalable solutions that augment sensing capabilities, ultimately making smart devices more intelligent and versatile.
Let us call it refreshing to look behind one’s comfort zone and consider another subfield within computer science. The IEEE/ACM International Conference on Software Engineering is an influential conference. In this column, I report on recent software engineering work related to pervasive computing.
Excessive, absent-minded, and aimless use of a smartphone can have a negative impact on both physical and mental health, with the potential to develop into an addiction. Because compulsive behavior can be triggered by the mere presence of a smartphone, conventional screen time management apps often fail to address this issue. We conducted expert interviews with therapists that revealed the potential of a few hours of abstinence with smartphones. Consequently, our research aims to offer a tangible physical opt-out solution while maintaining users’ autonomy to effectively reduce screen time. We designed a “mindful box” with a locking mechanism to securely store away digital devices. The box includes features that encourage intentional use, including gentle wake-up, meditation support, and cheer lists. In a one-week study with seven participants, the mindful box significantly reduced average screen time (p = 0.016). Users praised its effectiveness in combating distractions and limiting excessive device use.
Augmented reality (AR) technology is transforming minimally invasive surgeries by merging physical and virtual spaces to improve intraoperative guidance and training. We present SurgiKLAR, a mixed reality framework for improved surgical training, to clearly (“klar”) visualize segmented anatomies from preoperative scans coregistered with patient-specific models to simulate surgical procedures. The system is adapted for gynecology surgeries, featuring realistic uterine models and instruments for adaptive simulations, personalized guidance, and real-time alerts. Preliminary evaluations with a 3D-printed phantom demonstrated its potential application in preplanning complex scenarios. We conducted a two-phase user study to evaluate the usability of the SurgiKLAR system. Phase I assessed AR adoption challenges, and Phase II validated usability for training. The user feedback highlighted the importance of accurate visualizations and the need for extensive training programs. Future work involves an extension to diverse surgical domains, improved precision, and enhanced safety, thereby highlighting the transformative potential of cross-reality in surgery.
