Egocentric vision systems aim to understand the spatial surroundings and the wearer's behavior inside it, including motions, activities, and interactions. We argue that egocentric systems must additionally detect physiological states to capture a person's attention and situational responses, which are critical for context-aware behavior modeling. In this paper, we propose egoPPG, a novel vision task for egocentric systems to recover a person's cardiac activity to aid downstream vision tasks. We introduce PulseFormer, a method to extract heart rate as a key indicator of physiological state from the eye tracking cameras on unmodified egocentric vision systems. PulseFormer continuously estimates the photoplethysmogram (PPG) from areas around the eyes and fuses motion cues from the headset's inertial measurement unit to track HR values. We demonstrate egoPPG's downstream benefit for a key task on EgoExo4D, an existing egocentric dataset for which we find PulseFormer's estimates of HR to improve proficiency estimation by 14

Today's fitness bands and smartwatches typically track heart rates (HR) using optical sensors. Large behavioral studies such as the U.K. Biobank use activity trackers without such optical sensors and thus lack HR data, which could reveal valuable health trends for the wider population. In this paper, we present the first dataset of wrist-worn accelerometer recordings and electrocardiogram references in uncontrolled at-home settings to investigate the recent promise of IMU-only HR estimation via ballistocardiograms. Our recordings are from 42 patients during the night, totaling 310 hours. We also introduce a frequency-based method to extract HR via curve tracing from IMU recordings while rejecting motion artifacts. Using our dataset, we analyze existing baselines and show that our method achieves a mean absolute error of 0.88 bpm-76% better than previous approaches and the first to surpass established medical standards for heart rate monitors. Our results validate the potential of IMU-only HR estimation as a key indicator of cardiac activity in existing longitudinal studies to discover novel health insights.

Fatigue is the most common symptom among Multiple Sclerosis (MS) patients and severely affects the quality of life. We investigate how perceived fatigue can be predicted using biomarkers collected from an arm-worn wearable sensor for MS patients (n=51) and a healthy control group (n=23) at an unprecedented time resolution of more than five times per day. On average, during our two-week study, participants reported their level of fatigue 51 times totaling more than 3700 data points. Using interpretable Generalized Additive Models, we find that increased physical activity, heart rate, sympathetic activity, and parasympathetic activity while awake and asleep relate to perceived fatigue throughout the day—partly affected by dysfunction of the ANS. We believe our analysis opens up new research opportunities for fine-grained modeling of perceived fatigue based on passively collected physiological signals using wearables—for MS patients and healthy controls alike.

Background Low sleep quality is a common symptom of multiple sclerosis (MS) and substantially decreases patients’ quality of life. The autonomic nervous system (ANS) is crucial to healthy sleep, and the transition from wake to sleep produces the largest shift in autonomic activity we experience every day. For patients with MS, the ANS is often impaired. The relationship between the ANS and perceived sleep quality in patients with MS remains elusive. Objective In this study, we aim to quantify the impact of the ANS and MS on perceived sleep quality. Methods We monitored 77 participants over 2 weeks using an arm-worn wearable sensor and a custom smartphone app. Besides recording daily perceived sleep quality, we continuously recorded participants’ heart rate (HR) and HR variability on a per-second basis, as well as stress, activity, and the weather (20,700 hours of sensor data). Results During sleep, we found that reduced HR variability and increased motion led to lower perceived sleep quality in patients with MS (n=53) as well as the age- and gender-matched control group (n=24). An activated stress response (high sympathetic activity and low parasympathetic activity) while asleep resulted in lower perceived sleep quality. For patients with MS, an activated stress response while asleep reduced perceived sleep quality more heavily than in the control group. Similarly, the effect of increased stress levels throughout the day is particularly severe for patients with MS. For patients with MS, we found that stress correlated negatively with minimal observed HR while asleep and might even affect their daily routine. We found that patients with MS with more severe impairments generally recorded lower perceived sleep quality than patients with MS with less severe disease progression. Conclusions For patients with MS, stress throughout the day and an activated stress response while asleep play a crucial role in determining sleep quality, whereas this is less important for healthy individuals. Besides ensuring an adequate sleep duration, patients with MS might thus work to reduce stressors, which seem to have a particularly negative effect on sleep quality. Generally, however, sleep quality decreases with MS disease progression.

Recent work has shown that a person's sympathetic arousal can be estimated from facial videos alone using basic signal processing. This opens up new possibilities in the field of telehealth and stress management, providing a non-invasive method to measure stress only using a regular RGB camera. In this paper, we present SympCam, a new 3D convolutional architecture tailored to the task of remote sympathetic arousal prediction. Our model incorporates a temporal attention module (TAM) to enhance the temporal coherence of our sequential data processing capabilities. The predictions from our method improve accuracy metrics of sympathetic arousal in prior work by 48% to a mean correlation of 0.77. We additionally compare our method with common remote photoplethysmography (rPPG) networks and show that they alone cannot accurately predict sympathetic arousal "out-of-the-box". Furthermore, we show that the sympathetic arousal predicted by our method allows detecting physical stress with a balanced accuracy of 90% - an improvement of 61% compared to the rPPG method commonly used in related work, demonstrating the limitations of using rPPG alone. Finally, we contribute a dataset designed explicitly for the task of remote sympathetic arousal prediction. Our dataset contains synchronized face and hand videos of 20 participants from two cameras synchronized with electrodermal activity (EDA) and photoplethysmography (PPG) measurements. We will make this dataset available to the community and use it to evaluate the methods in this paper. To the best of our knowledge, this is the first dataset available to other researchers designed for remote sympathetic arousal prediction.

Multiple sclerosis (MS) is a neurological disease of the central nervous system that is the leading cause of non-traumatic disability in young adults. Clinical laboratory tests and neuroimaging studies are the standard methods to diagnose and monitor MS. However, due to infrequent clinic visits, it is fundamental to identify remote and frequent approaches for monitoring MS, which enable timely diagnosis, early access to treatment, and slowing down disease progression. In this work, we investigate the most reliable, clinically useful, and available features derived from mobile and wearable devices as well as their ability to distinguish people with MS (PwMS) from healthy controls, recognize MS disability and fatigue levels. To this end, we formalize clinical knowledge and derive behavioral markers to characterize MS. We evaluate our approach on a dataset we collected from 55 PwMS and 24 healthy controls for a total of 489 days conducted in free-living conditions. The dataset contains wearable sensor data – e.g., heart rate – collected using an arm-worn device, smartphone data – e.g., phone locks – collected through a mobile application, patient health records – e.g., MS type – obtained from the hospital, and self-reports – e.g., fatigue level – collected using validated questionnaires administered via the mobile application. Our results demonstrate the feasibility of using features derived from mobile and wearable sensors to monitor MS. Our findings open up opportunities for continuous monitoring of MS in free-living conditions and can be used to evaluate and guide the effectiveness of treatments, manage the disease, and identify participants for clinical trials.

Understanding a person’s perceived quality of sleep is an important problem, but hard due to its poor definition and high intra- as well as inter-individual variation. In the short term, sleep quality has an established impact on cognitive function during the following day as well as on fatigue. In the long term, good quality sleep is essential for mental and physical health and contributes to quality of life. Despite the need to better understand sleep quality as an early indicator for sleep disorders, perceived sleep quality has been rarely modeled for multiple consecutive days using biosignals. In this paper, we present novel insights on the association of cardiac activity and perceived sleep quality using an interpretable modeling approach utilizing the publicly available intensive-longitudinal study M2Sleep. Our method takes as input signals from commodity wearable devices, including motion and blood volume pulses. Despite processing only simple and clearly interpretable features, we achieve an accuracy of up to 70% with an AUC of 0.76 and reduce the error by up to 36% compared to related work. We further argue that collected biosignals and sleep quality labels should be normalized per-participant to enable a medically insightful analysis. Coupled with explainable models, this allows for the interpretations of effects on perceived sleep quality. Analysis revealed that besides higher skin temperature and sufficient sleep duration, especially higher average heart rate while awake and lower minimal activity of the parasympathetic and sympathetic nervous system while asleep increased the chances of higher sleep quality.

While highly important for a person’s mood, productivity, and physical performance, perceived sleep quality is challenging to model and, thus, predict with passive means such as physiological and behavioral signals alone. In this paper, we propose a novel method that diverges from traditional feature-based modeling of sleep quality. Instead, our method is unsupervised and derives states of cardiac activity from polysomnography (PSG) recordings of more than 6,800 participants. We then demonstrate that the proportion of time spent in these states strongly correlates with perceived sleep quality using a longitudinal study of 16 participants over one month. Our method classifies participants’ perceived sleep quality with a balanced accuracy of 68%, significantly exceeding prior methods and feature-based approaches that incorporate established metrics of cardiac activity. Interestingly, we find that the states of cardiac activity our method derives oppose traditional sleep stages—even though the states seem easily explainable based on simple metrics of cardiac activity. Thus, we provide evidence that there are still little-understood processes during sleep that need further investigation, potentially even a rethinking of sleep analysis, especially for perceived sleep quality.