Activities of daily living (ADL) identification with wearables has significant implications in healthy lifestyle management and offers an important sensor-based supervised learning research benchmark. Most ADL studies use single-modality (i.e., single sensor type like smartwatch only or earbuds only) data, while multisensor data fusion studies using early stage fusion of modalities from multisensors and multidevices are emerging. To improve classification performance and model interpretability, leveraging early stage fusion, late-stage fusion, and individual modalities, we introduced novel modality-aware dynamic fusion (MAD-Fusion) models for multisensor data-fusion-based ADL identification. Based on early stage fusion, we incorporated conformal prediction (CP) for uncertainty quantification, uncertainty late-stage fusion for cross-modality interpretability, and a multimodal strength-aware classification module. Trained on 36 independent subjects and tested on four independent subjects from the Samsung ADL dataset with multisensor (accelerometers and gyroscopes) and multidevice (earbuds and smartwatch), MAD-Fusion not only achieved the state-of-the-art classification performance (accuracy: 0.9504, F1-score: 0.9142), but also enabled better interpretability of contributions and uncertainties from different modalities. The additional contribution of each building block is validated systematically. Furthermore, we validated MAD-Fusion's superior performance on two public datasets in multisensor single-device settings (University of California, Irvine Human Activity Recognition (UCI-HAR) and University of Southern California Human Activity Dataset (USC-HAD) datasets). On all three datasets, MAD-Fusion manifested statistically significant superiority comparing against baselines of single-modality, early stage fusion, and late-stage fusion (p < 0.001). To conclude, the novel MAD-Fusion models improve the classification performance, uncertainty quantification, and interpretability for the ADL identification and can be applied to broader supervised learning areas requiring high performance, rigorous uncertainty quantification, and model interpretability.

Earbuds are instrumental in health monitoring but the orientation can variate among users, which may significantly impact the health-monitoring system generalizability. To study the effect of earbuds orientation heterogeneity and align kinematics across earbuds orientations, we collected a dataset with various rotations relative to a baseline orientation. We developed the coordinate transformation by estimating Euler angles in transformation matrices with either grid search or Markov Chain Monte Carlo (MCMC) sampling. Taking similar to 17 seconds with a personal laptop, the MCMC method accurately estimated the coordinate transformation matrices to enable the transformed tri-axial linear acceleration to better match the baseline tri-axial linear acceleration with an average relative error of 1.899% (0.186 m/s(2)) and a maximum relative error of 2.774% averaged over all test orientations. Using the estimated transformation matrices and Samsung dataset of identification of activities of daily living (ADL), we validated the statistically significant impact of earbuds orientation heterogeneity on ADL identification (p < 0.001), which can cause 14.0% reduction in mean accuracy and 18.7% reduction in mean macro-average F1-score. To sum up, the MCMC method developed can be applied in earbuds kinematics alignment to address orientation heterogeneity and enable better earbuds-based health monitoring.

Cough serves as a crucial bio-marker for evaluation and monitoring of pulmonary conditions. With growing interest towards automatic cough detection systems, it's important to acknowledge the existing hurdles on the way for a robust cough counter. These include high false positive rate caused by cough-like sounds in the environment, reduced sensitivity due to background noise interference. In this work, our objective is to tackle these obstacles through a comprehensive exploration of diverse strategies, including signal processing enhancements, innovative data augmentation techniques, and refined modeling approaches with emphasis on specificity to make the model robust in field environment. Our best model achieves sensitivity of 87.29% and specificity of 98.38%, while having a small footprint of 1.6 MB.

Human activity recognition has been an established active research area within the past few decades. While many researchers have tried to estimate some of the gait and running parameters, none was successful to provide a full suite of running dynamics parameters using commodity devices. Earbuds with their unique placement (in line with center of mass) provide an opportunity for activity recognition that never existed before with other commodity devices. Taking advantage of this opportunity, this work proposes a multi-modal approach to measure running dynamics using fusion of earbuds and smartwatch. Collecting a large dataset of 53 subjects, we developed various regression models to identify running parameters such as speed, cadence, stride length, vertical oscillation and ground contact time. These parameters were estimated in both jog and walk conditions and were evaluated in different device and context settings. Our MAPE ranges from 6.04% to 11.54% for various parameters.

Background: Amid the COVID-19 pandemic, the surge in hospital admissions and widespread use of broad-spectrum antibiotics have heightened the risk of hospital-acquired infections from multidrug-resistant (MDR) organisms, particularly Escherichia coli. It is imperative to implement stringent measures to curb the spread of antimicrobial resistance in hospitals and devise robust treatment strategies for patients grappling with such infections. To confront this challenge, a comprehensive study was undertaken to examine MDR extended-spectrum beta-lactamase (MDR-ESBL)-producing Escherichia coli isolates from patients with nosocomial infections following the COVID-19 pandemic in Northern Iran. Materials and Methods: The current study was conducted as a cross-sectional study. A total of 12,834 samples were collected from patients with healthcare-associated infections at four designated corona centers in Northern Iran, following the COVID-19 pandemic. Antimicrobial resistance was determined using standard broth micro-dilution, while resistance genes were accurately detected using the multiplex PCR method. Results: The results indicated that meropenem and ciprofloxacin had a resistance rate of 100% and 98.2%, respectively, while piperacillin-tazobactam showed the highest sensitivity rate at 54.4%. The frequency of specific genes, including blaIMP, blaTEM, AcrA, AcrB, blaCTX, blaOXA-58, aaclb, blaSHV, and aacla, were found to be 100%, 100%, 99.1%, 99.1%, 91.2%, 80.7%, 64.9%, 44.7%, and 37.7%, respectively. Conclusions: In the current study, over 50% of MDR-ESBL-producing Escherichia coli isolates exhibited resistance to antibiotics. A combination of antibiotics, including piperacillin-tazobactam and colistin, is recommended for treating extensively drug-resistant Escherichia coli infections.

Mouth breathing has been linked to a variety of negative health outcomes, including sleep-related disorders and dental problems. Detecting mouth breathing in the daily environment could be helpful for early intervention and reversing the negative impact. However, existing research has not adequately explored methods for detecting mouth breathing in everyday settings. This study presents a machine-learning approach using audio captured by commercially available earbuds to detect mouth breathing. By leveraging the growing popularity of earbuds for health monitoring, this approach offers a more convenient and non-invasive means of detecting mouth breathing. We conducted a data collection study with 30 participants to train a convolutional neural network-based model, which achieved an accuracy of 78.4% in detecting mouth breathing. Our findings suggest that audio-based mouth breathing detection using earbuds could be a promising tool for early intervention and improved health outcomes.

Activities of daily living is an important entity to monitor for promoting healthy lifestyle for chronic disease patients, children and the healthy population. This paper presents a smartwatch and earbuds inertial sensors based multi-modal power efficient end-to-end mobile system for continuous, passive and accurate detection of broad daily activity classes. We collected various posture, stationary and moving activity data from 40 diverse subjects using earbuds and smartwatch and develop the novel power optimized end-to-end operational system consisting of i) optimized device sampling rates and Bluetooth packet transfer rates, ii) data buffering mechanism, iii) background services, and iv) optimized model size, and demonstrating 93% macro recall score in detecting various activities. Our power optimized solution uses 80%, 40% and 33.33% less battery power for the smartphone, smartwatch, and earbuds respectively, compared to a power agnostic system with an estimated continuous no-charging run time of 50 hours, 16.67 hours, and 25 hours for the smartphone, smartwatch, and earbuds respectively.Clinical relevance— The end-to-end power optimized activity detection system presented in this paper will assist practicing clinicians toward treatment of various chronic disease patients (e.g. diabetes, hypertension, heart disease and obesity) by long-term, continuous monitoring of their lifestyle and sedentary behavior.

Human Activity Recognition (HAR) is one important digital health applications to track fitness or to avoid sedentary behavior. Due to the growing popularity of consumer wearable devices, smartwatches and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. Additionally, handling motion noise becomes more challenging when a single device is used. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 53 subjects collected from both controlled and uncontrolled noisy environments, we demonstrate the limitations of using a single modality activity classification. We identify various noise sources imposed in uncontrolled environment and propose two novel noise handling methods to ensure the robustness of activity state tracking. We build on top of a previous activity tracking effort and demonstrate a 7.8% sensitivity improvement against current state of the art in uncontrolled noisy environment.

Breathing rate is critical for the user’s respiratory health and is hard to track outside the clinical context, requiring specialized devices. Earables could provide a convenient solution to track the breathing rate anywhere by leveraging the user’s breathing-related motion and sound captured through the earables’ motion sensors and microphones. However, small non-breathing head movements or background noises during the assessment affect the estimation accuracy. While noise filtering improves accuracy, it can discard valid measurements. This paper presents a multimodal approach to tracking the user’s breathing rate using a signal-processing-based algorithm on motion sensors and a lightweight machine-learning algorithm on acoustic sensors from the earables that balances the accuracy and data retention. A user study with 30 participants shows that the system can accurately calculate breathing rate (Mean Absolute Error < 2 breaths per minute) while retaining most breathing sessions (75%) performed in real-world settings. This work provides an essential direction for remote breathing rate monitoring.

Human Activity Recognition (HAR) is one of the important applications of digital health that helps to track fitness or to avoid sedentary behavior by monitoring daily activities. Due to the growing popularity of consumer wearable devices, smartwatches, and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 44 subjects collected from both in-lab and in-home environments, we demonstrate the limitations of using a single modality as well as the importance of a multi-modal approach. Moreover, we also train and evaluate the performance of five different machine learning classifiers for various combinations of devices such as buds only, watch only, and both. We believe the detailed analyses presented in this paper may serve as a benchmark for the research community to explore and build upon in the future.

Breathing exercises reduce stress and improve overall mental well-being. There are various types of breathing exercises. Performing the exercises correctly may give the best outcome and doing it in wrong ways can sometimes have adverse effect. Providing real-time biofeedback can greatly improve the user experience in doing the right exercises in the right ways. In this paper, we present methods to passively track breathing biomarkers in real-time using wireless commodity earbuds and generate feedback on users' breathing performance. We use the earbud's low-power accelerometer to generate a comprehensive set of breathing biomarkers including breathing phase, breathing rate, depth of breathing, and breathing symmetry. We have conducted studies where the subjects performed different types of guided breathing exercises while wearing the earbuds. Our algorithms detect breathing phases with 90.91% F1-score and estimate breathing rate with 95.05% accuracy. We further show that our algorithms can be used to generate biofeedback towards designing engaging smartphone's user interactions that facilitate users to accurately perform various breathing exercises.

Breathing exercises reduce stress and improve overall mental well-being. There are various types of breathing exercises. Performing the exercises correctly may give the best outcome and doing it in wrong ways can sometimes have adverse effect. Providing real-time biofeedback can greatly improve the user experience in doing the right exercises in the right ways. In this paper, we present methods to passively track breathing biomarkers in real-time using wireless commodity earbuds and generate feedback on users' breathing performance. We use the earbud's low-power accelerometer to generate a comprehensive set of breathing biomarkers including breathing phase, breathing rate, depth of breathing, and breathing symmetry. We have conducted studies where the subjects performed different types of guided breathing exercises while wearing the earbuds. Our algorithms detect breathing phases with 90.91% F1-score and estimate breathing rate with 95.05% accuracy. We further show that our algorithms can be used to generate biofeedback towards designing engaging smartphone's user interactions that facilitate users to accurately perform various breathing exercises.

Coughing is a common symptom across different clinical conditions and has gained further relevance in the past years due to the COVID-19 pandemic. An automated cough detection for continuous health monitoring could be developed using Earbud, a wearable sensor platform with audio and inertial measurement unit (IMU) sensors. Though several previous works have investigated audio-based automated cough detection, audio-based methods can be highly power-consuming for wearable sensor applications and raise privacy concerns. In this work, we develop IMU-based cough detection using a template matching-based algorithm. IMU provides a low-power privacy-preserving solution to complement audio-based algorithms. Similarly, template matching has low computational and memory needs, suitable for on-device implementations. The proposed method uses feature transformation of IMU signal and unsupervised representative template selection to improve upon our previous work. We obtained an AUC (AUC-ROC) of 0.85 and 0.83 for cough detection in a lab-based dataset with 45 participants and a controlled free-living dataset with 15 participants, respectively. These represent an AUC improvement of 0.08 and 0.10 compared to the previous work. Additionally, we conducted an uncontrolled free-living study with 7 participants where continuous measurements over a week were obtained from each participant. Our cough detection method achieved an AUC of 0.85 in the study, indicating that the proposed IMU-based cough detection translates well to the varied challenging scenarios present in free-living conditions.

Persistent coughs are a major symptom of respiratory-related diseases. Increasing research attention has been paid to detecting coughs using wearables, especially during the COVID-19 pandemic. Microphone is most widely used sensor to detect coughs. However, the intense power consumption needed to process audio hinders continuous audio-based cough detection on battery-limited commercial wearables, such as earbuds. We present CoughTrigger, which utilizes a lower-power sensor, inertial measurement unit (IMU), in earbuds as a cough detection activator to trigger a higher-power sensor for audio processing and classification. It runs all-the-time as a standby service with minimal battery consumption and triggers the audio-based cough detection when a candidate cough is detected from IMU. Besides, the use of IMU brings the benefit of improved specificity of cough detection. Experiments are conducted on 45 subjects and CoughTrigger achieved 0.77 AUC score. We also validated its effectiveness on free-living data and through on-device implementation.

Tracking breathing phases (inhale and exhale) outside the hospitals can offer significant health and wellness benefits. For example, the breathing phases can provide fine-grained breathing information for breathing exercises. While previous works use smartphones and smartwatches for tracking breathing phases, in this work, we use earbuds for breathing phase detection, which can be a better form factor for breathing exercises as it requires less user attention from the user. We propose a convolutional neural network-based algorithm for detecting breathing phases using the audio captured through the earbuds during guided breathing sessions. We conducted a user study with 30 participants in both lab and home environments to develop and evaluate our algorithm. Our algorithm can detect the breathing phases with 85% accuracy by taking only a 500ms audio signal. Our work demonstrates the potential of using earbuds for tracking the breathing phases in real-time.

Lung health assessment is traditionally done mainly through X-ray images and spirometry tests which are time-consuming, cumbersome, and costly. In this paper, we investigate the potential of passively recordable contents such as speech, cough and heart signal for such an assessment. Our regression model is the first in the literature to achieve mean absolute error (MAE) of 7.47% for estimation of forced expiratory volume in 1 sec. (FEV1) over forced vital capacity (FVC) ratio using these contents. This is comparable to the state of the art active phone-based spirometry methods. Additionally our classification models achieve a F1-score of 0.982 for healthy v.s. diseased, 0.881 for obstructive v.s. non-obstructive, 0.854 for chronic obstructive pulmonary disease (COPD) v.s. asthma, and 0.892 for severe v.s. non-severe obstruction classification.

Breathing biomarkers, such as breathing rate, fractional inspiratory time, and inhalation-exhalation ratio, are vital for monitoring the user's health and well-being. Accurate estimation of such biomarkers requires breathing phase detection, i.e., inhalation and exhalation. However, traditional breathing phase monitoring relies on uncomfortable equipment, e.g., chestbands. Smartphone acoustic sensors have shown promising results for passive breathing monitoring during sleep or guided breathing. However, detecting breathing phases using acoustic data can be challenging for various reasons. One of the major obstacles is the complexity of annotating breathing sounds due to inaudible parts in regular breathing and background noises. This paper assesses the potential of using smartphone acoustic sensors for passive unguided breathing phase monitoring in a natural environment. We address the annotation challenges by developing a novel variant of the teacher-student training method for transferring knowledge from an inertial sensor to an acoustic sensor, eliminating the need for manual breathing sound annotation by fusing signal processing with deep learning techniques. We train and evaluate our model on the breathing data collected from 131 subjects, including healthy individuals and respiratory patients. Experimental results show that our model can detect breathing phases with 77.33% accuracy using acoustic sensors. We further present an example use-case of breathing phase-detection by first estimating the biomarkers from the estimated breathing phases and then using these biomarkers for pulmonary patient detection. Using the detected breathing phases, we can estimate fractional inspiratory time with 92.08% accuracy, the inhalation-exhalation ratio with 86.76% accuracy, and the breathing rate with 91.74% accuracy. Moreover, we can distinguish respiratory patients from healthy individuals with up to 76% accuracy. This paper is the first to show the feasibility of detecting regular breathing phases towards passively monitoring respiratory health and well-being using acoustic data captured by a smartphone.

Breathing biomarkers, such as breathing rate, fractional inspiratory time, and inhalation-exhalation ratio, are vital for monitoring the user's health and well-being. Accurate estimation of such biomarkers requires breathing phase detection, i.e., inhalation and exhalation. However, traditional breathing phase monitoring relies on uncomfortable equipment, e.g., chestbands. Smartphone acoustic sensors have shown promising results for passive breathing monitoring during sleep or guided breathing. However, detecting breathing phases using acoustic data can be challenging for various reasons. One of the major obstacles is the complexity of annotating breathing sounds due to inaudible parts in regular breathing and background noises. This paper assesses the potential of using smartphone acoustic sensors for passive unguided breathing phase monitoring in a natural environment. We address the annotation challenges by developing a novel variant of the teacher-student training method for transferring knowledge from an inertial sensor to an acoustic sensor, eliminating the need for manual breathing sound annotation by fusing signal processing with deep learning techniques. We train and evaluate our model on the breathing data collected from 131 subjects, including healthy individuals and respiratory patients. Experimental results show that our model can detect breathing phases with 77.33% accuracy using acoustic sensors. We further present an example use-case of breathing phase-detection by first estimating the biomarkers from the estimated breathing phases and then using these biomarkers for pulmonary patient detection. Using the detected breathing phases, we can estimate fractional inspiratory time with 92.08% accuracy, the inhalation-exhalation ratio with 86.76% accuracy, and the breathing rate with 91.74% accuracy. Moreover, we can distinguish respiratory patients from healthy individuals with up to 76% accuracy. This paper is the first to show the feasibility of detecting regular breathing phases towards passively monitoring respiratory health and well-being using acoustic data captured by a smartphone.

The prevalence of ubiquitous computing enables new opportunities for lung health monitoring and assessment. In the past few years, there have been extensive studies on cough detection using passively sensed audio signals. However, the generalizability of a cough detection model when applied to external datasets, especially in real-world implementation, is questionable and not explored adequately. Beyond detecting coughs, researchers have looked into how cough sounds can be used in assessing lung health. However, due to the challenges in collecting both cough sounds and lung health condition ground truth, previous studies have been hindered by the limited datasets. In this paper, we propose Listen2Cough to address these gaps. We first build an end-to-end deep learning architecture using public cough sound datasets to detect coughs within raw audio recordings. We employ a pre-trained MobileNet and integrate a number of augmentation techniques to improve the generalizability of our model. Without additional fine-tuning, our model is able to achieve an F1 score of 0.948 when tested against a new clean dataset, and 0.884 on another in-the-wild noisy dataset, leading to an advantage of 5.8% and 8.4% on average over the best baseline model, respectively. Then, to mitigate the issue of limited lung health data, we propose to transform the cough detection task to lung health assessment tasks so that the rich cough data can be leveraged. Our hypothesis is that these tasks extract and utilize similar effective representation from cough sounds. We embed the cough detection model into a multi-instance learning framework with the attention mechanism and further tune the model for lung health assessment tasks. Our final model achieves an F1-score of 0.912 on healthy v.s. unhealthy, 0.870 on obstructive v.s. non-obstructive, and 0.813 on COPD v.s. asthma classification, outperforming the baseline by 10.7%, 6.3%, and 3.7%, respectively. Moreover, the weight value in the attention layer can be used to identify important coughs highly correlated with lung health, which can potentially provide interpretability for expert diagnosis in the future.

Breathing rate is an important vital sign and an indicator of overall health and fitness. Traditionally breathing is monitored using specialized devices such as chestband or spirometers which are uncomfortable for daily use. Recent works show the feasibility of estimating breathing rate using earbuds' motion sensors. However, non-breathing head motion is one of the biggest challenges for breathing rate estimation using earbuds. In this paper, we propose algorithms to estimate breathing rate in presence of non-breathing head motion using inertial sensors embedded in commodity earbuds. Using the chestband as a reference device, we show that our algorithms can estimate breathing rate in resting positions with error rate 2.34 breaths per minute (BPM). Our algorithms can handle passive head motion and reduce the error by 27.78%. Furthermore, our algorithms can handle active head motion and help reduce the error by 45.70% when intentional non-breathing head motion is present in the data segment. It can be a big stride towards passive breathing monitoring in daily life using commodity earbuds.