Continuous clinical grade measurement of SpO2 in out-of-hospital settings remains a challenge despite the widespread use of photoplethysmography (PPG) based wearable devices for health and wellness applications. This article presents two SpO2 algorithms: PRR (pulse rate derived ratio-of-ratios) and GPDR (green-assisted peak detection ratio-of-ratios), that utilize unique pulse rate frequency estimations to isolate the pulsatile (AC) component of red and infrared PPG signals and derive SpO2 measurements. The performance of the proposed SpO2 algorithms are evaluated using an upper-arm wearable device derived green, red, and infrared PPG signals, recorded in both controlled laboratory settings involving healthy subjects (n=36) and an uncontrolled clinic application involving COVID-19 patients (n=52). GPDR exhibits the lowest root mean square error (RMSE) of 1.6±0.6% for a respiratory exercise test, 3.6 ±1.0% for a standard hypoxia test, and 2.2±1.3% for an uncontrolled clinic use-case. In contrast, PRR provides relatively higher error but with greater coverage overall. Mean error across all combined datasets were 0.2±2.8% and 0.3±2.4% for PRR and GPDR respectively. Both SpO2 algorithms achieve great performance of low error with high coverage on both uncontrolled clinic and controlled laboratory conditions.

Employment outcomes for autistic 1 individuals are often poorer relative to their neurotypical (NT) peers, resulting in a greater need for other forms of financial and social support. While a great deal of work has focused on developing interventions for autistic children, relatively less attention has been paid to directly addressing the employment challenges faced by autistic adults. One key impediment to autistic individuals securing employment is the job interview. Autistic individuals often experience anxiety in interview situations, particularly with open-ended questions and unexpected interruptions. They also exhibit atypical gaze patterns that may be perceived as, but not necessarily indicative of, disinterest or inattention. In response, we developed a closed-loop adaptive virtual reality (VR)–based job interview training platform, which we have named Career Interview Readiness in VR (CIRVR). CIRVR is designed to provide an engaging, adaptive, and individualized experience to practice and refine interviewing skills in a less anxiety-inducing virtual context. CIRVR contains a real-time physiology-based stress detection module, as well as a real-time gaze detection module, to permit individualized adaptation. We also present the first prototype of the CIRVR Dashboard, which provides visualizations of data to help autistic individuals as well as potential employers and job coaches make sense of the data gathered from interview sessions. We conducted a feasibility study with 9 autistic and 8 NT individuals to assess the preliminary usability and feasibility of CIRVR. Results showed differences in perceived usability of the system between autistic and NT participants, and higher levels of stress in autistic individuals during interviews. Participants across both groups reported satisfaction with CIRVR and the structure of the interview. These findings and feedback will support future work in improving CIRVR’s features in hopes for it to be a valuable tool to support autistic job candidates as well as their potential employers.

Photoplethysmography (PPG) and accelerometer (ACC) are commonly integrated into wearable devices for continuous unobtrusive pulse rate and activity monitoring of individuals during daily life. However, obtaining continuous and clinically accurate respiratory rate measurements using such wearable sensors remains a challenge. This article presents a novel algorithm for estimation of respiration rate (RR) using an upper-arm worn wearable device by deriving multiple respiratory surrogate signals from PPG and ACC sensing. This RR algorithm is retrospectively evaluated on a controlled respiratory clinical testing dataset from 38 subjects with simultaneously recorded wearable sensor data and a standard capnography monitor as an RR reference. The proposed RR method shows great performance and robustness in determining RR measurements over a wide range of 4–59 brpm with an overall bias of -1.3 brpm, mean absolute error (MAE) of 2.7±1.6 brpm, and a meager outage of 0.3±1.2%, while a standard PPG Smart Fusion method produces a bias of -3.6 brpm, an MAE of 5.5±3.1 brpm, and an outage of 0.7±2.5% for direct comparison. In addition, the proposed algorithm showed no significant differences (p=0.63) in accurately determining RR values in subjects with darker skin tones, while the RR performance of the PPG Smart Fusion method is significantly (P<0.001) affected by the darker skin pigmentation. This study demonstrates a highly accurate RR algorithm for unobtrusive continuous RR monitoring using an armband wearable device.

Abstract Background Social impairment is a core feature of schizophrenia presenting a major barrier to recovery. Although antipsychotic medications can reduce psychotic symptoms, social impairments often persist, contributing to poor outcome. Validated interventions, such as Social Skills Training (Bellack et al., 2004), target a broad range of social domains but they yield only modest effect sizes for social outcome (Pfammatter et al, 2006). Moreover, conventional social interventions suffer from high burden on the clients and therapists, low adherence, lack of personalization, and low generalizability. Importantly, social interventions are not widely available. Virtual reality (VR) offers a viable alternative to conventional therapies with several advantages including high acceptability, an extensive repertoire of stimuli, low-burden, low-cost and safety (Strickland, 1997). Importantly, VR allows for a precise targeting of social cognitive mechanisms. Social attention, the fast orientation and allocation of resources to social stimuli can be indexed by tracking eye movements. Abnormal eye scanning behavior during social interactions has been linked to poor social functioning in schizophrenia (Brunet-Gouet & Decety, 2006). We designed a VR-based social skills training game to directly target social attention in schizophrenia. Methods Eighteen individuals with schizophrenia (SZ) participated in the VR training game twice a week for 5 weeks. Eye movement patterns were recorded throughout the training. Subjects were required to solve social “missions” (e.g., obtain personal information through conversations with avatars) in different naturalistic scenarios. To start a mission, participants had to fixate on the chosen avatar. The latency to engage in a social interaction was indexed by the fixation time (social engagement latency). Each session consisted of 12 missions. We compared social engagement latency and eye gaze patterns pre- and post-training to assess the efficacy of the VR social skills training program. Results Social engagement latency significantly decreased after 10 sessions (d=0.78). This result suggests that prosocial attention benefitted from VR training. We also found that the standard deviation of dwell time (i.e., proportion of time spent looking at the avatar’s face during a conversation) significantly increased across training sessions (d=0.56). This result suggests an increased modulation of interpersonal engagement during social interaction. Additionally, participants’ emotion recognition ability significantly increased (η2 = 0.27), and negative symptoms significantly decreased (η2=0.34) from pre- to post- training. Importantly, these changes in social attention correlated with improvement in negative symptoms. Discussion Simulated and targeted social interactions with avatars in VR significantly improved social attention in individuals with schizophrenia. Importantly, improvements in negative symptoms and emotion perception after training suggest that this training protocol has an impact on a broad range of social functions. VR training is a promising alternative to traditional psychosocial interventions to target specific mechanisms underlying social functioning in schizophrenia.

Respiratory rate (RR) is an important vital sign marker of health, and it is often neglected due to a lack of unobtrusive sensors for objective and convenient measurement. The respiratory modulations present in simple photoplethysmogram (PPG) have been useful to derive RR using signal processing, waveform fiducial markers, and hand-crafted rules. An end- to-end deep learning approach based on residual network (ResNet) architecture is proposed to estimate RR using PPG. This approach takes time-series PPG data as input, learns the rules through the training process that involved an additional synthetic PPG dataset generated to overcome the insufficient data problem of deep learning, and provides RR estimation as outputs. The inclusion of a synthetic dataset for training improved the performance of the deep learning model by 34%. The final mean absolute error performance of the deep learning approach for RR estimation was 2.5±0.6 brpm using 5-fold cross-validation in two widely used public PPG datasets (n=95) with reliable RR references. The deep learning model achieved comparable performance to that of a classical method, which was also implemented for comparison. With large real-world data and reference ground truth, deep learning can be valuable for RR or other vital sign monitoring using PPG and other physiological signals.

Autism Spectrum Disorders (ASD) is a prevalent developmental disorder and is associated with high familial and societal cost. Early interventions during the first year can have the best developmental outcomes despite the fact that the earliest diagnosis of ASD is only possible by the age of two. Investigating brain response to basic stimuli like sight, smell and touch has proved to have the potential to find markers between individuals with ASD and their neurotypical peers during infancy. Since existing tactile stimulus delivering method tend to suffer from low accuracy, low availability and low tolerability, it is necessary to develop a precise, high-tolerable tactile stimulus delivering mechanism. The present study examined the feasibility and tolerability of Soft-Brush, a comfortable, mobile silicone tactile stimulator with tendon-driven mechanism, for delivering tactile stimulus in multisensory studies. Experiments have shown that Soft-Brush has high tolerance rate by the children during experiments resulting in reliable data collection.

Emotional functioning deficits are a core symptom of schizophrenia contributing to interpersonal difficulties. Emotion perception and expression mediate social interactions by allowing a shared representation of emotional experiences between individuals. Emotion recognition has been consistently found impaired in individuals with schizophrenia. Spontaneous facial mimicry, a process by which individuals automatically imitate each other’s facial expressions during interactions, has previously been shown to facilitate emotion recognition. However, studies assessing automatic facial mimicry in schizophrenia have yielded mixed results. Moreover, to our knowledge, spontaneous facial mimicry and emotion recognition have not been assessed simultaneously in individuals with schizophrenia. The primary aim of this study was to test whether automatic facial mimicry deficits underlie emotion recognition impairments in schizophrenia. 21 Individuals with schizophrenia (SZ) and 23 demographically matched controls (CO) completed a novel dynamic emotion recognition task (Emotion Recognition and Rating Application Task, ERRATA) while facial electromyographic (EMG) activity was recorded. ERRATA was designed to display faces of a diverse set of avatars with varying emotional expression at different levels of intensity. Each trial of the ERRATA task consisted of the presentation of an avatar face with neutral expression for 2.5 seconds, after which the avatar’s facial expression changed to express an emotion for 2 seconds. Participants were asked to identify the avatar’s emotion by selecting from a list of 7 words, and to provide a confidence rating. The BIOPAC MP150 system was used to record EMG in corrugator supercilli and zygomaticus major throughout the ERRATA. To investigate the relationship between functional EMG (fEMG) data, the task performance of the participant and the emotion displayed by the avatar, machine-learning method was implemented. Specifically, fEMG features were used to predict the emotions presented by the avatar, as well as participants’ choices. SZ performed significantly worse than CO on the ERRATA task. SZ emotion recognition impairment was correlated to negative, but not positive symptoms. When fEMG data were used to predict the emotions displayed by the avatars, the machine-learning models for CO and SZ achieved similar results, which suggests that spontaneous facial mimicry was unimpaired in SZ. However, when we used fEMG data to predict the participants’ selected emotions, the models for CO statistically outperformed the SZ models. In CO, fEMG and emotion recognition performance were closely associated but not in SZ. Although spontaneous facial mimicry was intact in SZ, their ability to identify emotions of others was impaired. These results indicate that whilst the automatic process underlying shared emotional experiences during social encounters is unaffected in schizophrenia, the conscious, explicit process of identifying emotion seems impaired. This finding suggests that emotion recognition deficits in SZ might involve a higher-order mechanism that allows for cross-indexing of unconscious internal states to conceptual categories of emotion.

Past research indicates that spontaneous mimicry facilitates the decoding of others' emotions, leading to enhanced social perception and interpersonal rapport. Individuals with schizophrenia (SZ) show consistent deficits in emotion recognition and expression associated with poor social functioning. Given the prominence of blunted affect in schizophrenia, it is possible that spontaneous facial mimicry may also be impaired. However, studies assessing automatic facial mimicry in schizophrenia have yielded mixed results. It is therefore unknown whether emotion recognition deficits and impaired automatic facial mimicry are related in schizophrenia. SZ and demographically matched controls (CO) participated in a dynamic emotion recognition task. Electromyographic activity in muscles responsible for producing facial expressions was recorded during the task to assess spontaneous facial mimicry. SZ showed deficits in emotion identification compared to CO, but there was no group difference in the predictive power of spontaneous facial mimicry for avatar's expressed emotion. In CO, facial mimicry supported accurate emotion recognition, but it was decoupled in SZ. The finding of intact facial mimicry in SZ bears important clinical implications. For instance, clinicians might be able to improve the social functioning of patients by teaching them to pair specific patterns of facial muscle activation with distinct emotion words.

Driving is essential for many people in developed countries to achieve independence. Individuals with Autism Spectrum Disorder (ASD), in addition to having social skill deficits, may experience difficulty in learning to drive due to deficits in attention-shifting, performing sequential tasks, integrating visual-motor responses, and coordinating motor response. Lacking confidence and feeling anxiety further exacerbates these concerns. While there is a growing body of research regarding assessment of driving behavior or comparisons of driving behaviors between individuals with and without ASD, there is a lack of driving simulator that is catered toward the needs of individuals with ASD. We present the development of a novel closed-loop adaptive Virtual Reality (VR) driving simulator for individuals with ASD that can infer one's engagement based on his/her physiological responses and adapts driving task difficulty based on engagement level in real-time. We believe that this simulator will provide opportunities for learning driving skills in a safe and individualized environment to individuals with ASD and help them with independent living. We also conducted a small user study with teenagers with ASD to demonstrate the feasibility and tolerability of such a driving simulator. Preliminary results showed that the participants found the engagement-sensitive system more engaging and more enjoyable than a purely performance-sensitive system. These findings could support future work into driving simulator technologies, which could provide opportunities to practice driving skills in cost-effective, supportive, and safe environments.

Social impairment is a core feature of schizophrenia. It is present throughout the course of the illness from the premorbid stage and resistant to treatments (Green et al, 2008). Given the important role of social deficits in poor outcome, it is imperative to improve social functioning in individuals with schizophrenia (SZ), but currently available pharmacological and psychosocial interventions have not proven to be very effective. However, recent technological advances have enabled the use of virtual reality (VR) to develop novel psychiatric interventions. We have previously reported preliminary feasibility and acceptability findings from a new VR-based social interaction training program that suggests the benefits of simulating and rehearsing social interactions situations in VR (Adery et al, 2018). In the present study, we evaluated predictors and mediators of the clinical outcome of this VR-intervention. 18 outpatients with schizophrenia completed 10 sessions of the social VR training over the course of 5 weeks. At each session, participants were asked to approach and interact with avatars by making appropriate conversations to accomplish social ‘missions’. There were 12 different missions varying in difficulty levels and social settings (café, bus stop and shop). Symptoms (BPRS, SAPS, SANS), emotion recognition (BLERT and a novel facial emotion recognition task), social functioning (Social Functioning Scale) and cognitive functioning (IQ, CogState) were examined in relation to the VR game performance at pre- and post-treatment. Multivariate regression models with backward elimination method were used to verify significant predictors of the outcome. Lastly, mediation analyses were evaluated using bootstrapping (n=5000) to specify the relationship between predictors and behavioral changes. Negative symptoms, especially alogia, anhedonia and asociality improved significantly by post-assessment. Additionally, attention and emotion recognition improved. Within the VR game, number of incorrect responses declined significantly throughout the training period. In multivariate regression models, baseline and changes in social functioning scores predicted clinical symptom improvement. Furthermore, IQ mediated the relationship between performance change in high-difficulty sessions and improvement in alogia. Alogia improvement mediated the relationship between interpersonal communication changes and VR performance. Simulation-based social skills VR training game improved clinical symptoms, especially negative symptoms, via changes in social functioning and emotion recognition in schizophrenia. These results suggest that VR-based social skills training may be an effective social intervention method. Although the small sample size limits the scope of our conclusion, the results of the present study illustrate the potential power of technology-based psychiatric interventions.

Bodily-self disturbances and anomalous emotional functioning are core features of schizophrenia that play a major role in social and functional outcome. Much has been written about abnormal perception and expression of emotions in schizophrenia but less is known about the bodily experience of emotions in this population. The prevalence of anomalous bodily-self experiences (Parnas & Handset, 2003), impaired simulation (Park et al, 2008) and interoception deficits (Ardizzi et al., 2016) in schizophrenia suggests that embodiment of emotions might be altered in this population. We investigated emotional embodiment in individuals with schizophrenia (SZ) and demographically-matched controls to determine whether SZ experience anomalous bodily sensations of emotions. We then implemented a novel Virtual Reality (VR) social skills intervention that required participants to simulate social interactions with avatars. The VR training was designed to target social attention and improve simulation of other people’s emotions, intentions, and actions in SZ, thereby improving embodiment of emotions. We recruited twenty-six individuals with schizophrenia (SZ) and 26 demographically matched controls (CO). At baseline, we assessed social functioning, cognitive functions, symptom and embodied emotions. An online body mapping task (Nummenmaa et al., 2014) was used to generate spatial maps of bodily sensations experienced during 14 emotions categories. Then, SZ participated in the 5-week, novel VR social skills intervention that targeted social attention and simulation. Naturalistic scenarios, in which subjects moved through variable sequences of steps to attain the goal of a “mission” (e.g. find out the birthday of the avatar etc.). Subject interacted with an avatar and practiced perspective-taking and pragmatics to advance to the next level of difficulty. At baseline, bodily sensation maps show overall reduced embodiment of emotions in SZ as compared to CO. Statistical pattern recognition with Linear Discriminant Analysis (LDA) revealed less unique bodily sensations of emotions in SZ. Similarity scores between the maps of CO and SZ revealed a specific deficit in embodiment of low-arousal emotions (i.e. depression, sadness, shame) in SZ. After five weeks of VR training, negative symptoms and emotional embodiment improved in SZ. Specifically, embodiment of low-arousal emotions increased. Moreover, changes in the body maps of emotion indicated increased concordance among SZ. Anomalous embodiment of emotions plays an important role in the poor social outcome of individuals with schizophrenia, but a 5-week VR training of social attention, simulation, perspective taking, and communication skills was effective in improving emotional embodiment. Further research is warranted to elucidate underlying social cognitive mechanisms that link self-disturbances and embodiment of emotions.

Sensory processing differences, including responses to auditory, visual, and tactile stimuli, are ideal targets for early detection of neurodevelopmental risks, such as autism spectrum disorder. However, most existing studies focus on the audiovisual paradigm and ignore the sense of touch. In this paper, we present a multisensory delivery system that can deliver audio, visual, and tactile stimuli in a controlled manner and capture peripheral physiological, eye gaze, and electroencephalographic response data. The novelty of the system is the ability to provide affective touch. In particular, we have developed a tactile stimulation device that delivers tactile stimuli to infants with precisely controlled brush stroking speed and force on the skin. A usability study of 10 3-20 month-old infants was conducted to investigate the tolerability and feasibility of the system. Results have shown that the system is well tolerated by infants and all the data were collected robustly. This paper paves the way for future studies charting the sensory response trajectories in infancy.

Emotion recognition impairment is a core feature of schizophrenia (SZ), present throughout all stages of this condition, and leads to poor social outcome. However, the underlying mechanisms that give rise to such deficits have not been elucidated and hence, it has been difficult to develop precisely targeted interventions. Evidence supports the use of methods designed to modify patterns of visual attention in individuals with SZ in order to effect meaningful improvements in social cognition. To date, however, attention-shaping systems have not fully utilized available technology (e.g., eye tracking) to achieve this goal. The current work consisted of the design and feasibility testing of a novel gaze-sensitive social skills intervention system called MASI-VR. Adults from an outpatient clinic with confirmed SZ diagnosis ( n = 10) and a comparison sample of neurotypical participants ( n = 10) were evaluated on measures of emotion recognition and visual attention at baseline assessment, and a pilot test of the intervention system was evaluated on the SZ sample following five training sessions over three weeks. Consistent with the

Deficits in social cognition and on social perception tasks are well studied and widely found in populations with schizophrenia. In addition, our work consistently replicates findings that individuals with schizophrenia report severe loneliness, significantly higher than healthy matches. Loneliness is a chronic, gnawing condition that induces distress and impedes life satisfaction and function across the spectrum of mental health. We also find social isolation impedes interpretation of social information and may lead to socio-perceptual deficits. The present study examines the effectiveness of a novel, adaptive virtual reality simulated social exposure training intervention (see Bekele et al, 2016) in both decreasing feelings of loneliness and improving social cognitive function in individuals with schizophrenia. We investigate baseline relationships between social isolation, loneliness and social cognition abilities, as well as pre to post intervention changes in function and subjective social well-being. Fifteen medicated SZ outpatients completed 10 virtual reality social skills training sessions over the course of 5 weeks. Training sessions depicted three naturalistic social scenarios in which participants were instructed to complete 12 total social “missions” to obtain information from VR avatar characters. Prior to training and following the final training session, participants were assessed using the CogState Brief Schizophrenia Battery Social Emotional cognition task and rated loneliness using the UCLA Loneliness Scale. Independent raters conducted pre- and post-training clinical interviews to assess changes in participants’ levels of positive, negative, and overall psychiatric symptoms Greater overall psychiatric symptoms were significantly correlated with higher levels of experienced loneliness, consistent with previous findings. There was a significant improvement in social emotional cognition accuracy, and a trend-level reduction in loneliness from pre-training to post-testing following social VR training. Previous research indicates that individuals higher on the psychosis spectrum perform worse at social cognition and social perception tasks. Our own research indicates that individuals higher on the psychosis spectrum also endorse higher levels of social distress via social isolation and loneliness. The present study attempts to enhance social cognitive and interpersonal abilities of individuals with schizophrenia while decreasing loneliness by strengthening social bonds and skills using a virtual reality training game. We find that following 10 sessions of VR social training, accuracy on measures of social cognition is improved significantly, however loneliness is reduced non-significantly. These initial results demonstrate potential feasibility of a novel VR social skills training game for improving social experience for patients with schizophrenia.

Social impairment is a core feature of schizophrenia that presents a major barrier toward recovery. Some of the psychotic symptoms are partly ameliorated by medication but the route to recovery is hampered by social impairments. Since existing social skills interventions tend to suffer from lack of availability, high-burden and low adherence, there is a dire need for an effective, alternative strategy. The present study examined the feasibility and acceptability of Multimodal Adaptive Social Intervention in Virtual Reality (MASI-VR) for improving social functioning and clinical outcomes in schizophrenia. Out of eighteen patients with schizophrenia who enrolled, seventeen participants completed the pre-treatment assessment and 10 sessions of MASI-VR, but one patient did not complete the post-treatment assessments. Therefore, the complete training plus pre- and post-treatment assessment data are available from sixteen participants. Clinical ratings of symptom severity were obtained at pre- and post-training. Retention rates were very high and training was rated as extremely satisfactory for the majority of participants. Participants exhibited a significant reduction in overall clinical symptoms, especially negative symptoms following 10 sessions of MASI-VR. These preliminary results support the feasibility and acceptability of a novel virtual reality social skills training program for individuals with schizophrenia.

Social impairment is a core feature of schizophrenia presenting a major barrier to recovery. Although antipsychotic medications can reduce psychotic symptoms, social functioning often remains poor, contributing to the financial burden of schizophrenia. Validated behavioral interventions, such as Social Skills Training (Liberman & Martin, 1988), target a broad range of social domains by practicing pragmatic living skills. But they yield only modest effect sizes for social outcome (Pfammatter et al, 2006). Conventional social interventions present further limitations including: time and effort required from patients and therapists, low adherence, lack of personalization, and low generalizability. Importantly, most people with mental illness do not currently have access to social interventions. The aim of this study was to design and implement an effective, high-compliance virtual reality (VR) social skills training game for people with schizophrenia. The advantages of the VR environment include flexibility, controllability, extensive repertoire of stimuli, low-burden, low-cost and safety (Strickland, 1997). The goal of the training game was to support social attention to improve social skills learning. We trained social skills by exercising problem-solving in naturalistic scenarios: the grocery store, a bus stop, and a cafeteria. Subjects moved through variable steps in a social “mission” to obtain personal information through conversations with avatars. Each mission began with the participant fixating on the avatar. Subjects then had to decide which avatar to approach and choose an appropriate response to the avatar’s prompts. If they chose an incorrect response, oral feedback was provided on why this response was not the most effective, and instructed them to try again. This occurred until the participant identified the most appropriate response, thus completing the mission and getting access to the next level of difficulty. Each training session concluded after completion of 12 total conversation missions. Eighteen individuals with schizophrenia (SZ) and twenty demographically matched controls (CO) participated in this study. At baseline, SZ and CO completed pre-training assessments. The CO group did not undergo VR training but participated in behavioral assessments so that we could compare SZ performance to normative data. SZ participated in the VR training twice a week for 5 weeks (10 sessions). After the 10th session, we re-examined social functioning, cognitive functions, and symptoms. SZ also completed a satisfaction survey upon training completion. Of the eighteen SZ participants enrolled in the study, sixteen completed the 10 sessions of training, yielding a retention rate of 89%. 80% of SZ subjects reported being “extremely satisfied” with the training. None reported not being satisfied. 93.3% rated the training as “acceptable” and the effort required to attend the study as “easy.” Regarding outcome, negative symptoms significantly decreased from pre-training to post-training. Performance on BLERT and CogState Social Emotional Cognitive Task significantly improved. The average time taken to complete a mission was significantly lower in the last session compared to the first, showing that participants became increasingly better at efficiently solving these social missions. These results show evidence for VR training as an acceptable and feasible intervention improving social functioning in SZ. Future work will test the adaptive social VR training against an active control condition in a pilot randomized controlled trial to evaluate the relative efficacy of the VR training on enhancing social attention and associated neural circuitry.

Individuals with schizophrenia (SZ) exhibit significant difficulties processing and perceiving socioemotional information conveyed by others. Increasing evidence suggests that SZ deficits in facial emotion recognition, in particular, contribute to impaired daily social functioning. Studies show that improving SZ patients’ visual attention to socially-relevant facial areas (eyes, nose, mouth) with targeted computer interventions will ameliorate deficits in emotion recognition. We tested whether 10 sessions of a novel, VR-based social simulation computer game would indirectly improve facial emotion recognition in SZ, and additionally whether potential gains were associated with changes in gaze patterns. Fifteen SZ outpatients completed a social simulation computer game intervention involving a pre-training visit, 10 training sessions scheduled approximately twice per week (days until completion: M=38.8, SD=16), and a post-training visit. During training sessions, participants played a novel, adaptive VR-based computer game that involved approaching and engaging in conversations with various Avatar game characters across several naturalistic settings (a bus stop, café, grocery store). Each game session required completion of 12 “social missions” to determine information about different characters (e.g., food preference), achieved by selecting the appropriate conversational prompts and follow-up questions from multiple options. At pre- and post-training visits, emotion recognition was assessed with a novel dynamic facial affect recognition task (DFAR) and the Bell Lysaker Emotion Recognition Task (BLERT). During the DFAR, participants viewed adult Avatar characters (50% female) making one of 8 dynamic facial expressions (anger, sadness, fear, disgust, joy, surprise, contempt) while gaze data and behavioral responses were recorded. The VR-based computer game and DFAR were developed in-house with Autodesk Maya 3D animation and Unity software (unity3d.com). Patients’ emotion recognition accuracy (BLERT) significantly improved from pre- to post-training. Patients’ accuracy on the facial affect recognition task (DFAR) also significantly improved following training for specific negative emotions (anger, contempt, fear, and sadness). Regarding changes in visual attention, patients made overall fewer fixations at post-training (fixation duration threshold = 200ms) across relevant social areas (eyes, nose, mouth) when viewing emotional Avatar faces compared to pre-training. A general reduction in fixations was not accompanied by an increase in mean fixation duration. Rather, shorter fixation durations were positively associated with DFAR performance accuracy. SZ patients’ participation in a novel, VR-based computerized social simulation training may yield indirect benefits in emotion recognition. Specifically, patients exhibited improvements on a validated assessment of emotion perception (BLERT) following the 10-session computer training. A decrease in the number of fixations on socially-informative facial regions during Avatars’ emotion expression on the DFAR may indicate an increased efficiency in scanning for socioemotional information. Though much work remains in probing the exact nature of treatment mechanism and durability of these improvements, these promising initial results demonstrate the potential of an VR-based computer game for improving core deficits in social cognition.

Individuals with Autism Spectrum Disorder (ASD), compared to typically-developed peers, may demonstrate behaviors that are counter to safe driving. The current work examines the use of a novel simulator in two separate studies. Study 1 demonstrates statistically significant performance differences between individuals with (N = 7) and without ASD (N = 7) with regards to the number of turning-related driving errors (p < 0.01). Study 2 shows that both the performance-based feedback group (N = 9) and combined performance- and gaze-sensitive feedback group (N = 8) achieved statistically significant reductions in driving errors following training (p < 0.05). These studies are the first to present results of fine-grained measures of visual attention of drivers and an adaptive driving intervention for individuals with ASD.

Sensory processing differences, including auditory, visual, and tactile, are ideal targets for early detection of neurodevelopmental risk. However, existing studies focus on the audiovisual paradigm but ignore the sense of touch. In this work, we present a multisensory delivery system that can deliver audiovisual stimuli and precisely controlled tactile stimuli to infants in a synchronized manner. The system also records multi-dimensional data including eye gaze and physiological data. A pilot study of six 3–8 month old infants was conducted to investigate the tolerability and feasibility of the system. Results have shown that the system is well tolerated by infants and all the data were collected robustly. This work paves the way for future studies charting the meaning of sensory response trajectories in infancy.

Autism Spectrum Disorder (ASD) is a highly prevalent neurodevelopmental disorder with enormous individual and social cost. In this paper, a novel virtual reality (VR)-based driving system was introduced to teach driving skills to adolescents with ASD. This driving system is capable of gathering eye gaze, electroencephalography, and peripheral physiology data in addition to driving performance data. The objective of this paper is to fuse multimodal information to measure cognitive load during driving such that driving tasks can be individualized for optimal skill learning. Individualization of ASD intervention is an important criterion due to the spectrum nature of the disorder. Twenty adolescents with ASD participated in our study and the data collected were used for systematic feature extraction and classification of cognitive loads based on five well-known machine learning methods. Subsequently, three information fusion schemes—feature level fusion, decision level fusion and hybrid level fusion—were explored. Results indicate that multimodal information fusion can be used to measure cognitive load with high accuracy. Such a mechanism is essential since it will allow individualization of driving skill training based on cognitive load, which will facilitate acceptance of this driving system for clinical use and eventual commercialization.