Understanding social interaction from various human behaviors is a complex task. Hyperscanning research tackles this challenge by delving into behavioral mechanisms through a neuroscience lens. While traditional studies focus on inter-brain synchrony in paired functional brain networks, they often lack methods for measuring interactions at the group level. In this study, we propose a multilayer network approach to estimate group brain synchrony and gain deeper insights into the brain's intricate organization. By utilizing the Prisoner's Dilemma Game, our goal is to find group interaction processes through distinct behaviors such as cooperation and defection. Thus, the inter-brain synchrony along with differences in network connectivity and structural properties within the functional group network were statistically analyzed between cooperation and defection.

Social interaction plays a crucial role in human societies, encompassing complex dynamics among individuals. To understand social interaction at the neural level, researchers have utilized hyperscanning in several social settings. These studies have mainly focused on inter-brain synchrony and the efficiency of paired functional brain networks, examining group interactions in dyads. However, this approach may not fully capture the complexity of multiple interactions, potentially leading to gaps in understanding inter-network differences. To overcome this limitation, the present study aims to bridge this gap by introducing methodological enhancements using the multilayer network approach, which is tailored to extract features from multiple networks. We applied this strategy to analyze the triad condition during social behavior processes to identify group interaction indices. Additionally, to validate our methodology, we compared the multilayer networks of triad conditions with group synchrony to paired conditions without group synchrony, focusing on statistical differences between alpha and beta waves. Correlation analysis between inter-brain and group networks revealed that this methodology accurately reflects the characteristics of actual behavioral synchrony. The findings of our study suggest that measures of paired brain synchrony and group interaction may exhibit distinct trends, offering valuable insights into interpreting group synchrony.

Ergonomic suitability is critical in tasks involving musculoskeletal movement. Many industries have examined best practices and assessed workers' ergonomic conditions during physical tasks. Prolonged awkward postures are a known cause of discomfort and restricted mobility in areas such as the arms, spine, and neck. Technologies like computer vision and human activity recognition can help identify and prioritize ergonomic improvements. This paper presents findings from a two-decade scoping review on the role of automation and study design in ergonomic assessments of physical workplace tasks. Articles were sourced from Scopus, PubMed, IEEE Xplore, Engineering Village, and Google Scholar. Eighty-four studies were analyzed to evaluate the use of technologies in data collection, experimentation, analysis, and validation. We examined how participant variables (e.g. sample size, body part of interest) and validation accuracy impact study outcomes across domains. Integrating advanced technologies into ergonomic evaluations can enhance worker safety and productivity by supporting real-time, evidence-based decision-making.

Looming sounds are known to influence visual function in the brain, even as early as the primary visual cortex. However, despite evidence that looming sounds have a larger impact on cortical excitability than stationary sounds, the influence of varying looming strengths on visual ability remains unclear. Here, we aim to understand how these signals influence low-level visual function. Fourteen healthy undergraduate students participated. They were blindfolded and received transcranial magnetic stimulation (TMS) to the primary visual cortex following auditory stimulation with different strength looming sounds. Participants reported whether they perceived a phosphene, or an illusory visual percept, following TMS stimulation. We hypothesized that rates of phosphene activity would increase with increasing levels of looming strength. A linear mixed-effect model showed that phosphene activity was significantly higher at higher strength of looming (F(1, 69) = 5.33, p = .024) and at higher TMS pulse strength (F(1, 18) = 4.71, p = .043). However, there was also a significant interaction between looming strength and pulse strength (F(1, 69) = 4.33, p = .041). At lower levels of TMS strength, phosphene rate increased with looming strength, while at higher levels of TMS strength the effect was reversed. These results suggest a complex relationship between looming strength and cortical activity, potentially reflecting the mixed contribution of total auditory energy and the rate of changes. This work will enhance our ability to predict audiovisual interactions and may help improve auditory warning systems designed to capture visual attention.

Artificial intelligence (AI) models can provide valuable predictions of remaining useful life (RUL), but it is difficult to understand how these ‘black box’ models come to their conclusions. Explainable AI (XAI) seeks ways to make the inner workings of these models more understandable, which in turn may assist in system development and adoption. Here we use SHAP analyses to visualize and interpret interactions between top performing parameters in a high-accuracy neural network model of RUL in a simulated fleet of NASA turbojet engines. Our analyses reveal complex relationships between even the best performing parameters and demonstrate a method for increasing explainability of an AI model by decomposing parameter relationships to better understand how parameter interactions influence model performance.

Cooking is a multitasking and rule-based task that can benefit from augmented reality (AR). This article introduces sAR Kitchen , an AR-based cooking assistant designed to incorporate spatial AR (sAR) into culinary training. We investigated the effects of instructions provided through our proposed sAR system compared to a monitor display featuring video tutorials in a task involving making playdough, which serves as a representation of a cooking task. We assessed perceived workload, usability, and performance measures (including task completion time, product quality, and cooking station messiness) through a user study involving 22 participants. We conducted a statistical comparison between the two methods to explore significant differences. In addition, we analyzed open-ended questions to provide further insights based on the participants' statements. The results indicated that sAR significantly reduced the perceived workload and improved the system's usability. Furthermore, the AR instructions for training proved to be as effective as conventional methods of instruction while also addressing some of the limitations associated with standard displays and enhancing the overall user experience. This study has the potential to pave the way for more efficient learning and training methods by incorporating sAR.

OCCUPATIONAL APPLICATIONSResults from our exploratory study of restaurant worker mental models of injury and safety emphasize the need for improved occupational safety in the culinary industry through targeted interventions for chefs and managers. The analysis we performed showed that managers possess more integrated and coherent mental models of injury and safety than chefs, reflected in network parameters showing better organization of safety concepts. Kitchen training programs should focus on bridging gaps in safety awareness and mitigating hazards such as burns, cuts, slips, and equipment-related risks. Given their stronger understanding of safety protocols, managers are well positioned to lead kitchen staff training. Leadership-focused training for managers is vital for improving communication, enforcing protocols, and leveraging their integrated mental models of injury and safety in daily practice. Continuous education tailored to the specific responsibilities of chefs and managers is essential for ensuring effective safety training and maintaining standards.

Safety is essential in the fast-paced and potentially hazardous environments of restaurant kitchens. This study explored the mental models that chefs and managers have regarding safety protocols. Pathfinder networking was employed to analyze participants’ perceptions of safety. The findings suggest a positive correlation between expertise and the quality of mental models: individuals with more skills or experience demonstrated superior mental models. The study also confirmed that greater skills or experience correlated with increased network coherence, similarity, and mental models. Notable differences in network coherence between chefs and managers were revealed, highlighting diverse expertise levels within these groups. The study underscores the critical role of comprehensive safety training and continual skill development in enhancing kitchen safety and efficiency. Understanding mental models and network coherence can enable restaurant workers to effectively manage complex tasks and uphold safety standards in dynamic environments.

In brain-computer interface (BCI), building accurate electroencephalogram (EEG) classifiers for specific mental tasks is critical for BCI performance. The classifiers are developed by machine learning (ML) and deep learning (DL) techniques, requiring a large dataset for training to build reliable and accurate models. However, collecting large enough EEG datasets is difficult due to intra-/inter-subject variabilities and experimental costs. This leads to the data scarcity problem, which causes overfitting issues to training samples, resulting in reducing generalization performance. To solve the EEG data scarcity problem and improve the performance of the EEG classifiers, we propose a novel EEG data augmentation (DA) framework using conditional generative adversarial networks (cGANs). An experimental study is implemented with two public EEG datasets, including motor imagery (MI) tasks (BCI competition IV IIa and III IVa), to validate the effectiveness of the proposed EEG DA method for the EEG classifiers. To evaluate the proposed cGAN-based DA method, we tested eight EEG classifiers for the experiment, including traditional MLs and state-of-the-art DLs with three existing EEG DA methods. Experimental results showed that most DA methods with proper DA proportion in the training dataset had higher classification performances than without DA. Moreover, applying the proposed DA method showed superior classification performance improvement than the other DA methods. This shows that the proposed method is a promising EEG DA method for enhancing the performances of the EEG classifiers in MI-based BCIs.

Multitasking is a common element in complex human-computer interactions and is known to impose deleterious mental workload demands. High mental workload is known to involve bilateral hemisphere activation, but the patterns of effective connectivity (directed causal influence or communication) among brain regions in such a context remain unclear. This study investigated the effect of mental workload on the causal influence brain regions exert over each other under a multitasking scenario. The Dynamic Causal Modeling (DCM) method was implemented to infer the flow of information and allocation of attentional resources. Thirty participants performed four subtasks with varying levels of workload on a computer-based multitasking program, simulating a pilot cockpit. Using eight brain regions commonly identified to be activated in multitasking conditions, nine candidate models were developed. Bayesian model averaging was then used to quantify the connectivity strengths among the brain regions. Linear regression was conducted to study the relationships between connection strengths and subtask performances. The results showed that the causal connections shifted from the left to both sides of the brain with increased workload. Linear regression analysis showed that the subtask performance could be predicted by connectivity strengths. Thus, by studying the brain dynamics of mental workload, we may be able to develop a predictor that supplements subjective self-report measures.

Studies have investigated electroencephalogram (EEG)-based emotion recognition using hand-crafted EEG features (e.g., differential entropy) or the annotated emotion categories without any additional emotion factors (e. g., context). The effectiveness of raw EEG-based emotion recognition remains for further investigation. In this study, we investigated the effectiveness of multi-task learning (MTL) for raw EEG-based convolutional neural networks (CNNs) in emotion recognition with auxiliary context information. Thirty subjects participated in this study, where their brain signals were collected when watching six types of emotion images (social/nonsocialfear, social/nonsocial-sad, and social/nonsocial-neutral). For the MTL architecture, we utilized temporal and spatial filtering layers from raw EEG-based CNNs as shared and task-specific layers for emotion and context classification tasks. Subject-dependent classifications and five repeated five-fold cross-validation were performed to test the classification accuracy for all comparison models. Our results showed that (1) the MTL classifier had a significantly higher classification accuracy and improved the performance of the single-task learnings (STLs) for both emotion and context, and (2) the ShallowConvNet was the best network architecture among the considered CNNs for the MTL with statistically significant improvement to the raw EEG-based STLs. This shows that the MTL can be a promising method for emotion recognition in utilizing the raw EEG-based CNN classifiers and emphasizes the importance of considering context information.

The demand for public datasets has increased as data-driven methodologies have been introduced in the field of brain-computer interfaces (BCIs). Indeed, many BCI datasets are available in various platforms or repositories on the web, and the studies that have employed these datasets appear to be increasing. Motor imagery is one of the significant control paradigms in the BCI field, and many datasets related to motor tasks are open to the public already. However, to the best of our knowledge, these studies have yet to investigate and evaluate the datasets, although data quality is essential for reliable results and the design of subject− or system-independent BCIs. In this study, we conducted a thorough investigation of motor imagery/execution EEG datasets recorded from healthy participants published over the past 13 years. The 25 datasets were collected from six repositories and subjected to a meta-analysis. In particular, we reviewed the specifications of the recording settings and experimental design, and evaluated the data quality measured by classification accuracy from standard algorithms such as Common Spatial Pattern (CSP) and Linear Discriminant Analysis (LDA) for comparison and compatibility across the datasets. As a result, we found that various stimulation types, such as text, figure, or arrow, were used to instruct subjects what to imagine and the length of each trial also differed, ranging from 2.5 to 29 s with a mean of 9.8 s. Typically, each trial consisted of multiple sections: pre-rest (2.38 s), imagination ready (1.64 s), imagination (4.26 s, ranging from 1 to 10 s), the post-rest (3.38 s). In a meta-analysis of the total of 861 sessions from all datasets, the mean classification accuracy of the two-class (left-hand vs. right-hand motor imagery) problem was 66.53%, and the population of the BCI poor performers, those who are unable to reach proficiency in using a BCI system, was 36.27% according to the estimated accuracy distribution. Further, we analyzed the CSP features and found that each dataset forms a cluster, and some datasets overlap in the feature space, indicating a greater similarity among them. Finally, we checked the minimal essential information (continuous signals, event type/latency, and channel information) that should be included in the datasets for convenient use, and found that only 71% of the datasets met those criteria. Our attempts to evaluate and compare the public datasets are timely, and these results will contribute to understanding the dataset’s quality and recording settings as well as the use of using public datasets for future work on BCIs.

Brain-computer interface (BCI) has helped people by allowing them to control a computer or machine through brain activity without actual body movement. Despite this advantage, BCI cannot be used widely because some people cannot achieve controllable performance. To solve this problem, researchers have proposed stimulation methods to modulate relevant brain activity to improve BCI performance. However, multiple studies have reported mixed results following stimulation, and the comparative study of different stimulation modalities has been overlooked. Accordingly, this study was designed to compare vibrotactile stimulation and transcranial direct current stimulation's (tDCS) effects on brain activity modulation and motor imagery BCI performance among inefficient BCI users. We recruited 44 subjects and divided them into sham, vibrotactile stimulation, and tDCS groups, and low performers were selected from each stimulation group. We found that the latter's BCI performance in the vibrotactile stimulation group increased significantly by 9.13% (p < 0.01), and while the tDCS group subjects' performance increased by 5.13%, it was not significant. In contrast, sham group subjects showed no increased performance. In addition to BCI performance, pre-stimulus alpha band power and the phase locking values (PLVs) averaged over sensory motor areas showed significant increases in low performers following stimulation in the vibrotactile stimulation and tDCS groups, while sham stimulation group subjects and high performers showed no significant stimulation effects across all groups. Our findings suggest that stimulation effects may differ depending upon BCI efficiency, and inefficient BCI users have greater plasticity than efficient BCI users.

Domain experts utilize a decision-support system depending on an artificial intelligence (AI) algorithm. Likewise, researchers in brain-computer interface (BCI) have recently employed deep learning (DL) algorithms for decoding and analyzing neural signals. Despite its outstanding performance, the BCI technology with the DLs has pointed out that it has a potential problem of low transparency due to algorithmic complexity of the models. On this problem, explainable artificial intelligence (XAI) can be a solution to make an AI algorithm and its decisions more interpretable. However, the explanation from the XAI has been emphasized that it should be designed corresponding with the user's different expectations which are contextually variable. Thus, our study aims to propose an explanation interface for the BCI expert under Pragmatism structuralizing an explanation with scientific knowledge in a contrastive manner. For this work, we conduct a contextual design process with five BCI experts, specifically conducting a contextual inquiry and work modeling to extract design requirements from their expertise in their work environment; next, designing and evaluating an interactive prototype of the explanation interface. The results indicated that our prototype has the advantages of increasing contextual understanding and intuitive interface design. Yet, there were also challenges on the explanation for novice users without prior knowledge on the XAI and objective understanding of the AI model with enough interpretability. This study contributes to providing a theoretical framework based on Pragmatism and designing a user-centered XAI system for domain experts in a specific context.

This paper presents sAR Kitchen, a cooking assistant that aims to enhance culinary training through the use of spatial augmented reality (sAR). We conducted a user study with twenty-two participants to investigate the effects of instructions given by our proposed sAR system and a monitor display with video tutorials in a playdough-making task representing the cooking task. Our study measured participants' perceived workload, usability, and performance regarding task completion time, product quality, and cooking station messiness. The results showed that sAR significantly decreased the perceived workload and increased the system's usability compared to traditional video-based tutorials. Additionally, our open-ended questions revealed that participants found the sAR instructions more engaging and interactive. Overall, this study demonstrates the potential of sAR to provide more user-friendly learning methods in culinary training.

Recently, convolutional neural network (CNN)-based classification models have shown good performance for motor imagery (MI) brain-computer interfaces (BCI) using electroencephalogram (EEG) in end-to-end learning. Although a few explainable artificial intelligence (XAI) techniques have been developed, it is still challenging to interpret the CNN models for EEG-based BCI classification effectively. In this research, we propose 3D-EEGNet as a 3D CNN model to improve both the explainability and performance of MI EEG classification. The proposed approach exhibited better performances on two MI EEG datasets than the existing EEGNet, which uses a 2D input shape. The MI classification accuracies are improved around 1.8% and 6.1% point in average on the datasets, respectively. The permutation-based XAI method is first applied for the reliable explanation of the 3D-EEGNet. Next, to find a faster XAI method for spatio-temporal explanation, we design a novel technique based on the normalized discounted cumulative gain (NDCG) for selecting the best among a few saliency-based methods due to their higher time complexity than the permutation-based method. Among the saliency-based methods, DeepLIFT was selected because the NDCG scores indicated its results are the most similar to the permutation-based results. Finally, the fast spatio-temporal explanation using DeepLIFT provides deeper understanding for the classification results of the 3D-EEGNet and the important properties in the MI EEG experiments.

Looming sounds have been shown to influence visual perception, suggesting they may make effective auditory warning tones for use with visual tasks. To explore the use of looming sounds as warning signals and to determine whether the strength of a looming sound will impact its effectiveness, we tested five looming sounds with different amplitude increases as warning tones in a contrast sensitivity task where participants made judgements concerning the orientation of low-contrast sinusoidal gratings. Reaction time, accuracy, and contrast threshold were measured for each sound condition. Our results indicate that accuracy was higher and reaction time was faster when a sound was present compared to when no sound was present, and contrast threshold was significantly lowered in sound trials compared to silent trials. However, there was no difference in accuracy, reaction time, or contrast threshold by strength of looming. These results suggest that while auditory warning sounds do enhance performance on a basic visual task, the benefit was not unique to looming sounds. This experiment will help inform the design of warnings by providing insight into the underlying effects of looming sounds on visual performance.

Although the high performance of the convolutional neural networks (CNNs) for brain-computer interface (BCI) tasks based on raw electroencephalography (EEG) signals, the explanation of the prediction result remains challenging owing to their complex structure and numerous parameters. We propose a novel framework for explaining CNNs for EEG-based BCI tasks by using the perturbation-based influence scores. The method supports the interpretation of CNN classification for EEG signals at both the example-level and the feature-level. The experiments on the BCIC III-IVa dataset demonstrate that the proposed method is effective for not only the interpretation of the predictive models, but also for the improvement of the classification accuracy.

AbstractBackgroundLow‐intensity transcranial focused ultrasound (tFUS) has gained considerable attention as a promising noninvasive neuromodulatory technique for human brains. However, the complex morphology of the skull hinders scholars from precisely predicting the acoustic energy transmitted and the region of the brain impacted during the sonication. This is due to the fact that different ultrasound frequencies and skull morphology variations greatly affect wave propagation through the skull.PurposeAlthough the acoustic properties of human skull have been studied for tFUS applications, such as tumor ablation using a multielement phased array, there is no consensus about how to choose a single‐element focused ultrasound (FUS) transducer with a suitable frequency for neuromodulation. There are interests in exploring the magnitude and dimension of tFUS beam through human parietal bone for modulating specific brain lobes. Herein, we aim to investigate the wave propagation of tFUS on human skulls to understand and address the concerns above.MethodsBoth experimental measurements and numerical modeling were conducted to investigate the transmission efficiency and beam pattern of tFUS on five human skulls (C3 and C4 regions) using single‐element FUS transducers with six different frequencies (150–1500 kHz). The degassed skull was placed in a water tank, and a calibrated hydrophone was utilized to measure acoustic pressure past it. The cranial computed tomography scan data of each skull were obtained to derive a high‐resolution acoustic model (grid point spacing: 0.25 mm) in simulations. Meanwhile, we modified the power‐law exponent of acoustic attenuation coefficient to validate numerical modeling and enabled it to be served as a prediction tool, based on the experimental measurements.ResultsThe transmission efficiency and −6 dB beamwidth were evaluated and compared for various frequencies. An exponential decrease in transmission efficiency and a logarithmic decrease of −6 dB beamwidth with an increase in ultrasound frequency were observed. It is found that a >750 kHz ultrasound leads to a relatively lower tFUS transmission efficiency (<5%), whereas a <350 kHz ultrasound contributes to a relatively broader beamwidth (>5 mm). Based on these observations, we further analyzed the dependence of tFUS wave propagation on FUS transducer aperture size.ConclusionsWe successfully studied tFUS wave propagation through human skulls at different frequencies experimentally and numerically. The findings have important implications to predict tFUS wave propagation for ultrasound neuromodulation in clinical applications, and guide researchers to develop advanced ultrasound transducers as neural interfaces.

Past research has recognized culture and gender variation in the experience of emotion, yet this has not been examined on a level of effective connectivity. To determine culture and gender differences in effective connectivity during emotional experiences, we applied dynamic causal modeling (DCM) to electroencephalography (EEG) measures of brain activity obtained from Chinese and American participants while they watched emotion-evoking images. Relative to US participants, Chinese participants favored a model bearing a more integrated dorsolateral prefrontal cortex (dlPFC) during fear v. neutral experiences. Meanwhile, relative to males, females favored a model bearing a less integrated dlPFC during fear v. neutral experiences. A culture-gender interaction for winning models was also observed; only US participants showed an effect of gender, with US females favoring a model bearing a less integrated dlPFC compared to the other groups. These findings suggest that emotion and its neural correlates depend in part on the cultural background and gender of an individual. To our knowledge, this is also the first study to apply both DCM and EEG measures in examining culture-gender interaction and emotion.