Objective.Effective smoothing of electroencephalogram (EEG) signals while maintaining the original signal's features is important in EEG signal analysis and brain-computer interface. This paper proposes a novel EEG signal-smoothing algorithm and its potential application in cognitive conflict (CC) processing.Approach.Instead of being processed in the time domain, the input signal is visualized in increasing line width, the representation frame of which is converted into a binary image. An effective thinning algorithm is employed to obtain a unit-width skeleton as the smoothed signal.Main results.Experimental results on data fitting have verified the effectiveness of the proposed approach on different levels of signal-to-noise (SNR) ratio, especially on high noise levels (SNR⩽5 dB), where our fitting error is only 86.4%-90.4% compared to that of its best counterpart. The potential application of the proposed algorithm in EEG-based CC processing is comprehensively evaluated in a classification and a visual inspection task. The employment of the proposed approach in pre-processing the input data has significantly boosted theF1score of state-of-the-art models by more than 1%. The robustness of our algorithm is also evaluated via a visual inspection task, where specific CC peaks, i.e. the prediction error negativity and error-related positive potential (Pe), can be easily observed at multiple line-width levels, while the insignificant ones are eliminated.Significance.These results demonstrated not only the advance of the proposed approach but also its impact on classification accuracy enhancement.

Smoothing filters are widely used in EEG signal processing for noise removal while preserving signals’ features. Inspired by our recent work on Upscale and Downscale Representation (UDR), this paper proposes a cascade arrangement of some effective image-processing techniques for signal filtering in the image domain. The UDR concept is to visualize EEG signals at an appropriate line width and convert it to a binary image. The smoothing process is then conducted by skeletonizing the signal object to a unit width and projecting it back to the time domain. Two successive UDRs could result in a better-smoothing performance, but their binary image conversion should be restricted. The process is computationally ineffective, especially at higher line width values. Cascaded Thinning UDR (CTUDR) is proposed, exploiting morphological operations to perform a two-stage upscale and downscale within one binary image representation. CTUDR is verified on a signal smoothing and classification task and compared with conventional techniques, such as the Moving Average, the Binomial, the Median, and the Savitzky Golay filters. Simulated EEG data with added white Gaussian noise is employed in the former, while cognitive conflict data obtained from a 3D object selection task is utilized in the latter. CTUDR outperforms its counterparts, scoring the best fitting error and correlation coefficient in signal smoothing while achieving the highest gain in Accuracy (0.7640%) and F-measure (0.7607%) when used as a smoothing filter for training data of EEGNet.

Objective. Brain-computer interface (BCI) technology is poised to play a prominent role in modern work environments, especially a collaborative environment where humans and machines work in close proximity, often with physical contact. In a physical human robot collaboration (pHRC), the robot performs complex motion sequences. Any unexpected robot behavior or faulty interaction might raise safety concerns. Error-related potentials, naturally generated by the brain when a human partner perceives an error, have been extensively employed in BCI as implicit human feedback to adapt robot behavior to facilitate a safe and intuitive interaction. However, the integration of BCI technology with error-related potential for robot control demands failure-free integration of highly uncertain electroencephalography (EEG) signals, particularly influenced by the physical and cognitive state of the user. As a higher workload on the user compromises their access to cognitive resources needed for error awareness, it is crucial to study how mental workload variations impact the error awareness as it might raise safety concerns in pHRC. In this study, we aim to study how cognitive workload affects the error awareness of a human user engaged in a pHRC. Approach. We designed a blasting task with an abrasive industrial robot and manipulated the mental workload with a secondary arithmetic task of varying difficulty. EEG data, perceived workload, task and physical performance were recorded from 24 participants moving the robot arm. The error condition was achieved by the unexpected stopping of the robot in 33% of trials. Main results. We observed a diminished amplitude for the prediction error negativity (PEN) and error positivity (Pe), indicating reduced error awareness with increasing mental workload. We further observed an increased frontal theta power and increasing trend in the central alpha and central beta power after the unexpected robot stopping compared to when the robot stopped correctly at the target. We also demonstrate that a popular convolution neural network model, EEGNet, could predict the amplitudes of PEN and Pe from the EEG data prior to the error. Significance. This prediction model could be instrumental in developing an online prediction model that could forewarn the system and operators of the diminished error awareness of the user, alluding to a potential safety breach in error-related potential-based BCI system for pHRC. Therefore, our work paves the way for embracing BCI technology in pHRC to optimally adapt the robot behavior for personalized user experience using real-time brain activity, enriching the quality of the interaction.

A recent development in deep learning techniques has attracted attention to the decoding and classification of electroencephalogram (EEG) signals. Despite several efforts to utilize different features in EEG signals, a significant research challenge is using time-dependent features in combination with local and global features. Several attempts have been made to remodel the deep learning convolution neural networks (CNNs) to capture time-dependency information. These features are usually either handcrafted features, such as power ratios, or splitting data into smaller-sized windows related to specific properties, such as a peak at 300 ms. However, these approaches partially solve the problem but simultaneously hinder CNNs' capability to learn from unknown information that might be present in the data. Other approaches, like recurrent neural networks, are very suitable for learning time-dependent information from EEG signals in the presence of unrelated sequential data. To solve this, we have proposed an encoding kernel (EnK), a novel time-encoding approach, which uniquely introduces time decomposition information during the vertical convolution operation in CNNs. The encoded information lets CNNs learn time-dependent features in addition to local and global features. We performed extensive experiments on several EEG data sets—physical human-robot collaborations, P300 visual-evoked potentials, motor imagery, movement-related cortical potentials, and the Dataset for Emotion Analysis Using Physiological Signals. The EnK outperforms the state of the art with an up to 6.5% reduction in mean squared error (MSE) and a 9.5% improvement in F1-scores compared to the average for all data sets together compared to base models. These results support our approach and show a high potential to improve the performance of physiological and non-physiological data. Moreover, the EnK can be applied to virtually any deep learning architecture with minimal effort.

Navigation is a coordinated and goal-oriented movement through the environment, in which vision is an integral part of acquiring spatial information. People with blindness and vision impairment rely on alternative senses to navigate in daily life. Auditory cues are critical in building a spatial representation and effective navigation. Our work investigates the neurophysiological response to explore how auditory cues can shape a person's spatial representation and memory of an environment. In our experiment, a person who is blind was presented with verbal and non-verbal cues, either at close-reaching distance or globally. Electroencephalography, walking trajectories and tactile drawing data were collected to evaluate spatial representation and memory formed by different auditory cues and the role of theta oscillations in such relationships. Our results indicated a consistent increase in theta power at turning and starting points, a higher cognitive load with verbal strategies, and enhanced path recollection with global strategies. These findings emphasise the role of theta as a key indicator of cognitive workload and spatial memory processing. The preliminary results contribute to a broader understanding of the underlying processing mechanism of auditory information processing to support spatial navigation.

Robots for physical Human-Robot Collaboration (pHRC) systems need to change their behavior and how they operate in consideration of several factors, such as the performance and intention of a human co-worker and the capabilities of different human-co-workers in collision avoidance and singularity of the robot operation. As the system's admittance becomes variable throughout the workspace, a potential solution is to tune the interaction forces and control the parameters based on the operator's requirements. To overcome this issue, we have demonstrated a novel closed-loop-neuroadaptive framework for pHRC. We have applied cognitive conflict information in a closed-loop manner, with the help of reinforcement learning, to adapt to robot strategy and compare this with open-loop settings. The experiment results show that the closed-loop-based neuroadaptive framework successfully reduces the level of cognitive conflict during pHRC, consequently increasing the smoothness and intuitiveness of human-robot collaboration. These results suggest the feasibility of a neuroadaptive approach for future pHRC control systems through electroencephalogram (EEG) signals.

Wearable smart glasses are an emerging technology gaining popularity in the assistive technologies industry. Smart glasses aids typically leverage computer vision and other sensory information to translate the wearer's surrounding into computer-synthesized speech. In this work, we explored the potential of a new technique known as "acoustic touch" to provide a wearable spatial audio solution for assisting people who are blind in finding objects. In contrast to traditional systems, this technique uses smart glasses to sonify objects into distinct sound auditory icons when the object enters the device's field of view. We developed a wearable Foveated Audio Device to study the efficacy and usability of using acoustic touch to search, memorize, and reach items. Our evaluation study involved 14 participants, 7 blind or low-visioned and 7 blindfolded sighted (as a control group) participants. We compared the wearable device to two idealized conditions, a verbal clock face description and a sequential audio presentation through external speakers. We found that the wearable device can effectively aid the recognition and reaching of an object. We also observed that the device does not significantly increase the user's cognitive workload. These promising results suggest that acoustic touch can provide a wearable and effective method of sensory augmentation.

Robots for physical human–robot collaboration (pHRC) often need to adapt their admittance and how they operate due to several factors. As the admittance of the system becomes variable throughout the workspace, it is not always straightforward for the operator to predict the robot’s behavior. Previous work demonstrated that cognitive conflicts can be detected during one-dimensional tasks. This work assesses whether cognitive conflicts can also be detected during two-dimensional tasks in pHRC and a classification problem is formulated. Different robot admittance profiles anticipating the stimulus translated into different levels of cognitive conflict. Several commonly used classification algorithms for EEG signals were evaluated to classify different levels of cognitive conflict. Results demonstrate that cognitive conflict level is lower when the admittance smoothly decreases before unexpected events when compared to conditions in which the admittance abruptly decreases before the stimulus. Among the classification algorithms, the convolutional neural network has shown the best results to classify different levels of cognitive conflict. Results suggest the feasibility of adaptive approaches for future pHRC control systems that close the loop on users through EEG signals. The detected human cognitive state can also be used to assess and improve the predictability of human–robot teams in various pHRC applications.

Smoothing filters are widely used in EEG signal processing for noise removal while preserving important features. Unlike common approaches in the time domain, a recent effective algorithm using the Upscale and Downscale Representation (UDR) technique has been introduced to process the signal in the image domain. The idea of UDR is to visualize the input with an appropriate line width, convert it to a binary image, and then smooth it by skeletonizing the signal object to a unit width and projecting it back to the time domain. We propose in this paper a cascaded UDR (CUDR) where the interested signal is filtered twice. CUDR's performance is verified on simulated data with added white Gaussian noise and compared with the cascaded arrangement of some conventional techniques. Experimental results have demonstrated the outperformance of CUDR in terms of the fitting error when dealing with noisy signals, especially at a low signal-to-noise ratio.

The study explores various scenarios for providing shorelining information via spatial-audio auditory sensory augmentation with a view to assisting people who are blind or have low-vision. Shorelining is a strategy often associated with the use of the "white cane" and generally refers to path-following approaches that rely on contours of structures within a built environment. The aim is to support clear, logical paths of travel. In one approach, auditory shorelining cues are provided via a sequence of two auditory earcons rendered spatially along the shoreline and at the participant’s left or right side. In another approach, the auditory cues are rendered spatially as "gate posts" in between which the participant should walk. The rendering of the auditory earcons follows the participant’s path of travel so that at any given time, there is only ever a sequence of two auditory earcons that provides local shorelining information. The auditory earcons are rendered via two methods: (i) using loudspeakers appropriately placed along the path of travel and (ii) using binaural rendering of virtual speakers via smart glasses. We compared the performance of spatial-audio sensory augmentation with non-spatialized spoken language instruction. Participants’ performance was measured in terms of task accuracy, time, and walking behavior. The cognitive workload was measured using a self-reporting questionnaire. Results indicate that sensory augmentation provided via spatial-audio earcons can provide faster and smoother navigational assistance and guidance compared with spoken verbal instructions. The results also suggest that current binaural rendering using simple augmented-reality tool sets is not as robust as real sound in providing navigational guidance. Lastly, the "gate post" shorelining approach seems more effective than the left and right shorelining technique.

In recent years, virtual reality (VR) has become a mature technology with the capability to deliver higher levels of immersion and presence to users. Besides, it has been widely adopted to design and develop a new range of immersive tools, applications, and experiences in health, training, entertainment, art, etc. On the other end, brain-computer interfaces (BCIs) have shown the capability to communicate and control using brain signals. Fusing the capabilities of VR and BCI has the potential to increase the overall communication bandwidth of interaction. Although multiple kinds of BCI paradigms can be used, each comes with different pros and cons. In this chapter, we aim to explore existing BCI techniques as well as introduce a lesser-known BCI paradigm for VR known as cognitive conflict. The fusion of VR and BCI opens opportunities and applications for clinical, training, entertainment, and evaluation purposes.

Correct detection of peaks in electroencephalogram (EEG) signals is of essence due to the significant correlation of those potentials with cognitive performance and disorders. This paper proposes a novel and non-parametric approach to detect prediction error negativity (PEN) in cognitive conflict processing. The PEN candidates are first located from the input signal via an adaptation of a recent effective method for local maxima extraction, processed in a multi-scale manner. The found candidates are then fused and ranked based on their shape and location-based features. False positives caused by candidates' magnitude are eliminated by rotating the sorted candidate list where the one with the second-best ranking score will be identified as PEN. The EEG data collected from a 3D object selection task have been used to verify the efficacy of the proposed approach. Compared with the state-of-the-art peak detection techniques, the proposed method shows an improvement of at least 2.67% in accuracy and 6.27% in sensitivity while requires only about 4 ms to process an epoch. The accuracy and computational efficiency of the proposed technique in the detection of PEN in cognitive conflict processing would lead to promising applications in performance improvement of brain-computer interfaces (BCIs).

Human beings' emotional states are strong yet delicate where any trigger of any size can cause it to reverse/worsen instantly into an uncontrollable state, such as a Panic attack. Panic Attacks are an episode of heightened negative emotions that causes difficulty in breathing, increased heartbeat and shaking to name a few. This study explores Virtual Reality (VR), an immersive technology, as a solution to manipulate emotional states from negative to provide an instant positive effect, instant being at 10 seconds for this study. By experimenting on nine participants who went through two negative and positive scenarios, their heart rate, as they changed from negative to positive scenario were analyzed. The findings proved to be fruitful as all participants experienced a decrease in heart rate, at least in one of the markers set at either 10 seconds or 1 minute. This study is an initial step with positive outcomes that VR does have the potential to manipulate and influence emotional states from negative to positive instantly.

This work explores an auditory sensory augmentation paradigm we call acoustic touch, to assist people who are blind with reaching for close objects. The sensory augmentation system is constructed based on the Nreal augmented-reality glasses using a custom application running on an android phone. The system recognizes and localizes objects visually using cameras in the glasses, then renders objects as sound within a limited field-of-view, so we shall refer to the glasses as a foveated audio device. The repetition of the sound varies depending on the location of the object within the field of view of the foveated audio device. Psychophysical tests of the spatial perception of multiple objects are conducted comparing the acoustic touch paradigm with two other conditions: (1) a verbal clock face description of object locations and (2) a sequential audio presentation of the objects using Bluetooth speakers located with the objects. We report on the results of the psychophysical study with blind and blindfolded sighted participants.

Although beacon- and map-based spatial strategies are the default strategies for navigation activities, today’s navigational aids mostly follow a beacon-based design where one is provided with turn-by-turn instructions. Recent research, however, shows that our reliance on these navigational aids is causing a decline in our spatial skills. We are processing less of our surrounding environment and relying too heavily on the instructions given. To reverse this decline, we need to engage more in map-based learning, which encourages the user to process and integrate spatial knowledge into a cognitive map built to benefit flexible and independent spatial navigation behaviour. In an attempt to curb our loss of skills, we proposed a navigation assistant to support map-based learning during active navigation. Called the virtual global landmark (VGL) system, this augmented reality (AR) system is based on the kinds of techniques used in traditional orienteering. Specifically, a notable landmark is always present in the user’s sight, allowing the user to continuously compute where they are in relation to that specific location. The efficacy of the unit as a navigational aid was tested in an experiment with 27 students from the University of Technology Sydney via a comparison of brain dynamics and behaviour. From an analysis of behaviour and event-related spectral perturbation, we found that participants were encouraged to process more spatial information with a map-based strategy where a silhouette of the compass-like landmark was perpetually in view. As a result of this technique, they consistently navigated with greater efficiency and better accuracy.

Modern work environments have extensive interactions with technology and greater cognitive complexity of the tasks, which results in human operators experiencing increased mental workload. Air traffic control operators routinely work in such complex environments, and we designed tracking and collision prediction tasks to emulate their elementary tasks. The physiological response to the workload variations in these tasks was elucidated to untangle the impact of workload variations experienced by operators. Electroencephalogram (EEG), eye activity, and heart rate variability (HRV) data were recorded from 24 participants performing tracking and collision prediction tasks with three levels of difficulty. Our findings indicate that variations in task load in both these tasks are sensitively reflected in EEG, eye activity and HRV data. Multiple regression results also show that operators' performance in both tasks can be predicted using the corresponding EEG, eye activity and HRV data. The results also demonstrate that the brain dynamics during each of these tasks can be estimated from the corresponding eye activity, HRV and performance data. Furthermore, the markedly distinct neurometrics of workload variations in the tracking and collision prediction tasks indicate that neurometrics can provide insights on the type of mental workload. These findings have applicability to the design of future mental workload adaptive systems that integrate neurometrics in deciding not just "when" but also "what" to adapt. Our study provides compelling evidence in the viability of developing intelligent closed-loop mental workload adaptive systems that ensure efficiency and safety in complex work environments.

Analyzing the effects landmarks have on spatial learning is an active area of research in the study of human navigation processes and one that is key to understanding the links between human brain dynamics, landmark encoding, and spatial learning outcomes. This article presents a study on whether electroencephalography (EEG) signals related to virtual global landmarks combined with deep learning can be used to predict the accuracy and efficacy of spatial learning. Virtual global landmarks are silhouettes of actual landmarks projected into the navigator's vision via a heads-up display. They serve as a notable frame of reference in addition to the local landmarks we all typically use for route navigation. From a mobile virtual reality scenario involving 55 participants, the results of the study suggest that the EEG data associated with those who were exposed to global landmarks shows a visibly better capacity for predicting the quality of spatial learning levels than those who were not. As such, the EEG features associated with processing VGLs have a greater functional relation to the quality of spatial learning. This finding opens up a future direction of enquiry into landmark encoding and navigational ability. It may also provide a potential avenue for the early diagnosis of Alzheimer's disease.

In the above article [1], we detected an error in reporting the 10-fold cross-validation result. The correct 10-fold cross-validation result in Table IV is uploaded in this letter.