Motor-Imagery based Brain-Computer Interfaces (BCIs) can provide alternative communication pathways to neurologically impaired patients. The combination of BCIs and Virtual Reality (VR) can provide induced illusions of movement to patients with low-level of motor control during motor rehabilitation tasks. Unfortunately, current BCI systems lack reliability and good performance levels in comparison with other types of computer interfaces. To date, there is little evidence on how BCI-based motor training needs to be designed for transferring rehabilitation improvements to real life. Based on our previous work, we showcase the development and assessment of NeuRow, a novel multiplatform immersive VR environment that makes use of multimodal stimulation through vision, sound and vibrotactile feedback and delivered through a VR Head Mounted Display. In addition, we integrated the Adaptive Performance Engine (APE), a statistical approach to optimize user control in a selfpaced BCI-VR paradigm. In this paper, we describe the development and pilot assessment of NeuRow as well as its integration and assessment with APE.
With the availability of miniaturized low cost sensors and the general availability and easy applicability of algorithms for activity recognition, we investigate how various sensors can be deployed in a harsh environment, the industrial shop-floor. We review related work and provide an in-depth review of our own experiences were sensors wer used to enable recognition of activity, task progress and also mental and cognitive states of assembly workers. The recognition process is based on stationary (RGBD cameras, stereo vision depth sensors) and wearable devices (IMUs, GSR, ECG, mobile eye tracker). We describe in detail the used sensors, the challenges of fusing the data from these various sources together in real-time and how to interpret that data semantically.
Motor-Imagery based Brain-Computer Interfaces (BCIs) can provide alternative communication pathways to neurologically impaired patients. The combination of BCIs and Virtual Reality (VR) can provide induced illusions of movement to patients with low-level of motor control during motor rehabilitation tasks. Unfortunately, current BCI systems lack reliability and good performance levels in comparison with other types of computer interfaces. To date, there is little evidence on how BCI-based motor training needs to be designed for transferring rehabilitation improvements to real life. Based on our previous work, we showcase the development and assessment of NeuRow, a novel multiplatform immersive VR environment that makes use of multimodal stimulation through vision, sound and vibrotactile feedback and delivered through a VR Head Mounted Display. In addition, we integrated the Adaptive Performance Engine (APE), a statistical approach to optimize user control in a self-paced BCI-VR paradigm. In this paper, we describe the development and pilot assessment of NeuRow as well as its integration and assessment with APE.
Hypertension and diabetes are chronic conditions that have a considerable prevalence in the elderly. It is estimated that both hypertensive patients and people with diagnosed diabetes double cost of normotensive individuals and those in the absence of diabetes, respectively. It is therefore important to pay attention to these chronic conditions, both from a health and economical point of view, especially in scenarios with budget limitations. Clinical identification of chronic patients can be performed by feeding data of the patient encounters with the healthcare system to population classification systems such as Clinical Risk Groups (CRGs). CRGs classify individuals in unique and excluding health status categories taking both demographic and clinical information during certain period of time. In this work, we characterize healthy and chronic hypertensive and diabetic population at different chronic statuses (CRG) according to gender, age, diagnoses and drugs. After this characterization, we propose to use a supervised machine learning approach, in particular Support Vector Machines, to construct a data-driven model identifying the patient health status. We conclude that drugs and diagnoses are quite informative to discriminate patients with hypertension and diabetes, achieving promising results with the use of data-driven models.
Deep learning (DL) has transformed the field of data analysis by dramatically improving the state of the art in various classification and prediction tasks. Especially in the area of computer vision and speech processing, DL has recently demonstrated better performance and generalisation properties, compared to classical machine learning approaches, which are based on the extraction of hand-crafted model-based features followed by classification. Hand gestures and speech constitute two of the most important modalities in human-to-human communication and man-machine interaction. In biomedical engineering, a lot of new work is directed towards electromyography-based gesture recognition. In this paper, we present a brief overview of DL methods for electromyography-based hand gesture recognition and then we select from literature a simple model based on Convolutional Neural Networks that we consider as the baseline model. The proposed modifications to the baseline model yield a 3% classification improvement. In the current paper, we concentrate on the explanatory analysis of this performance improvement. An ablation study identifies which modifications are the most important ones, and label smoothing is investigated to verify if the results can be improved by reducing a priori bias. The analysis helps in understanding the limitations of the model and exploring new ways to improve the performance.
Training military readiness can significantly reduce potentially avoidable mistakes in real life situations. Virtual Reality (VR) has been widely used to provide a controlled and immersive medium for training both trainees' physical and cognitive skills. Despite the tremendous advances in VR-based training for military personnel, the attention has been mainly paid on improving simulation's realism through hardware tools and enhancing graphics and data input paradigms, rather than augmenting the human-computer interaction. Biocybernetic adaptation is a technique from the physiological computing field that allows creating real-time modulations based on detected human states indicated by psychophysiological responses. Although very sophisticated, the creation of biocybernetic loops has been mainly confined to research laboratories and very complex and invasive setups. Moreover, the combination of VR applications and biocybernetic adaptation has rarely been pursued beyond exploratory experiments. The Biocyber Physical System (BioPhyS) for military training in VR constitutes the first fully integrated, distributed and replicable VR simulator that is biocybernetically modulated. BioPhyS uses neurophysiological and cardiovascular measurements recorded from wearable sensors to detect calmness and cognitive readiness states to create dynamic changes in a VR target shooting simulator. The design process, psychophysiological modeling, and biocybernetic loop technology integration are shown, describing a pilot study carried out with a group of non-military participants. We highlight the software elements used for the VR-biocybernetic integration, and the psychophysiological model created for the real-time system as well as the timeline used to develop the functional prototype. We conclude this paper with a set of guidelines for developing meaningful physiological adaptations in VR applications.
The opportunity to develop new natural user interfaces has come forward due to the recent development of inexpensive full body tracking sensors, which has made this technology accessible to millions of users. In this paper, we present a comparative study between two natural user interfaces, and a cardiorespiratory training exergame developed based on the study results. The focus was on studying interfaces that could easily be used by an elderly population for interaction with floor projection displays. One interface uses both feet position to control a cursor and feet distance to trigger activation. In the alternative interface, the cursor is controlled by forearm ray casting into the projection floor and interaction is activated by hand pose. These modes of interaction were tested with 19 elderly participants in a point-and-click and a drag-and-drop task using a between-subjects experimental design. The usability, perceived workload and performance indicators were measured for each interface. Results show a clear preference towards the feet-controlled interface and a marginally better performance for this method. The results from the study served as a guide to the design of a cardiorespiratory fitness exergame for the elderly. The game “Grape Stomping” uses ground projection and mapping to display real-size winery elements. These virtual elements are used to simulate, in a playful way, the process of grape maceration through repeated stomping. A playtest session with nine elderly users was completed and its insights are presented in addition to the description of the game.
In many different research areas it is important to understand human behavior, e.g., in robotic learning or human-computer interaction. To learn new robotic behavior from human demonstrations, human movements need to be recognized to select which sequences should be transferred to a robotic system and which are already available to the system and therefore do not need to be learned. In interaction tasks, the current state of a human can be used by the system to react to the human in an appropriate way. Thus, the behavior of the human needs to be analyzed. To apply the identification and recognition of human behavior in different applications, it is of high interest that the used methods work autonomously with minimum user interference. This paper focuses on the analysis of human manipulation behavior in tasks of different complexity while keeping manual efforts low. By identifying characteristic movement patterns in the movement, human behaviors are decomposed into elementary building blocks using a fully automatic segmentation algorithm. With a simple k-Nearest Neighbor classification these identified movement sequences are assigned to known movement classes. To evaluate the presented approach, pick-and-place, ball-throwing, and lever-pulling movements were recorded with a motion tracking system. It is shown that the proposed method outperforms the widely used Hidden Markov Model-based classification. Especially in case of a small number of labeled training examples, which considerably minimizes manual efforts, our approach still has a high accuracy. For simple lever-pulling movements already one training example per class sufficed to achieve a classification accuracy of above \(95\%\).
Research on adaptive systems based on human psychophysiological assessments has been growing rapidly over the last decade. One fundamental component of such a system is human state assessment, on which the adaptation depends, at least partly. This is critical as the confidence of operators in such system will be determined to some extent by the accuracy of the models responsible for the assessment. This chapter presents work carried out in order to better understand the relationship between bio-behavioral data, psychological state, and operational performance of the operators. Modeling physiological parameters and performance was performed through a manipulation of three factors. The first factor was the size of smoothing window for performance. The second factor was the performance decrement threshold for labelling functional and sub-functional states. Finally, the third factor was the mode of classification being either prospective or descriptive. We used two types of classifiers, a linear and a non-linear classifier, and compared performance. Insights emerging from this work support that the use of multiple sources of bio-behavioral data, combined in a non-linear fashion, increases the psychometric qualities of state classifiers. This suggests that if such systems are to be used in safety-critical systems, they should be implemented using a wide variety of sensors to increase classification accuracies of performance.
In the current digital era, cardiovascular activity analysis is becoming ubiquitous with the increasing availability of embedded systems, and, in some cases, these systems are even endowed with the capabilities of detecting health risk factors identified through electrocardiographic (ECG) markers. Aligned with this trend, and inspired by the results from our pilot study, we focus this research on exploring the potential of a low-cost and open source device for heart rhythm analysis, with emphasis on Atrial Fibrillation (AF). AF is the most common cardiac arrhythmia, defined as a complex cardiac disease which is highly correlated to stroke and heart failure, and its prevalence is especially high in elderly population. Given the importance and impact of such cardiovascular disease, we performed ECG acquisitions in a hospital setting to evaluate the potential of this device to identify heart rhythms. These ECG acquisitions were performed in 10 patients, accomplished simultaneously with the low-cost device and the gold standard devices used in the clinical routine of the hospital. The data collected from the low-cost device was analysed by the cardiologist specialized in rhythmology, along with the conventional ECG exam analysis, whom suggested 100% accuracy of the low-cost device in differentiating a sinus rhythm from AF. The results also present great accuracy in the detection of atypical cardiac events through ECG analysis, as well as a good agreement for the corroboration of the numerical data from the devices used for this study, as the RR intervals and QTc intervals values suggest.
This work aims at demonstrating that it is possible to detect emotions using a single EEG channel with an accuracy that is comparable to that obtained in studies carried out with devices that have a high number of channels. In this article the Neurosky Maindwave device, which only a single electrode at the FP1 position, the MatLab and the IBM SPSS Modeler were used to acquire, process and classify the signals respectively. It is remarkable the accuracy achieved in relation to the inexpensive hardware employed for the acquisition of the EEG signal. The result of this study allows us to determine when the brain response is more intense after undergoing the subject, in the experimentation, to the stimuli that generate those emotions. This let us decide which brain power bands are most significants and which moments are the most appropriate to carry out this detection of emotions.
Brain Computer Interface (BCI) on the basis of Electroencephalography (EEG) has gained prominence over the past decade, especially with the proliferation of cheap EEG devices including user-made equipment. The main shortcoming of EEG, particularly with this type of equipment, is that it is frequently contaminated by various artifacts. Moreover a number of researchers and end users are currently using off-the-shelf equipment as a “black box” approach without any qualitative testing. This exposed an evident necessity to validate the equipment’s suitability. In this paper we provide vital groundwork by identifying and categorizing artifacts using our specific low fidelity equipment. This work forms part of a wider project where we assess the viability of this equipment, primarily in the artifacts domain, as a precursor for further studies. Our promising results show that we were able to effectively identify and categorize the most commonly encountered artifacts with the aforementioned equipment.
Anxiety is a common health issue that can occur throughout one’s existence. In this paper we describe the process design and the evaluation of Inner Flower, a tangible biofeedback device aimed at reducing anxiety. Inner Flower is meant to be used during daily life to help users regulate their anxiety and improve their overall mental health. It serves as a breathing guide that is being adapted to users through heart rate measurements. The Inner Flower does not intrude into users’ routine, instead it acts as an ambient display that operates in the peripheral vision. The Inner Flower was evaluated through two studies. The first one took place in a replica apartment in order to assess its usability and how people would appropriate the device. The second study assessed the effect of the device when stressors were presented to participants. We show how the final design of the Inner Flower was praised by users and how it can reduce symptoms of stress when users focus their attention on it. We then draw future research directions to further assess the effect of an ambient display and determine the importance of the biofeedback component of the device.
Selecting a suitable set of features, which is able to represent the data to be processed while retaining the relevant distinctive information, is one of the most important issues in classification problems. While different features can be extracted from the raw data, only few of them are actually relevant and effective for the classification process. Since relevant features are often unknown a priori, many candidate features are usually introduced. This degrades both the speed and the predictive accuracy of the classifier due to the presence of redundancy in the feature candidate set. We propose a class of features for image classification based on the notion of irredundant bidimensional pair-patterns, and we present an algorithm for image classification based on their extraction. The devised technique scales well on parallel multi-core architectures, as witnessed by the experimental results that have been obtained exploiting a benchmark image dataset.
Multi-factor authentication presents a robust method to secure our private information, but typically requires multiple actions by the user resulting in a high cost to usability and limiting adoption. A usable system should also be unobtrusive and inconspicuous. We present and discuss a system with the potential to engage all three factors of authentication (inherence, knowledge, and possession) in a single step using an earpiece that implements brain-based authentication using electroencephalography (EEG). We demonstrate its potential by collecting EEG data using manufactured custom-fit earpieces with embedded electrodes and testing a variety of authentication scenarios. Across all participants’ best-performing “passthoughts”, we are able to achieve 0% false acceptance and 0.36% false rejection rates, for an overall accuracy of 99.82%, using one earpiece with three electrodes. Furthermore, we find no successful attempts simulating impersonation attacks. We also report on perspectives from our participants. Our results suggest that a relatively inexpensive system using a single electrode-laden earpiece could provide a discreet, convenient, and robust method for one-step multi-factor authentication.
