Recent advances in wearable healthcare sensor technology have triggered radical changes in various applications of physical medicine and rehabilitation. Many consumer wearable devices are capable of measuring pulse rate information using photo-plethysmography (PPG) techniques. However, sampling intervals of these wearable devices tend to be longer than traditional instruments in clinical applications and research environments. Such sampling properties cannot satisfy requirements on sensor data to achieve reliable signal processing and to conduct following-up analysis and evaluation. This paper aims to apply data interpolation to PPG data acquired with low sampling rates, so that traditional heart rate variability (HRV) methods like power spectrum analysis can be utilized for stress evaluation. The interpolated data are compared with signals obtained using a traditional electrocardiography (ECG) device.

This paper attempts to estimate muscle contraction force from surface electromyography (EMG) signals during biceps curl movements. A data-driven EMG-force estimation model has been developed in this study, based on deep convolutional neural networks (CNNs). Information of EMG signals from both time and frequency domains has been utilized as input data to two CNN branches of the estimation model. Raw time series of EMG signals are directly input into one CNN branch in order to extract time domain characteristics, while frequency information is fed into the other CNN branch. Then these two branches are summarized at a concatenation layer, which is followed by three full-connected layers to estimate force levels. End point force at hand, considering the dumbbell curl exercises, is measured in the experiments with a testing apparatus using a single point load cell. This force data is used as training data in order to train the deep CNN EMG-force model. Eight subjects participated in the data acquisition experiments. For validation, different structures of the CNN model are examined with an estimation index of coefficient of determination. Then cross-validation with an EMG-force database of eight subjects has been conducted to investigate the estimation performance of the proposed method.

This paper reports some experimental results of a preliminary study on development of a mental stress recognition method using a deep learning-based model. This model acquires time-varying characteristics in heart rate variability (HRV) data with a long short-term memory (LSTM) deep network. The method proposed in this paper attempts to directly input differential R-R intervals (RRI) time series into the LSTM network. Training and classification performance have been discussed with stress level recognition experiments, in which HRV data were obtained with mental stress levels induced in a virtual reality environment. Test configurations have examined with different number of classes, segmentation lengths, and datasets. Comparing with the experimental results of the authors' previous studies, it is found that using differential RRI can largely ease the convergence difficulty in LSTM training process.

Psychological stressors used in mental stress tests largely effect quality and outcomes of experimental studies. This paper aims to develop a novel mental stress induction method using autonomic responses in virtual reality (VR) environments. Recent progress in VR technology promises desirable experience of immersion in a virtual space. Mental stress tests in VR are easy to configure and more flexible in experimental procedure. These virtual tests can provide close/improved outcomes comparing with real tests that use actual apparatuses and configurations. VR media contents of a roller coaster is investigated as stress load in this study. Nine subjects took part in the experiments and the electrocardiogram (ECG) data are analyzed with three short-term HRV analysis methods.

Vision-based myoelectric prosthetic hand uses a camera integrated into its body for object detection and environment understanding, where the results provide necessary information for grasp planning. It is expected that the semi-automatic prosthesis control can be realized with this method. However, such a control method usually suffers from heavy computation due to the requirement of real-time image processing to keep up with the arm movements of the user. This paper presents a distributed control system that assigns heavy processing tasks to one or multiple processing nodes through the network, which greatly reduces the computation burdens of the processor embedded in the prosthetic hand. In this control scheme, the embedded system in the prosthetic hand is only used for gathering necessary data for grasp planning, while the processing nodes in the network are responsible for processing and managing the collected data. A test platform is built to verify the proposed control scheme. The test platform streams user electromyography (EMG) signals and images simultaneously to the GPU server. The GPU sever analyzes the received data and generates the corresponding motor commands in real time. A case study that uses a 3-DoF gripper to continuously grasp several objects is performed using this test platform.

Heart rate variability (HRV) is an important biomarker for monitoring physical and mental conditions of human beings when they are executing a task. This study attempts to achieve HRV analysis to investigate variation of user’s stress level during driving a powered wheelchair. In such cases, situations and environment issues during driving are temporary and always change dynamically. This means most of the traditional HRV analysis methods are not appropriate, since there is a requirement of relatively long HRV data for each evaluation, for example five minutes. In order to deal with this problem, this paper applies four short-term HRV analysis indices to evaluate user’s stress level during driving and compares evaluation performance of these indices. Wheelchair driving experiments have been conducted with two test courses, namely, a straight course and a crank course. Since the time durations of driving through these courses were relatively short, the evaluation window length of HRV data was set as 15 and 20 s, respectively. With an overlap of 10 s, the evaluation has been achieved with time resolution as 5 and 10 s.

Dynamic changes in autonomic stress responses may provide details on autonomic nervous system functions. Time-varying evaluation can be achieved with a sliding window, however, in order to learn dynamic changes, an evaluation method needs to not only conduct calculation with a short sliding step but also derive evaluation indices with a narrow window. Stress analysis using HRV data shorter than one minute is still a challenge in this field. This paper investigates a Poincar plot analysis method for stress evaluation based on short term heart rate variability (HRV) data. First a sliding window, with no overlap, is used to segment data in order to form Poincaré plots. Then a simple index, which corresponds to mean distance between two adjacent points in the plot, is calculated on each evaluation window. The window length is defined with time duration and four lengths are examined in this paper, namely, 15, 30, 45, and 60 s. Two mental stress induction experiments, mental arithmetic and Stroop color-word tests, are utilized to validate the proposed method.

This paper aims to evaluate stress levels of a user when he/she is riding on a powered wheelchair. The levels of stress are considered relating to riding comfort and can be utilized for wheelchair design. This study was conducted with a powered wheelchair that can be controlled with a joystick or user posture. In addition, two test courses were used in this study, namely, a straight course and a crank one. Since the time durations for driving through each course were relatively short, a heart rate variability (HRV) analysis based on Lorenz plot and a mean distance measure has been applied. Experiments were conducted with different conditions in order to induce variations in stress levels of users.

This paper proposes a novel prosthetic hand control method that incorporates spatial information of target objects obtained with a RGB-D sensor into a myoelectric control procedure. The RGB-D sensor provides not only two-dimensional (2D) color information but also depth information as spatial cues on target objects, and these pieces of information are used to classify objects in terms of shape features. The shape features are then used to determine an appropriate grasp strategy/motion for control of a prosthetic hand. This paper uses a two-channel image format for classification, which contains grayscale and depth information of objects, and the image data is classified with a deep convolutional neural network (DCNN). Compared with previous studies based only on 2D color images, it is expected that the spatial information would improve classification accuracy, and consequently better grasping decision and prosthetic control can be achieved. In this study, a dataset of image pairs, consisting of grayscale images and their corresponding depth images, has been created to validate the proposed method. This database includes images of simple three-dimensional (3D) solid objects from six categories, namely, triangular prism, triangular pyramid, quadrangular prism, rectangular pyramid, cone, and cylinder. Image classification experiments were conducted with this database. The experimental results indicate that spatial information possesses high potential in classifying shape features of objects.

Motion classification based on pattern recognition of surface EMG (sEMG) signals is a promising approach for prosthetic control. We present a pattern recognition model that combines long short-term memory (LSTM) network with multiplayer perceptron (MLP) for sEMG signals feature learning and classification. The LSTM network captures temporal dependencies of the sEMG signals while the MLP has no inherent temporal dynamics but focuses on the static characteristics. The combination of the two networks would learn a feature space that contains both the dynamic and static information of the sEMG signals, which helps to improve the motion classification accuracy. The architecture of the proposed network was optimized by investigating the appropriate width and depth of the neural network as well as the dropout to achieve the best classification results. The performance of the proposed pattern recognition model was evaluated using Ninapro database. The results show that the proposed model can produce better classification accuracy than most of the well-known recognition techniques.

In recent years, prosthetic hands for amputees have seen rapid progress while myoelectric prosthetic control is facing a big challenge. It is difficult to control a multi-DOF hand only with electromyographic (EMG) signals to achieve complicated motions and natural and effortless operations, especially in the scenario of practical applications. This study proposes a semi-automatic myoelectric control method combining EMG signals with a vision-based object classifier to control a prosthetic hand. Information of target objects, such as shape features, dimensions, and postures can be obtained from images and then utilized to generate control commands for motors, in conjunction with user's muscle activities via EMG signals. EMG patterns are recognized to represent user's intension of motions. In the meanwhile, EMG power levels are used to modulate motor speed in a proportional manner. Prosthetic hand control experiments have been conducted to verify the proposed method.

This paper proposes a novel control method which combines an EMG pattern classifier with a vision-based object classifier to control various motions of a prosthetic hand. A deep convolutional neural network is adopted for the object recognition, and the posture of the prosthetic hand is controlled based on the recognition result of the object. To verify the validity of the proposed control method, the experiment was executed with 25 target objects. 3000 images for each target object were collected during the approach phase of hand motion to the object. High recognition performance was confirmed with an accuracy over 80%, although the misclassification was observed at the early phase of the approach motion. These results revealed that the proposed method has high potential to control various motions of the prosthetic hand.

This paper proposes a simulation system that provides close to the real experience of wearing and operating a virtual hand prosthesis with the help of augmented reality (AR) technology. The system allows user wearing a virtual hand prosthesis in an AR scene and the created scene is visual feedback through a display. Meanwhile, the user can operate the virtual hand with his own electromyography (EMG) signals. The system aims at improving the sense of reality when the user tries to operate a virtual hand prosthesis. Efforts have been made in three aspects. We design a virtual hand 3D model which approximates to a real hand in terms of dimensions and kinematics and present an AR environment realization technique using the Microsoft Kinect. To anticipate the hand movement, an EMG-based motion discrimination method using recurrent neural network is also proposed. User evaluation shows that the system improves the realism in wearing and controlling a virtual hand. The system is developed as a prototype for performing task training or control evaluation for myoelectric hand prosthesis.

Mobile health (mHealth) devices, such as smart phones and wristband fitness watches, are capable of measuring heart rate data using the photoplethysmography technology. In recent years, these devices have been used to obtain healthcare information in people's everyday life. However, it is difficult to apply traditional spectral analysis methods for the mHealth heart rate data due to the limited sampling features of mHealth devices. Data of the mHealth devices are recorded with uneven and relatively long sampling intervals, constrained by their hardware issues, i.e., processing speed, memory quantity, etc. This paper attempts to develop a stress analysis method for heart rate data obtained with mHealth devices. The heart rate data are evaluated using Poincare plot. Stress analysis indices, which are based on complex correlation measures of time -varying characteristics in Poincare plots, are examined using stress induction experiments with nine subjects.

This paper proposes a prosthetic control method which incorporates a novel object classifier with a conventional EMG-based motion classifier. The proposed method uses not only color information but spatial information to reduce the misclassification in previous research. The depth images are created based on spatial information which is acquired by Kinect. The deep convolutional neural network is adopted for the object classification, and the posture of the prosthetic hand is controlled based on the classification result of the object. To verify the validity of the proposed control method, the experiments have been carried out with 6 target objects. The 300 images for each target object were acquired in various directions. Their shapes resemble each other in particular perspective. We trained the deep convolutional neural network using the hybrid images which involve gray scale and depth information. In the experiments, the depth information improved the learning performance with high classification accuracy. These results revealed that the proposed method has high potential to improve object classification ability.

Objective evaluation method for stress level analysis is important for disease prevention, health promotion, and improvement of quality of life. The past few years witnessed rapid progress in wearable health devices. Recently, many wearable devices are capable of measuring pulse rate (PR) data that reflect stress level. The rise of these wearable devices brings exciting possibilities of monitoring one's stress level and its variation in his/her everyday life. However, sampling characteristics of such wearable health devices, e.g., sampling frequency, are usually not comparable with those of instruments for clinic or research usage. For this reason, it is difficult to achieve stress evaluation by simply applying traditional methods, such as spectral analysis for heart rate variability, to the PR data measured with these wearable devices. This study aims to develop a novel evaluation index for stress analysis. The proposed method is based on Poincare plots of PR data, which may be measured using wearable health devices, and the proposed index is developed based on inter-point distance measure. Experiments with mental stress induction were conducted with seven subjects to validate the proposed stress evaluation index. In order to evaluate dynamic change of stress level, short-term, one-minute length of duration, PR data was investigated.

This paper attempts to develop a novel prosthetic control system based on an Internet of Things (IoT) paradigm. The proposed method is able to employ not only information from muscle activities of the user and status of a prosthetic hand but also a wide range of data obtained from objects and items in the environment. The sensor data can be static features, dynamic statuses, and even contextual information of the operation. Fusion of these sensor data composes a rich information foundation to support multi-DoF and dexterous prosthetic hands. It is expected that much more reliable reasoning and more autonomous control decision can be developed using an IoT-based control system. The proposed method is verified with a case study using objects with simple sensor units and a Myo armband for electromyographic (EMG) signals.

The aim of our study is to develop a pattern recognition method to classify simultaneous motions from surface electromyographic (EMG) signals, with which a multi-degrees of freedom (DOFs) myoelectric control interface can be realized. Wavelet transform has been adopted in order to extract time-frequency (TF) feature patterns from EMG signals. However, an essential problem is that the TF feature vectors usually occupy a high-dimension space, which may cause degradation in classification performance. To deal with this problem, this paper applied a sub-band feature reduction method to the wavelet-based TF features. This method divides the frequency bandwidth into several sub-bands, and defines feature patterns on each sub-band to replace the original feature vectors. Then, the dimension-reduced feature patterns are sent to a multi-layer perceptron (MLP) to conduct motion classification. Experiments were carried out with four subjects to verify the proposed method. In this study, eleven forearm motions, including seven single motions and four simultaneous motions, have been examined with EMG signals measured from six channels. In addition, the number of sub-bands was investigated. The results showed that, except cases using six sub-bands for frequency division, the average overall classification rates exceed 90%. Relatively high classification performance was achieved.