So far, the current methods in the clinical application do not facilitate continuous monitoring for pain and are unreliable, especially for vulnerable patients. In contrast, several automated methods have been proposed for this task by using facial features that were extracted independently from every frame of a given sequence. However, the obtained results were poor due to the failure to represent movement dynamics. To solve this problem, this work introduces three distinct methods regarding classification to monitor continuous pain intensity: (1) A Random Forest classifier (RFc) baseline method, (2) Long-Short Term Memory (LSTM) method, and (3) LSTM using sample weighting method (LSTM-SW). In this study, we conducted experiments with 11 datasets regarding classification, then compared results to regression results in Othman et al. (2021). Experimental results showed that the LSTM & LSTM-SW methods for continuous automatic pain intensity recognition performed better than guessing and RFc except with small datasets such as the reduced tonic datasets.

The selection of a suitable region of interest (ROI) is of great importance in camera-based vital signs estimation, as it represents the first step in the processing pipeline. Since all further processing relies on the quality of the signal extracted from the ROI, the tracking of this area is decisive for the performance of the overall algorithm. To overcome the limitations of classical approaches for the ROI, such as partial occlusions or illumination variations, a custom neural network for pixel-precise face segmentation called FaSeNet was developed. It achieves better segmentation results on two datasets compared to state-of-the-art architectures while maintaining high execution efficiency. Furthermore, the Matthews Correlation Coefficient was proposed as a loss function providing a better fitting of the network weights than commonly applied losses in the field of multi-class segmentation. In an extensive evaluation with a variety of algorithms for vital signs estimation, our FaSeNet was able to achieve better results in both heart and respiratory rate estimation. Thus, a ROI for vital signs estimation could be created that is superior to other approaches.

This study focuses on improving healthcare quality by introducing an automated system that continuously monitors patient pain intensity. The system analyzes the Electrodermal Activity (EDA) sensor modality modality, compares the results obtained from both EDA and facial expressions modalities, and late fuses EDA and facial expressions modalities. This work extends our previous studies of pain intensity monitoring via an expanded analysis of the two informative methods. The EDA sensor modality and facial expression analysis play a prominent role in pain recognition; the extracted features reflect the patient's responses to different pain levels. Three different approaches were applied: Random Forest (RF) baseline methods, Long-Short Term Memory Network (LSTM), and LSTM with the sample-weighting method (LSTM-SW). Evaluation metrics included Micro average F1-score for classification and Mean Squared Error (MSE) and intraclass correlation coefficient (ICC [3, 1]) for both classification and regression. The results highlight the effectiveness of late fusion for EDA and facial expressions, particularly in almost balanced datasets (Micro average F1-score around 61%, ICC about 0.35). EDA regression models, particularly LSTM and LSTM-SW, showed superiority in imbalanced datasets and outperformed guessing (where the majority of votes indicate no pain) and baseline methods (RF indicates Random Forest classifier (RFc) and Random Forest regression (RFr)). In conclusion, by integrating both modalities or utilizing EDA, they can provide medical centers with reliable and valuable insights into patients' pain experiences and responses.

Pain is a reliable indicator of health issues; it affects patients’ quality of life when not well managed. The current methods in the clinical application undergo biases and errors; moreover, such methods do not facilitate continuous pain monitoring. For this purpose, the recent methodologies in automatic pain assessment were introduced, which demonstrated the possibility for objectively and robustly measuring and monitoring pain when using behavioral cues and physiological signals. This paper focuses on introducing a reliable automatic system for continuous monitoring of pain intensity by analyzing behavioral cues, such as facial expressions and audio, and physiological signals, such as electrocardiogram (ECG), electromyogram (EMG), and electrodermal activity (EDA) from the X-ITE Pain Dataset. Several experiments were conducted with 11 datasets regarding classification and regression; these datasets were obtained from the database to reduce the impact of the imbalanced database problem. With each single modality (Uni-modality) experiment, we used a Random Forest [RF] baseline method, a Long Short-Term Memory (LSTM) method, and a LSTM using a sample weighting method (called LSTM-SW). Further, LSTM and LSTM-SW were used with fused modalities (two modalities = Bi-modality and all modalities = Multi-modality) experiments. Sample weighting was used to downweight misclassified samples during training to improve the performance. The experiments’ results confirmed that regression is better than classification with imbalanced datasets, EDA is the best single modality, and fused modalities improved the performance significantly over the single modality in 10 out of 11 datasets.

Pain is a complex phenomenon, involving sensory and emotional experience, that is often poorly understood, especially in infants, anesthetized patients, and others who cannot speak. Technology supporting pain assessment has the potential to help reduce suffering; however, advances are needed before it can be adopted clinically. This survey paper assesses the state of the art and provides guidance for researchers to help make such advances. First, we overview pain’s biological mechanisms, physiological and behavioral responses, emotional components, as well as assessment methods commonly used in the clinic. Next, we discuss the challenges hampering the development and validation of pain recognition technology, and we survey existing datasets together with evaluation methods. We then present an overview of all automated pain recognition publications indexed in the Web of Science as well as from the proceedings of the major conferences on biomedical informatics and artificial intelligence, to provide understanding of the current advances that have been made. We highlight progress in both non-contact and contact-based approaches, tools using face, voice, physiology, and multi-modal information, the importance of context, and discuss challenges that exist, including identification of ground truth. Finally, we identify underexplored areas such as chronic pain and connections to treatments, and describe promising opportunities for continued advances.

Face and person detection are important tasks in computer vision, as they represent the first component in many recognition systems, such as face recognition, facial expression analysis, body pose estimation, face attribute detection, or human action recognition. Thereby, their detection rate and runtime are crucial for the performance of the overall system. In this paper, we combine both face and person detection in one framework with the goal of reaching a detection performance that is competitive to the state of the art of lightweight object-specific networks while maintaining real-time processing speed for both detection tasks together. In order to combine face and person detection in one network, we applied multi-task learning. The difficulty lies in the fact that no datasets are available that contain both face as well as person annotations. Since we did not have the resources to manually annotate the datasets, as it is very time-consuming and automatic generation of ground truths results in annotations of poor quality, we solve this issue algorithmically by applying a special training procedure and network architecture without the need of creating new labels. Our newly developed method called Simultaneous Face and Person Detection (SFPD) is able to detect persons and faces with 40 frames per second. Because of this good trade-off between detection performance and inference time, SFPD represents a useful and valuable real-time framework especially for a multitude of real-world applications such as, e.g., human–robot interaction.

Vision-based 3D human pose estimation approaches are typically evaluated on datasets that are limited in diversity regarding many factors, e.g., subjects, poses, cameras, and lighting. However, for real-life applications, it would be desirable to create systems that work under arbitrary conditions ("in-the-wild"). To advance towards this goal, we investigated the commonly used datasets HumanEva-I, Human3.6M, and Panoptic Studio, discussed their biases (that is, their limitations in diversity), and illustrated them in cross-database experiments (for which we used a surrogate for roughly estimating in-the-wild performance). For this purpose, we first harmonized the differing skeleton joint definitions of the datasets, reducing the biases and systematic test errors in cross-database experiments. We further proposed a scale normalization method that significantly improved generalization across camera viewpoints, subjects, and datasets. In additional experiments, we investigated the effect of using more or less cameras, training with multiple datasets, applying a proposed anatomy-based pose validation step, and using OpenPose as the basis for the 3D pose estimation. The experimental results showed the usefulness of the joint harmonization, of the scale normalization, and of augmenting virtual cameras to significantly improve cross-database and in-database generalization. At the same time, the experiments showed that there were dataset biases that could not be compensated and call for new datasets covering more diversity. We discussed our results and promising directions for future work.

Automatic systems enable continuous monitoring of patients' pain intensity as shown in prior studies. Facial expression and physiological data such as electrodermal activity (EDA) are very informative for pain recognition. The features extracted from EDA indicate the stress and anxiety caused by different levels of pain. In this paper, we investigate using the EDA modality and fusing two modalities (frontal RGB video and EDA) for continuous pain intensity recognition with the X-ITE Pain Database. Further, we compare the performance of automated models before and after reducing the imbalance problem in heat and electrical pain datasets that include phasic (short) and tonic (long) stimuli. We use three distinct real-time methods: A Random Forest (RF) baseline methods [Random Forest classifier (RFc) and Random Forest regression (RFr)], Long-Short Term Memory Network (LSTM), and LSTM using sample weighting method (called LSTM-SW). Experimental results (1) report the first results of continuous pain intensity recognition using EDA data on the X-ITE Pain Database, (2) show that LSTM and LSTM-SW outperform guessing and baseline methods (RFc and RFr), (3) confirm that the electrodermal activity (EDA) with most models is the best, (4) show the fusion of the output of two LSTM models using facial expression and EDA data (called Decision Fusion = DF). The DF improves results further with some datasets (e.g. Heat Phasic Dataset (HTD)).

We address the problem of facial expression analysis. The proposed approach predicts both basic emotion and valence/arousal values as a continuous measure for the emotional state. Experimental results including cross-database evaluation on the AffectNet, Aff-Wild, and AFEW dataset shows that our approach predicts emotion categories and valence/arousal values with high accuracies and that the simultaneous learning of discrete categories and continuous values improves the prediction of both. In addition, we use our approach to measure the emotional states of users in an Human-Robot-Collaboration scenario (HRC), show how these emotional states are affected by multiple difficulties that arise for the test subjects, and examine how different feedback mechanisms counteract negative emotions users experience while interacting with a robot system.

Prior work on automated methods demonstrated that it is possible to recognize pain intensity from frontal faces in videos, while there is an assumption that humans are very adept at this task compared to machines. In this paper, we investigate whether such an assumption is correct by comparing the results achieved by two human observers with the results achieved by a Random Forest classifier (RFc) baseline model (called RFc-BL) and by three proposed automated models. The first proposed model is a Random Forest classifying descriptors of Action Unit (AU) time series; the second is a modified MobileNetV2 CNN classifying face images that combine three points in time; and the third is a custom deep network combining two CNN branches using the same input as for MobileNetV2 plus knowledge of the RFc. We conduct experiments with X-ITE phasic pain database, which comprises videotaped responses to heat and electrical pain stimuli, each of three intensities. Distinguishing these six stimulation types plus no stimulation was the main 7-class classification task for the human observers and automated approaches. Further, we conducted reduced 5-class and 3-class classification experiments, applied Multi-task learning, and a newly suggested sample weighting method. Experimental results show that the pain assessments of the human observers are significantly better than guessing and perform better than the automatic baseline approach (RFc-BL) by about 1%; however, the human performance is quite poor due to the challenge that pain that is ethically allowed to be induced in experimental studies often does not show up in facial reaction. We discovered that downweighting those samples during training improves the performance for all samples. The proposed RFc and two-CNNs models (using the proposed sample weighting) significantly outperformed the human observer by about 6% and 7%, respectively.

Background In patients with limited communication skills, the use of conventional scales or external assessment is only possible to a limited extent or not at all. Multimodal pain recognition based on artificial intelligence (AI) algorithms could be a solution. Objective Overview of the methods of automated multimodal pain measurement and their recognition rates that were calculated with AI algorithms. Methods In April 2018, 101 studies on automated pain recognition were found in the Web of Science database to illustrate the current state of research. A selective literature review with special consideration of recognition rates of automated multimodal pain measurement yielded 14 studies, which are the focus of this review. Results The variance in recognition rates was 52.9-55.0% (pain threshold) and 66.8-85.7%; in nine studies the recognition rate was >= 80% (pain tolerance), while one study reported recognition rates of 79.3% (pain threshold) and 90.9% (pain tolerance). Conclusion Pain is generally recorded multimodally, based on external observation scales. With regard to automated pain recognition and on the basis of the 14 selected studies, there is to date no conclusive evidence that multimodal automated pain recognition is superior to unimodal pain recognition. In the clinical context, multimodal pain recognition could be advantageous, because this approach is more flexible. In the case of one modality not being available, e.g., electrodermal activity in hand burns, the algorithm could use other modalities (video) and thus compensate for missing information.

Bei Patienten mit eingeschränkten Kommunikationsfähigkeiten ist der Einsatz von konventionellen Skalen bzw. Fremdbeurteilung nur bedingt oder nicht möglich, eine multimodale Schmerzerkennung basierend auf Algorithmen der künstlichen Intelligenz (KI) könnte eine Lösung darstellen. Überblick über Methoden der automatisierten multimodalen Schmerzmessung und deren Erkennungsraten, welche mit Algorithmen der KI berechnet wurden. Zur Darstellung des aktuellen Forschungsstands wurden mittels selektiver Literaturrecherche im April 2018 in der Datenbank Web of Science 101 Studien zur automatisierten Schmerzerkennung identifiziert. 14 dieser Studien berichten speziell Erkennungsraten der automatisierten multimodalen Schmerzmessung, diese werden als Fokus des vorliegenden narrativen Reviews dargestellt. Die Varianz der Erkennungsraten liegt bei 52,9–55,0 % (Schmerzschwelle) bzw. 66,8–85,7 % (Schmerztoleranz), bei 9 Studien ist die Erkennungsrate ≥80 % (Schmerztoleranz), eine Arbeit berichtet hohe Erkennungsraten von 79,3 % (Schmerzschwelle) und 90,9 % (Schmerztoleranz). Schmerzen werden, basierend auf Fremdbeobachtungsskalen, grundsätzlich multimodal erfasst. In Bezug auf die automatisierte Schmerzerkennung ist auf Basis der 14 ausgewählten Studien noch nicht abschließend beantwortet, ob eine multimodale Schmerzerkennung einer unimodalen überlegen ist. Im klinischen Kontext könnte eine multimodale Schmerzerkennung von Vorteil sein, da dieser Ansatz flexibler einsetzbar ist. Der Algorithmus könnte im Fall der fehlenden Verfügbarkeit einer Modalität, z. B. der elektrodermalen Aktivität bei Handverbrennungen, auf andere Modalitäten (Video) zurückgreifen und somit fehlende Informationen kompensieren.

In computer vision, occlusions are mainly known as a challenge to cope with. For instance, partial occlusions of the face may lower the performance of facial expression recognition systems. However, when incorporated into the training, occlusions can be also helpful in improving the overall performance. In this paper, we propose and evaluate occlusion augmentation as a simple but effective regularizing tool for improving the general performance of deep learning based facial expression and action unit recognition systems, even if no occlusion is present in the test data. In our experiments we consistently found significant performance improvements on three databases (Bosphorus, RAF-DB, and AffectNet) and three CNN architectures (Xception, MobileNet, and a custom model), suggesting that occlusion regularization works independently of the dataset and architecture. Based on our clear results, we strongly recommend to integrate occlusion regularization into the training of all CNN-based facial expression recognition systems, because it promises performance gains at very low cost.

Die objektive Erfassung subjektiv erlebter Schmerzen ist ein bislang unzureichend gelöstes Problem. In den letzten Jahren wurden Datensätze erstellt, um Algorithmen der künstlichen Intelligenz zu trainieren, welche Muster der Schmerzintensität erkennen. Die multimodale Erkennung von Schmerzen mit maschinellen Lernverfahren könnte eine Möglichkeit bieten, eine Über- bzw. Unterversorgung mit Analgetika zu verringern, explizit bei Patienten mit eingeschränkter Kommunikationsfähigkeit. In dieser Studie wurde die Methodik einer automatisierten multimodalen Erkennung von Schmerzintensität und -modalität mittels maschineller Lernverfahren der künstlichen Intelligenz untersucht. Im Einzelnen wurden multimodale Erkennungsraten von experimentell induzierten phasischen Elektro- und Hitzeschmerzreizen mit uni- und bimodalen Erkennungsraten verglichen. Basierend auf der X‑ITE Pain Database wurden gesunde Probanden mit phasischem elektroinduziertem Schmerz und Hitzeschmerz stimuliert und die Schmerzreaktionen mit multimodaler Sensorik (akustisch, videobasiert, physiologisch) erfasst. Nach komplexer Signalverarbeitung wurden mit maschinellen Lernverfahren Erkennungsraten der Schmerzintensität (Baseline vs. Schmerzschwelle, Schmerztoleranz, Mittelwert von Schmerzschwelle und -toleranz) und der Vergleich der Schmerzmodalität elektrisch vs. Hitze berechnet. Letztendlich erfolgte ein statistischer Vergleich uni- vs. multimodaler sowie bi- vs. multimodaler Erkennungsraten. Multimodale Erkennungsraten der Schmerzintensität sind überwiegend den unimodalen Erkennungsraten signifikant überlegen, unabhängig von der Schmerzmodalität. Die multimodale Erkennung der Schmerzmodalität unterscheidet signifikant besser zwischen hitze- und elektroinduziertem Schmerz. Multimodale Erkennungsraten sind des Weiteren überwiegend den bimodalen Erkennungsraten überlegen. Dem multimodalen Ansatz der Erkennung von Schmerzintensität und -modalität sollte im Vergleich zur Unimodalität Vorrang gegeben werden. Es ist in weiteren klinischen Studien zu klären, ob eine multimodale Erkennung von Schmerzintensität und -modalität einer bimodalen Erkennung tatsächlich überlegen ist.

We address the problem of emotional state detection from facial expressions. Our proposed approach simultaneously detects faces and predicts both discrete emotion categories and continuous valence/arousal values from raw input images. We train and evaluate our approach on 3 different datasets, compare our approach to other state-of-the-art approaches and perform a cross-database evaluation. In this way, we found, that our approach generalizes well and is suitable for real-time applications.

Automatic understanding of facial behavior is hampered by factors such as occlusion, illumination, non-frontal head pose, low image resolution, or limitations in labeled training data. The EmotioNet 2020 Challenge addresses these issues through a competition on recognizing facial action units on in-the-wild data. We propose to combine multi-task and self-training to make best use of the small manually / fully labeled and the large weakly / partially labeled training datasets provided by the challenge organizers. With our approach (and without using additional data) we achieve the second place in the 2020 challenge - with a performance gap of only 0.05% to the challenge winner and of 5.9% to the third place. On the 2018 challenge evaluation data our method outperforms all other known results.

So far, all studies investigating the facial expression of pain have validated methods on the same database, whereas the cross-database performance is less considered. This may be due to poor performance of well-trained models on other databases. In this paper, we propose two distinct methods to classify based on the temporal information. To explore the generalization capability of pain recognition models, we do cross-database validations on two benchmark pain databases: BioVid and X-ITE. We also experiment with combining both databases. Experimental results (1) show that our methods can be successfully used to classify pain (both methods perform similarly well), (2) demonstrate that the performance is robust by verifying them cross-database, and (3) present that the performance of pain assessment is improved with more data (combined-database).

In this paper, we propose two simple yet effective methods to estimate facial attributes in unconstrained images. We use a straight forward and fast face alignment technique for preprocessing and estimate the face attributes using MobileNetV2 and Nasnet-Mobile, two lightweight CNN (Convolutional Neural Network) architectures. Both architectures perform similarly well in terms of accuracy and speed. A comparison with state-of-the-art methods with respect to processing time and accuracy shows that our proposed approach perform faster than the best state-of-the-art model and better than the fastest state-of-the-art model. Moreover, our approach is easy to use and capable of being deployed on mobile devices.

Facial expression analysis is challenged by the numerous degrees of freedom regarding head pose, identity, illumination, occlusions, and the expressions itself. It currently seems hardly possible to densely cover this enormous space with data for training a universal well-performing expression recognition system. In this paper we address the sub-challenge of generalizing to head poses that were not seen in the training data, aiming at getting along with sparse coverage of the pose subspace. For this purpose we (1) propose a novel face normalization method called FaNC that massively reduces pose-induced image variance; (2) we compare the impact of the proposed and other normalization methods on (a) action unit intensity estimation with the FERA 2017 challenge data (achieving new state of the art) and (b) facial expression recognition with the Multi-PIE dataset; and (3) we discuss the head pose distribution needed to train a pose-invariant CNN-based recognition system. The proposed FaNC method normalizes pose and facial proportions while retaining expression information and runs in less than 2 ms. When comparing results achieved by training a CNN on the output images of FaNC and other normalization methods, FaNC generalizes significantly better than others to unseen poses if they deviate more than 20° from the poses available during training. Code and data are available.

Experimental economic laboratories run many studies to test theoretical predictions with actual human behaviour, including public goods games. With this experiment, participants in a group have the option to invest money in a public account or to keep it. All the invested money is multiplied and then evenly distributed. This structure incentivizes free riding, resulting in contributions to the public goods declining over time. Face-to-face Communication (FFC) diminishes free riding and thus positively affects contribution behaviour, but the question of how has remained mostly unknown. In this paper, we investigate two communication channels, aiming to explain what promotes cooperation and discourages free riding. Firstly, the facial expressions of the group in the 3-minute FFC videos are automatically analysed to predict the group behaviour towards the end of the game. The proposed automatic facial expressions analysis approach uses a new group activity descriptor and utilises random forest classification. Secondly, the contents of FFC are investigated by categorising strategy-relevant topics and using meta-data. The results show that it is possible to predict whether the group will fully contribute to the end of the games based on facial expression data from three minutes of FFC, but deeper understanding requires a larger dataset. Facial expression analysis and content analysis found that FFC and talking until the very end had a significant, positive effect on the contributions.