The key to facial expression recognition is to learn discriminative spatial-temporal representations that embed facial expression dynamics. Previous studies predominantly rely on pre-trained Convolutional Neural Networks (CNNs) to learn facial appearance representations, overlooking the relationships between facial regions. To address this issue, this paper presents an Appearance- and Relation-aware Parallel Graph attention fusion Network (ARPGNet) to learn mutually enhanced spatial-temporal representations of appearance and relation information. Specifically, we construct a facial region relation graph and leverage the graph attention mechanism to model the relationships between facial regions. The resulting relational representation sequences, along with CNN-based appearance representation sequences, are then fed into a parallel graph attention fusion module for mutual interaction and enhancement. This module simultaneously explores the complementarity between different representation sequences and the temporal dynamics within each sequence. Experimental results on three facial expression recognition datasets demonstrate that the proposed ARPGNet outperforms or is comparable to state-of-the-art methods.

Facial expression recognition rarely explores complex spatiotemporal dependencies among facial regions at different scales. This paper proposes a transformer-based three-layer hierarchical architecture that incorporates multi-scale spatiotemporal aggregation for dynamic facial expression recognition. The hierarchical structure consists of bottom-to-top layers, each comprising transformer encoders with local self-attention mechanisms. These encoders gradually expand their receptive fields through hierarchical spatiotemporal aggregation, enabling the modeling of spatiotemporal context dependencies among facial regions at different scales and across consecutive frames. Consequently, the bottom-to-top layers correspond to learning the fine-grained, coarse-grained, and global facial representations. To evaluate the performance of our proposed framework, we conducted extensive experiments on four public datasets. The comparison results demonstrate that our proposed framework outperforms the state-of-the-art, with accuracies of 79.09%, 62.19%, 64.85%, and 59.79% on the RML, eNTERFACE'05, RAVDESS, and AFEW datasets, respectively. Ablation experiments, statistical significance tests, and visualization analyses indicate that the proposed framework successfully learns emotional-salient facial representations.

Video emotion recognition aims to infer human emotional states from the audio, visual, and text modalities. Previous approaches are centered around designing sophisticated fusion mechanisms, but usually ignore the fact that text contains global semantic information, while speech and face video show more fine-grained temporal dynamics of emotion. From the perspective of cognitive sciences, the process of emotion expression, either through facial expression or speech, is implicitly regulated by high-level semantics. Inspired by this fact, we propose a multimodal interaction enhanced representation learning framework for emotion recognition from face video, where a semantic enhancement module is first designed to guide the audio/visual encoder using the semantic information from text, then the multimodal bottleneck Transformer is adopted to further reinforce the audio and visual representations by modeling the cross-modal dynamic interactions between the two feature sequences. Experimental results on two benchmark emotion databases indicate the superiority of our proposed method. With the semantic enhanced audio and visual features, it outperforms the state-of-the-art models which fuse the features or decisions from the audio, visual and text modalities.

Characterizing spatial information and modelling temporal dynamics of facial images are key challenges for dynamic facial expression recognition (FER). In this paper, we propose an end-to-end multi-scale multi-attention network (MSMA-Net) for dynamic FER. In our model, the spatio-temporal features are encoded at two scales, i.e. the entire face and local facial patches. For each scale, we adopt a 2D convolutional neural network (CNN) to capture frame-based spatial information, and a 3D CNN to depict the short-term dynamics in the temporal sequence. Moreover, we propose a multi-attention mechanism by considering both spatial and temporal attention models. The temporal attention is applied on the image sequence to highlight expressive frames within the whole sequence, and the spatial attention mechanism is applied at the patch level to learn salient facial features. Comprehensive experiments on publicly available datasets (Aff-Wild2, RML, and AFEW) show that the proposed MSMA-Net model automatically highlights salient expressive frames, within which salient facial features are learned, allowing better or very competitive results compared to state-of-the-art methods.

In the temporal process of expressing the emotions, some intervals embed more salient emotion information than others. In this paper, by introducing an attentive temporal pooling module into the deep neural network (DNN) architecture, we present a simple but effective speech emotion recognition (SER) framework, which is able to automatically highlight the emotionally salient segments while suppressing the influence of less relevant ones. For an input speech utterance, the extracted feature sequence of hand-crafted low-level descriptors (LLDs) are evenly split into several overlapping temporal segments, and the segment-level features are computed by performing functionals on the LLDs of each segment. These segment-level features are then input into a DNN model outputting the emotion probabilities as well as the more condensed representation of each segment. An attentive temporal pooling module, consisting of an auxiliary DNN and a Gaussian Mixture Model (GMM), is proposed to learn the emotional saliency weights of different temporal segments from the condensed representations, which are then assigned to the segment-level emotion probabilities for the final utterance-level prediction. Notably, the attentive temporal pooling module and the DNN architecture for feature abstraction can be jointly trained using only the utterance-level labels, while without any frame-level or segment-level supervisory information. Experimental results on the three public released emotion datasets RML, EMO-DB, and IEMOCAP show that the proposed framework obtains state-of-the-art performance on SER.

Mental health applications are increasingly interested in using audio-visual and physiological measurements to detect the emotional state of a person, where significant researches aim to detect episodic emotional state. The availability of wearable devices and advanced signals is attracting researchers to explore the detection of a continuous sequence of emotion categories, referred to as emotion stream, for understanding mental health. Currently, there are no established databases for experimenting with emotion streams. In this paper, we make two contributions. First, we collect a Multi-modal EMOtion Stream (MEMOS) database in the scenario of social games. Audio-video recordings of the players are made via mobile phones and aligned Electrocardiogram (ECG) signals are collected by wearable sensors. Totally 40 multi-modal sessions have been recorded, each lasting between 25 to 70 minutes. Emotional states with time boundaries are self-reported and annotated by the participants while watching the video recordings. Secondly, we propose a two-step emotional state detection framework to automatically determine the emotion categories with their time boundaries along the video recordings. Experiments on the MEMOS database provide the baseline result for temporal emotional state detection research, with average mean-average-precision (mAP) score as 8.109% on detecting the five emotions (happiness, sadness, anger, surprise, other negative emotions) in videos. It is higher than 5.47% where the emotions are detected by averaging the frame-level confidence scores (obtained by Face++ emotion recognition API) in the segments from a sliding window. We expect that this paper will introduce a novel research problem and provide a database for related research.

In this paper, we aim at the AVEC2019 State of Mind Sub-Challenge (SoMS), and propose a multimodal state of mind assessment framework, for valence and arousal, respectively. For valence, sentiment analysis is firstly performed on the English text obtained via German speech recognition and translation to classify the audio visual session into positive/negative narrative. Then each overlapping 60s segment of the session is input into an audio visual SoM assessment model trained for positive/negative narratives. The mean prediction of all the segments is adopted as the final prediction of the audio visual session. For arousal, the first step of positive/negative classification is not performed. For the audio-visual SoM assessment models, we propose to extract the functional features (Function) and VGGish based deep learning features (VGGish) from speech, and the abstract visual features based on convolutional neural network (CNN) from the baseline visual features. For each feature stream, a long short term memory (LSTM) model is trained to predict the valence/arousal values of a segment, and a support vector regression (SVR) model is adopted for the final decision fusion. Experiments on the USoM dataset show that the model with Function, baseline ResNet features and baseline VGG features obtains promising prediction results for valence, with concordance correlation coefficient (CCC) up to 0.531 on the test set, which is much higher than the baseline result 0.219.

In this paper, we present our system designed for the video emotion recognition task of the Multimodal Emotion Challenge (MEC 2017). Histogram of Oriented Gradients (HOG), face shape (SHAPE), and geometric (GEO) features are extracted from the detected face images as hand-crafted video features. A pre-trained VGG-Face model is fine-tuned with the face images and emotion labels from the training set of CHEAVD 2.0, the outputs of the penultimate fully-connected layer (FC6) and the last fully-connected layer (FC7) are adopted as Deep Convolutional Neural Network (DCNN) based features. For each video clip, the hand-crafted features and DCNN based features are input into corresponding hidden Markov models (HMMs, one for each emotion class), respectively, for the initial emotion recognitions. The output logarithm likelihood probabilities from the HMMs are then ranked, and the orders constitute an eight-dimensional feature vector as inputs to a Naive Bayes classifier for decision fusion. Experimental results on the CHEAVD 2.0 database show that the combination of FC6, GEO, SHAPE and HOG features obtains the highest macro average precisions (MAPs) on both the validation set (46.61%) and test set (43.88%), which are 12.51% and 22.18% higher than the baseline results, respectively.

This paper addresses multi-modal depression analysis. We propose a multi-modal fusion framework composed of deep convolutional neural network (DCNN) and deep neural network (DNN) models. Our framework considers audio, video and text streams. For each modality, handcrafted feature descriptors are input into a DCNN to learn high-level global features with compact dynamic information, then the learned features are fed to a DNN to predict the PHQ-8 scores. For multi-modal fusion, the estimated PHQ-8 scores from the three modalities are integrated in a DNN to obtain the final PHQ-8 score. Moreover, in this work, we propose new feature descriptors for text and video. For the text descriptors, we select the participant»s answers to the questions associated with psychoanalytic aspects of depression, such as sleep disorder, and make use of the Paragraph Vector (PV) to learn the distributed representations of these sentences. For the video descriptors, we propose a new global descriptor, the Histogram of Displacement Range (HDR), calculated directly from the facial landmarks to measure their displacements and speed. Experiments have been carried out on the AVEC2017 depression sub-challenge dataset. The obtained results show that the proposed depression recognition framework obtains very promising accuracy, with the root mean square error (RMSE) as 4.653, mean absolute error (MAE) as 3.980 on the development set, and RMSE as 5.974, MAE as 5.163 on the test set.

In this paper, we design a hybrid depression classification and depression estimation framework from audio, video and text descriptors. It contains three main components: 1) Deep Convolutional Neural Network (DCNN) and Deep Neural Network (DNN) based audio visual multi-modal depression recognition frameworks, trained with depressed and not-depressed participants, respectively; 2) Paragraph Vector (PV), Support Vector Machine (SVM) and Random Forest based depression classification framework from the interview transcripts; 3) A multivariate regression model fusing the audio visual PHQ-8 estimations from the depressed and not-depressed DCNN-DNN models, and the depression classification result from the text information. In the DCNN-DNN based depression estimation framework, audio/video feature descriptors are first input into a DCNN to learn high-level features, which are then fed to a DNN to predict the PHQ-8 score. Initial predictions from the two modalities are fused via a DNN model. In the PV-SVM and Random Forest based depression classification framework, we explore semantic-related text features using PV, as well as global text-features. Experiments have been carried out on the Distress Analysis Interview Corpus - Wizard of Oz (DAIC-WOZ) dataset for the Depression Sub-challenge at the 2017 Audio-Visual Emotion Challenge (AVEC), results show that the proposed depression recognition framework obtains very promising results, with root mean square error (RMSE) as 3.088, mean absolute error (MAE) as 2.477 on the development set, and RMSE as 5.400, MAE as 4.359 on the test set, which are all lower than the baseline results.

In this paper, we propose the deep neural network - switching Kalman filter (DNN-SKF) based frameworks for both single modal and multi-modal continuous affective dimension estimation. The DNN-SKF framework firstly models the complex nonlinear relationship between the input (audio, visual, or lexical) features and the affective dimensions via the non-recurrent DNN, then models the temporal dynamics embedded in the emotions via the segmental linear SKF. Affective dimension estimation experiments are carried out on the Audio Visual Emotion Challenge (AVEC2012) database. Single modal estimation results are compared to those from the Support Vector Regression (SVR) models and Bidirectional Long Short-Term Memory Recurrent Neural Network (BLSTM-RNN) models, results show that for all modalities, and for all affective dimensions except for arousal from the audio features, the DNN-SKFs outperform SVR and BLSTM-RNN models in estimating the affective dimensions. Multi-modal estimation results are compared with the state of the art results on the competition of AVEC2012. Results show that both on the development set and test set, the proposed DNN-SKF models obtain the best performance in estimating the affective dimensions. On the test set, with the audio visual features, the average Pearson correlation coefficient (COR) is improved to 0.326 from 0.226 of the linear regression method [1], while with the audio visual and lexical features, the COR is improved to 0.355 from 0.344 of the particle filter fusion method (SVR-PF) [2].

In this paper, we target the CCPR 2016 Multimodal Emotion Recognition Challenge (MEC 2016) which is based on the Chinese Natural Audio-Visual Emotion Database (CHEAVD) of movies and TV programs showing (nearly) spontaneous human emotions. Low level descriptors (LLDs) are proposed as audio features. As visual features, we propose using histogram of oriented gradients (HOG), local phase quantisation (LPQ), shape features and behavior-related features such as head pose and eye gaze. The visual features are post processed to delete or smooth the all-zero feature vector segments. Single modal emotion recognition is performed using fully connected hidden Markov models (HMMs). For multimodal emotion recognition, two schemes are proposed: in the first scheme the normalized probability vectors from the HMMs are input to a support vector machine (SVM) for final recognition. For the second scheme, the final emotion is estimated using audio or video features depending if the face has been detected on the full video. Moreover, to make full use of the labeled data and to overcome the problem of unbalanced data, we use the training set and validation set together to train the HMMs and SVMs with parameters optimized via cross-validation experiments. Experimental results on the test set show that the macro average precisions (MAPs) of audio, visual, and multi-modal emotion recognition reach 42.85 %, 54.24 %, and 53.90 %, respectively, which are much higher than the corresponding baseline results of 24.02 %, 34.28 %, and 30.63 %.

In this paper, a capability opening engine for Internet of Vehicles (IOV) PaaS is designed and implemented to solve the problems currently existing in IOV industry such as high threshold of development and industry barriers. Through the analysis of service capability opening technologies, Web Service is preferred to implement the capability opening interfaces. Partial typical interfaces are designed after the requirement analysis of capability opening interfaces. The static and dynamic structure of the system will be introduced as a key point. At last, the application in logistics location service verifies the feasibility of the system.

A greater of multimedia services are based on the IP network, such as VoIP and IPTV. During the vertical handover process between Mobile Satellite System (MSS) and UMTS, delay jitter, packet loss and disorder will happen to multimedia services. We used the method of extension of RTP and RTCP protocols, modification of terminal buffer size and RTP packet play time to make the service not disrupted during handover. Analysis results demonstrate that this scheme could achieve the seamless media stream handover between MSS and UMTS.