Previous research on fabric drape has not provided an objective and comprehensive characterization of drape characteristics. In light of this, we proposed an approach that utilizes a neural network-based framework for characterizing the umbrella drape of woven fabrics. Fabric drapes with the same macro-level mechanical characteristics can be categorized together, thereby establishing objective classification criteria. Our method involved feature extraction and classification from drape images/point clouds via neural networks, namely ResNet18 and the deep graph convolutional neural network (DGCNN). We assessed the effectiveness of both networks through supervised learning and selected the best candidate to distinguish/retrieve drape styles from unlabeled data. Moreover, a sketch down-sampling (SDS) tailored to accurately represent point clouds of umbrella-shaped drapes was devised. In all, 5160 drape meshes were collected by RGB-D cameras and Geomagic TM . Two neural networks were trained for 30 epochs using stochastic gradient descent with a momentum of 0.9. The learning rate was set to 0.1 for ResNet18 and 0.001 for the DGCNN. Experimental results demonstrated that the DGCNN coupled with the SDS method was the optimal feature extraction solution for woven fabric drapes, given that the accuracy reached 97% with the coefficient of variation of 7%. Therefore, our approach offered an objective and precise quantification of fabric drape, which provided a possible downstream application for searching fabrics based on drape similarity.

In this work, we take a further step towards satisfying practical demands in Chinese lyric generation from musical short-video creators, in respect of the challenges on songs' format constraints, creating specific lyrics from open-ended inspiration inputs, and language rhyme grace. One representative detail in these demands is to control lyric format at word level, that is, for Chinese songs, creators even expect fix-length words on certain positions in a lyric to match a special melody, while previous methods lack such ability. Although recent lyric generation community has made gratifying progress, most methods are not comprehensive enough to simultaneously meet these demands. As a result, we propose ChipSong, which is an assisted lyric generation system built based on a Transformer-based autoregressive language model architecture, and generates controlled lyric paragraphs fit for musical short-video display purpose, by designing 1) a novel Begin-Internal-End (BIE) word-granularity embedding sequence with its guided attention mechanism for word-level length format control, and an explicit symbol set for sentence-level length format control; 2) an open-ended trigger word mechanism to guide specific lyric contents generation; 3) a paradigm of reverse order training and shielding decoding for rhyme control. Extensive experiments show that our ChipSong generates fluent lyrics, with assuring the high consistency to pre-determined control conditions.

While transformers and their variant conformers show promising performance in speech recognition, the parameterized property leads to much memory cost during training and inference. Some works use cross-layer weight-sharing to reduce the parameters of the model. However, the inevitable loss of capacity harms the model performance. To address this issue, this paper proposes a parameter-efficient conformer via sharing sparsely-gated experts. Specifically, we use sparsely-gated mixture-of-experts (MoE) to extend the capacity of a conformer block without increasing computation. Then, the parameters of the grouped conformer blocks are shared so that the number of parameters is reduced. Next, to ensure the shared blocks with the flexibility of adapting representations at different levels, we design the MoE routers and normalization individually. Moreover, we use knowledge distillation to further improve the performance. Experimental results show that the proposed model achieves competitive performance with 1/3 of the parameters of the encoder, compared with the full-parameter model.

Approaches toward ordinal speech emotion recognition (SER) tasks are commonly based on the categorical classification algorithms, where the rank-order emotions are arbitrarily treated as independent categories. To employ the ordinal information between emotional ranks, we propose to model the ordinal SER tasks under a COnsistent RAnk Logits (CORAL) based deep learning framework. Specifically, a multi-class ordinal SER task is transformed into a series of binary SER sub-tasks predicting whether an utterance's emotion is larger than a rank. All the sub-tasks are jointly solved by one single network with a mislabelling cost defined as the the sum of the individual cross-entropy loss for each sub-task. Having the VGGish as our basic network structure, via minimizing above CORAL based cost, a VGGish-CORAL network is implemented in this contribution. Experimental results on a real-world call center dataset and the widely used IEMOCAP corpus demonstrate the effectiveness of VGGish-CORAL compared to the categorical VGGish.

In this paper, we present our system designed for the video emotion recognition task of the Multimodal Emotion Challenge (MEC 2017). Histogram of Oriented Gradients (HOG), face shape (SHAPE), and geometric (GEO) features are extracted from the detected face images as hand-crafted video features. A pre-trained VGG-Face model is fine-tuned with the face images and emotion labels from the training set of CHEAVD 2.0, the outputs of the penultimate fully-connected layer (FC6) and the last fully-connected layer (FC7) are adopted as Deep Convolutional Neural Network (DCNN) based features. For each video clip, the hand-crafted features and DCNN based features are input into corresponding hidden Markov models (HMMs, one for each emotion class), respectively, for the initial emotion recognitions. The output logarithm likelihood probabilities from the HMMs are then ranked, and the orders constitute an eight-dimensional feature vector as inputs to a Naive Bayes classifier for decision fusion. Experimental results on the CHEAVD 2.0 database show that the combination of FC6, GEO, SHAPE and HOG features obtains the highest macro average precisions (MAPs) on both the validation set (46.61%) and test set (43.88%), which are 12.51% and 22.18% higher than the baseline results, respectively.

To model the categorical speech emotion recognition task in a temporal manner, the first challenge arising is how to transfer the categorical label for each utterance into a label sequence.To settle this, we make a hypothesis that an utterance is consisting of emotional and non-emotional segments, and these non-emotional segments correspond to silent regions, short pauses, transitions between phonemes, unvoiced phonemes, etc.With this hypothesis, we propose to treat an utterance's label sequence as a chain of two states: the emotional state denoting the emotional frame and Null denoting the non-emotional frame.Then, we exploit a recurrent neural network based connectionist temporal classification model to automatically label and align an utterance's emotional segments with emotional labels, while non-emotional segments with Nulls.Experimental results on the IEMOCAP corpus validate our hypothesis and also demonstrate the effectiveness of our proposed method compared to the state-of-the-art algorithms.

In this work, a framework based on Convolution Neural Network (CNN) is proposed for speech emotion recognition (SER). We focus on extracting the most salient frames via the proposed CNN structure from the entire frame sequence to represent the utterance. A particular pooling method named global k-max pooling is utilized in our CNN structure (GCNN) to achieve the above objective. We implemented SER experiments on Interactive Emotional Dyadic Motion Capture (IEMOCAP), results are compared to those of some other CNN structures to validate the advancement of the presented framework. The experimental results turn out that GCNN outperforms others CNN models. Besides, experiments are also done to explore how many key frames should be output from GCNN to involve salient emotional information, results illuminate that limited length representation is properer while too long representation is likely containing redundant information decreasing the performance of the model.

This paper presents a work on the video-based emotion recognition task introduced in the Multimodal Emotion Recognition Challenge 2017. Encouraged by the widely used convoluational neural network based feature extraction methods in computer vision tasks, we leverage a fine-tuned VGGFace-16 network to generate features for each face image. Then, we explore a multi-dichotomy Recurrent Neural Network-Deep Neural Network (RNN-DNN) based framework for emotion classification. This framework first aggregate VGGFace-based face features from a same video to a global feature representation via its RNN layer, and further map the global feature representation to an emotional category using its dichotomy DNN layers. Experimental results on the challenge database demonstrate the effectiveness of our proposed system when compared to the baseline. Specifically, our best results reach the macro average precisions of 52.3% and 42.7% respectively on the validation and test data.

To model the categorical speech emotion recognition tasks in a sequential approach, the first challenge is how to transfer the categorical label for each utterance into a label sequence. To settle this, we make a hypothesis that an utterance is consisting of emotional and non-emotional segments alternatively, and these non-emotional segments correspond to silent regions, short pauses, transits between phonemes, fricative phonemes, etc. With this hypothesis, we propose to treat an utterance's label sequence as a chain of two kinds of states: emotional states denoting emotional frames and Nulls denoting non-emotional frames. Then, we exploit a connectionist temporal classification based recurrent neural network (CTC-RNN) to automatically label and align an utterance's emotional segments with emotional labels, while non-emotional segments with non-emotional labels. Experimental results on the IEMOCAP corpus demonstrate the effectiveness of our proposed method compared to state-of-the-art emotion recognition algorithms.

In this paper, we propose an audio visual multimodal depression recognition framework composed of deep convolutional neural network (DCNN) and deep neural network (DNN) models. For each modality, corresponding feature descriptors are input into a DCNN to learn high-level global features with compact dynamic information, which are then fed into a DNN to predict the PHQ-8 score. For multi-modal depression recognition, the predicted PHQ-8 scores from each modality are integrated in a DNN for the final prediction. In addition, we propose the Histogram of Displacement Range as a novel global visual descriptor to quantify the range and speed of the facial landmarks' displacements. Experiments have been carried out on the Distress Analysis Interview Corpus-Wizard of Oz (DAIC-WOZ) dataset for the Depression Sub-challenge of the Audio-Visual Emotion Challenge (AVEC 2016), results show that the proposed multi-modal depression recognition framework obtains very promising results on both the development set and test set, which outperforms the state-of-the-art results.

Coping with scarcity of labeled data is a common problem in sound classification tasks. Approaches for classifying sounds are commonly based on supervised learning algorithms, which require labeled data which is often scarce and leads to models that do not generalize well. In this paper, we make an efficient combination of confidence-based Active Learning and Self-Training with the aim of minimizing the need for human annotation for sound classification model training. The proposed method pre-processes the instances that are ready for labeling by calculating their classifier confidence scores, and then delivers the candidates with lower scores to human annotators, and those with high scores are automatically labeled by the machine. We demonstrate the feasibility and efficacy of this method in two practical scenarios: pool-based and stream-based processing. Extensive experimental results indicate that our approach requires significantly less labeled instances to reach the same performance in both scenarios compared to Passive Learning, Active Learning and Self-Training. A reduction of 52.2% in human labeled instances is achieved in both of the pool-based and stream-based scenarios on a sound classification task considering 16,930 sound instances.

In this paper, we propose a feature selection and representation combination method to generate discriminative features for speech emotion recognition. In feature selection stage, a Multiple Kernel Learning (MKL) based strategy is used to obtain the optimal feature subset. Specifically, features selected at least n times among 10-fold cross validation are collected to build a new feature subset named n-subset, then the n-subset resulting in the highest classification accuracy is viewed as the optimal one. In feature representation stage, the optimal feature subset is mapped to a hidden representation using a denoising autoencoder (DAE). The model parameters are learned by minimizing the squared error between the original and the reconstructed input. The hidden representation is then used as the final feature set in the MKL model for emotion recognition. Our experimental results show significant performance improvement compared to using the original features in both of the inner-corpus and cross-corpus scenarios.

对语音情感识别的研究现状和进展进行了归纳和总结,对未来语音情感识别技术发展趋势进行了展望.从5个角度逐步展开进行归纳总结,即情感描述模型、具有代表性的情感语音库、语音情感特征提取、语音情感识别算法研究和语音情感识别技术应用,旨在尽可能全面地对语音情感识别技术进行细致的介绍与分析,为相关研究人员提供有价值的学术参考;最后,立足于研究现状的分析与把握,对当前语音情感识别领域所面临的挑战与发展趋势进行了展望.侧重于对语音情感识别研究的主流方法和前沿进展进行概括、比较和分析.

The Gaussian Copula Model (GCM) plays an important role in the state-of-the-art financial analysis field for modeling the dependence of financial assets. However, the existing implementations of GCM are all computationallydemanding and time-consuming. In this paper, we propose a Dataflow Engine (DFE) design to accelerate the GCM computation. Specifically, a commonly used CPU-friendly GCM algorithm is converted into a fully-pipelined dataflow graph through four steps of optimization: recomposing the algorithm to be pipeline-friendly, removing unnecessary computation, sharing common computing results, and reducing the computing precision while maintaining the same level of accuracy for the computation results. The performance of the proposed DFE design is compared with three CPU-based implementations that are well-optimized. Experimental results show that our DFE solution not only generates fairly accurate result, but also achieves a maximum of 467x speedup over a single-thread CPU-based solution, 120x speedup over a multi-thread CPUbased solution, and 47x speedup over an MPI-based solution.

State-of-the-art dimensional speech emotion recognition systems are trained using continuously labelled instances. The data labelling process is labour intensive and time-consuming. In this paper, we propose to apply active learning to reduce according efforts: The unlabelled instances are evaluated automatically, and only the most informative ones are intelligently picked by an informativeness measure function for a human to label. Specifically, we estimate the informativeness of each unlabelled instance based on a binary-classification confidence score for an emotion being predicted to be negative or positive on a given emotional dimension. For verification, we consider a pool-based and a stream-based scenario run on part of the continuous AVEC 2012 task to demonstrate the feasibility of the proposed approach in practice. In the result, our approach requires significantly less human labelled data instances to reach a given performance than passive learning does in both scenarios. Index Terms: Active Learning, Speech Emotion Recognition, Affective Computing, Continuous Emotion Representation

Emotional speech databases(ESDs) provide training and test data for the construction of speech emotion recognition model.Nowadays,a number of ESDs have been built by domestic and abroad researchers in the field of speech emotion recognition,under the background of their own tasks.However,since the complexity of emotion and the lack of uniform standard on ESD building,only a limit number of databases have been generally accepted by most researchers.This paper presents a review on the most representative ESDs in state-of-the-art research community on the base of investigation and analysis on references.The goal of this review is to provide reliable in-formation and reference to related researchers when they build or choose suitable database for their research.

In this paper, we use the concept of dynamic trend of emotion to describe how a human's emotion changes over time, which is believed to be important for understanding one's stance toward current topic in interactions. However, the importance of this concept - to our best knowledge - has not been paid enough attention before in the field of speech emotion recognition (SER). Inspired by this, this paper aims to evoke researchers' attention on this concept and makes a primary effort on the research of predicting correct dynamic trend of emotion in the process of SER. Specifically, we propose a novel algorithm named Order Preserving Network (OPNet) to this end. First, as the key issue for OPNet construction, we propose employing a probabilistic method to define an emotion trend-sensitive loss function. Then, a nonlinear neural network is trained using the gradient descent as optimization algorithm to minimize the constructed loss function. We validated the prediction performance of OPNet on the VAM corpus, by mean linear error as well as a rank correlation coefficient γ as measures. Comparing to k-Nearest Neighbor and support vector regression, the proposed OPNet performs better on the preservation of actual dynamic trend of emotion.

This paper introduces an approach for performing distributed speech emotion recognition in a client-server architecture. In this architecture, the client side deals only with feature extraction, compression and bit-stream formatting, while the server side performs bit-stream decoding, feature decompression and emotion recognition, which requires more computational resources. Taking into account the trade-off between the required transmission bandwidth and recognition accuracy, we propose to employ a vector quantization approach based on independent codebooks for feature sub-spaces. Extensive test runs are conducted to reveal the impact of quantization parameters on the compression rate and recognition performance. In the result, by using a quantization strategy involving 32 subvectors and 9 bit codeword length, almost 30 times compression can be reached without a considerable increase of the error rate.

Speech emotion recognition (SER) in use today lacks the ability to evaluate reliability of recognition results although it has matured to the degree of first applicability. In this paper, we thus propose a novel confidence measure for SER systems. The confidence measure is based on human labeller agreement. This information is used to build a series of emotion scoring models to provide multiple agreement levels for a hypothesised emotion state. A fusion is carried out on multiple agreement levels for a confidence score. Experimental results on the FAU Aibo Emotion Corpus of the INTERSPEECH 2009 Emotion Challenge show that the proposed confidence score has strong correlation with the unweighted average recall of the target task ?? emotion ??, thus effectively indicating the usefulness of the confidence measures.