In this work, we propose a novel Generative Adversarial Stacked Autoencoder that learns to map facial expressions, with up to plus or minus 60 degrees, to an illumination invariant facial representation of 0 degrees. We accomplish this by using a novel convolutional layer that exploits both local and global spatial information, and a convolutional layer with a reduced number of parameters that exploits facial symmetry. Furthermore, we introduce a generative adversarial gradual greedy layer-wise learning algorithm designed to train Adversarial Autoencoders in an efficient and incremental manner. We demonstrate the efficiency of our method and report state-of-the-art performance on several facial emotion recognition corpora, including one collected in the wild.

Deep neural networks have proven to be efficient systems for learning complex data representations. However, one of their main constraints is their inability to deal with changes in the data distribution. For instance, in real-time facial expression recognition, the data used to evaluate a model commonly differs in quality compared to that used to train the model, leading to poor generalization performance. In this work we propose a novel Deep Convolutional Neural Network (CNN) architecture pre-trained as a Stacked Convolutional Autoencoder (SCAE) to address emotion recognition in unconstrained environments. The SCAE is trained in a greedy layer-wise unsupervised fashion, and combines convolutional and fully connected layers and learns to encode facial expression images as an illumination and facial pose invariant feature vector. The CNN offers state-of-the-art classification rate of 99.52% on a combined corpus of gamma corrected version of the CK+, JAFFE, FEEDTUM and KDEF datasets. When evaluated on unseen data obtained in unconstrained environments, our approach achieves 79.75%, an increase of over 28% compared to a CNN without our pretraining approach, supporting the methodology proposed in this work.

As autonomous vehicles become more common on the roads, their advancement draws on safety concerns for vulnerable road users, such as pedestrians and cyclists. This paper presents a review of recent developments in pedestrian and cyclist detection and intent estimation to increase the safety of autonomous vehicles, for both the driver and other road users. Understanding the intentions of the pedestrian/cyclist enables the self-driving vehicle to take actions to avoid incidents. To make this possible, development of methods/techniques, such as deep learning (DL), for the autonomous vehicle will be explored. For example, the development of pedestrian detection has been significantly advanced using DL approaches, such as; Fast Region-Convolutional Neural Network (R-CNN) , Faster R-CNN and Single Shot Detector (SSD). Although DL has been around for several decades, the hardware to realise the techniques have only recently become viable. Using these DL methods for pedestrian and cyclist detection and applying it for the tracking, motion modelling and pose estimation can allow for a successful and accurate method of intent estimation for the vulnerable road users. Although there has been a growth in research surrounding the study of pedestrian detection using vision-based approaches, further attention should include focus on cyclist detection. To further improve safety for these vulnerable road users (VRUs), approaches such as sensor fusion and intent estimation should be investigated.

The domain of automatic face detection is a challenging problem that has made great progress in the last two decades. Much of this progress is due to the continuous advancements in deep learning and computer vision. Nonetheless, contemporary state-of-the-art face detection models rely on exhaustive search or are unable to deal with changes in the data distribution. In this work, we propose a novel Deep Reinforcement Learning (DRL) approach for face detection on data with nonuniform conditions. More specifically, we address illumination and rotation invariance in face detection. Firstly, we train a Stacked Convolutional Autoencoder (SCAE) in a greedy layer-wise unsupervised fashion for illumination invariant feature extraction. We then train a deep Q-network on the illumination invariant features produced by the SCAE model, to learn an action-value policy that allows an agent to place a bounding box around a face. The proposed approach achieves state-of-the-art recognition rates on images with varying degrees of illumination and images that contain faces with some degree of rotation.

With the continued advancement of autonomous vehicles and their implementation in public roads, accurate detection of vulnerable road users (VRUs) is vital for ensuring safety. To provide higher levels of safety for these VRUs, an effective detection system should be employed that can correctly identify VRUs in all types of environments (e.g. VRU appearance, crowded scenes) and conditions (e.g. fog, rain, night-time). This paper presents optimal methods of sensor fusion for pedestrian and cyclist detection using Deep Neural Networks (DNNs) for higher levels of feature abstraction. Typically, visible sensors have been utilized for this purpose. Recently, thermal sensors system or combination of visual and thermal sensors have been employed for pedestrian detection with advanced detection algorithm. DNNs have provided promising results for improving the accuracy of pedestrian and cyclist detection. This is because they are able to extract features at higher levels than typical hand-crafted detectors. Previous studies have shown that amongst the several sensor fusion techniques that exist, Halfway Fusion has provided the best results in terms of accuracy and robustness. Although sensor fusion and DNN implementation have been used for pedestrian detection, there is considerably less research undertaken for cyclist detection.

We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016.  https://doi.org/10.1007/978-3-319-44188-7_6), to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of \(96.26\%\), when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167, 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.

In this work we propose a novel method to address illumination invariance for facial expression recognition. We propose a Deep Convolutional Network (CNN) pre-trained as a Deep Stacked Convolutional Autoencoder (SCAE) in a greedy layer-wise unsupervised fashion. The SCAE model learns to encode facial expression images and produce a feature vector with relatively similar illumination, regardless of the luminance level of the input image. Moreover, we propose fine-tuning the stacked shallow autoencoders after each one of these is trained greedily, rather than just at the end, and show that this approach significantly improves the set of illumination invariant features learnt by the SCAE. Finally, we propose the use of a variant rectifier linear unit transfer function that helps the SCAE model reduce or increase the illumination of images with high or low luminance, and show that the lower and upper bounds greatly influence classification performance. The method proposed provides an increase in classification accuracy of 4% on the KDEF dataset and 8% on the CK+ dataset.

Human robot interaction is a rapidly growing topic of interest in today’s society. The development of real time emotion recognition will further improve the relationship between humans and social robots. However, contemporary real time emotion recognition in unconstrained environments has yet to reach the accuracy levels achieved on controlled static datasets. In this work, we propose a Deep Convolutional Neural Network (CNN), pre-trained as a Stacked Convolutional Autoencoder (SCAE) in a greedy layer-wise unsupervised manner, for emotion recognition from facial expression images taken by a NAO robot. The SCAE model is trained to learn an illumination invariant down-sampled feature vector. The weights of the encoder element are then used to initialize the CNN model, which is fine-tuned for classification. We train the model on a corpus composed of gamma corrected versions of the CK+ , JAFFE, FEEDTUM and KDEF datasets. The emotion recognition model produces a state-of-the-art accuracy rate of 99.14% on this corpus. We also show that the proposed training approach significantly improves the CNN’s generalisation ability by over 30% on nonuniform data collected with the NAO robot in unconstrained environments.

Emotion recognition is critical for everyday living and is essential for meaningful interaction. If we are to progress towards human and machine interaction that is engaging the human user, the machine should be able to recognize the emotional state of the user. Deep Convolutional Neural Networks (CNN) have proven to be efficient in emotion recognition problems. The good degree of performance achieved by these classifiers can be attributed to their ability to self-learn a down-sampled feature vector that retains spatial information through filter kernels in convolutional layers. Given the view that random initialization of weights can lead to convergence to non-optimal local minima, in this paper we explore the impact of training the initial weights in an unsupervised manner. We study the effect of pre-training a Deep CNN as a Stacked Convolutional Auto-Encoder (SCAE) in a greedy layer-wise unsupervised fashion for emotion recognition using facial expression images. When trained with randomly initialized weights, our CNN emotion recognition model achieves a performance rate of 91.16% on the Karolinska Directed Emotional Faces (KDEF) dataset. In contrast, when each layer of the model, including the hidden layer, is pre-trained as an Auto-Encoder, the performance increases to 92.52%. Pre-training our CNN as a SCAE also reduces training time marginally. The emotion recognition model developed in this work will form the basis of a real-time empathic robot system.

Deep Learning (DL) has shown real promise for the classification efficiency for emotion recognition problems. In this paper we present experimental results for a deeply-trained model for emotion recognition through the use of facial expression images. We explore two Convolutional Neural Network (CNN) architectures that offer automatic feature extraction and representation, followed by fully connected softmax layers to classify images into seven emotions. The first architecture explores the impact of reducing the number of deep learning layers and the second splits the input images horizontally into two streams based on eye and mouth positions. The first proposed architecture produces state of the art results with an accuracy rate of 96.93 % and the second architecture with split input produces an average accuracy rate of 86.73 %, respectively.

Social robots are gradually becoming part of society. However, social robots lack the ability to adequately interact with users in a natural manner and are in need of more human-like abilities. In this paper we present experimental results on emotion recognition through the use of facial expression images obtained from the KDEF database, a fundamental first step towards the development of an empathic social robot. We compare the performance of Support Vector Machines (SVM) and a Multilayer Perceptron Network (MLP) on facial expression classification. We employ Gabor filters as an image pre-processing step before classification. Our SVM model achieves an accuracy rate of 97.08 %, whereas our MLP achieves 93.5 %. These experiments serve as benchmark for our current research project in the area of social robotics.

This paper explores the extraction of features from the speech wave to perform intelligent emotion recognition. A feature extract tool (openSmile) was used to obtain a baseline set of 998 acoustic features from a set of emotional speech recordings from a microphone. The initial features were reduced to the most important ones so recognition of emotions using a supervised neural network could be performed. Given that the future use of virtual education agents lies with making the agents more interactive, developing agents with the capability to recognise and adapt to the emotional state of humans is an important step.

This paper presents a hybrid neural approach to emotion recognition from speech, which combines feature selection using principal component analysis (PCA) with unsupervised neural clustering through self-organising map (SOM). Given the importance that is associated with emotions in humans, it is unlikely that robots will be accepted as anything more that machines if they do not express and recognise emotions. In this paper, we describe the performance of an unsupervised approach to emotion recognition that achieves similar performance to current supervised intelligent approaches. Performance, however, reduces when the system is tested using samples from a male volunteer not in the training set using a low cost microphone. Through the use of an unsupervised neural approach, it is possible to go beyond the basic binary classification of emotions to consider the similarity between emotions and whether speech can express multiple emotions at the same time.

This paper describes an attention-gating recurrent self-organising map approach for emergent speech representation. Inspired by evidence from human cognitive processing, the architecture combines two main neural components. The first component, the attention-gating mechanism, uses actor–critic learning to perform selective attention towards speech. Through this selective attention approach, the attention-gating mechanism controls access to working memory processing. The second component, the recurrent self-organising map memory, develops a temporal-distributed representation of speech using phone-like structures. Representing speech in terms of phonetic features in an emergent self-organised fashion, according to research on child cognitive development, recreates the approach found in infants. Using this representational approach, in a fashion similar to infants, should improve the performance of automatic recognition systems through aiding speech segmentation and fast word learning.

If social robots are to become widespread there is a need for them to offer spoken communication in a manner close to humans. This should be achieved by grounding the speech being recognised by the robot into the real world instead of trying to recognise the speech signal in isolation. This paper examines a hierarchical recurrent self-organising map (H-RSOM) memory model for emergent speech representation, which makes use of evidence from human cognitive processing. The H-RSOM memory architecture is constrained by the structure of the cerebral cortex and working memory, neurocognitive evidence on word representation and infant speech acquisition. By associating the speech signal representation with semantic (visual) features the HRSOM offers a temporal-distribution speech representation based on phone like-speech sounds and semantic (visual) features. Using this representational approach it should be possible in the future, in a similar fashion to infants, to develop a robotic automatic speech recogniser that offers grounding of words in meaning.

The MiCRAM project has been developing detailed neural models of the inferior colliculus (IC) as the basis for understanding the neural networks that underlie early auditory processing at that level. The initial phase of this project has been to develop biologically plausible models of the physiologically distinct cell types of the IC. This has led to the recognition that some of the gross behavior that produces delay sensitivity may be the result of detailed dynamic interactions between multiple channels.