We propose a lightweight neural front-end framework for on-device speech generation and highlight its benefits towards low-resource language scaling. While data-driven models have shown potential in front-end literature, especially since they can enable fast language expansion, they are often extremely large and of high latency. There is limited work focusing on their usability in real-time settings, and none for on-device TTS applications. At the cost of small performances trade-offs, we build lightweight neural Grapheme-to-Phoneme and verbalization models which achieve, on average across three languages, a p90 latency reduction of 95.98% per token on single-threaded CPU@2.30GHz, with respect to a traditional transformer-based baseline, while having 99.26% less parameters. Additionally, leveraging pre-trained teacher models to bootstrap lightweight students, we enable low-resource language scaling on both Grapheme-to-Phoneme conversion and verbalization.

Phonetic information and linguistic knowledge are an essential component of a Text-to-speech (TTS) front-end. Given a language, a lexicon can be collected offline and Grapheme-to-Phoneme (G2P) relationships are usually modeled in order to predict the pronunciation for out-of-vocabulary (OOV) words. Additionally, post-lexical phonology, often defined in the form of rule-based systems, is used to correct pronunciation within or between words. In this work we showcase a multilingual unified front-end system that addresses any pronunciation related task, typically handled by separate modules. We evaluate the proposed model on G2P conversion and other language-specific challenges, such as homograph and polyphones disambiguation, post-lexical rules and implicit diacritization. We find that the multilingual model is competitive across languages and tasks, however, some trade-offs exists when compared to equivalent monolingual solutions.

Advances in Internet of Things (IoT) devices and in Machine Learning (ML) applications can provide valuable insights and predictions on personal health by optimizing data generation and processing. Nevertheless, the flow of data about the health status of a patient brings a variety of technical, legal and economic challenges that need to be addressed through an interdisciplinary approach.In this context, based on the action research methodology, the paper introduces an exemplary health-related activity recognition platform based on IoT, developed as a part of European-funded project Horizon 2020 in collaboration with academia and industry.The platform proposes innovative solutions on how personal healthcare data can be processed and analysed, protecting users' privacy. The main strength of the platform is the interdisciplinary approach used within a triple-helix model, involving a variety of institutions, companies and researchers from different academic fields. In this perspective, the paper shows the potential that the integration of IoT and ML models have to offer and the main challenges that still need to be addressed.

Recent studies show that neural natural processing models for medical code prediction suffer from a label imbalance issue. This study aims to investigate further imbalance in a medical code prediction dataset in terms of demographic variables and analyse performance differences in demographic groups. We use sample-based metrics to correctly evaluate the performance in terms of the data subject. Also, a simple label distance metric is proposed to quantify the difference in the label distribution between a group and the entire data. Our analysis results reveal that the model performs differently towards different demographic groups: significant differences between age groups and between insurance types are observed. Interestingly, we found a weak positive correlation between the number of training data of the group and the performance of the group. However, a strong negative correlation between the label distance of the group and the performance of the group is observed. This result suggests that the model tends to perform poorly in the group whose label distribution is different from the global label distribution of the training data set. Further analysis of the model performance is required to identify the cause of these differences and to improve the model building.

There has been significant progress in dialogue systems research.However, dialogue systems research in the healthcare domain is still in its infancy.In this paper, we analyse recent studies and outline three building blocks of a taskoriented dialogue system in the healthcare domain: i) privacy-preserving data collection; ii) medical knowledge-grounded dialogue management; and iii) human-centric evaluations.To this end, we propose a framework for developing a dialogue system and show preliminary results of simulated dialogue data generation by utilising expert knowledge and crowdsourcing.

Extracting temporal information is critical to process health-related text. Temporal information extraction is a challenging task for language models because it requires processing both texts and numbers. Moreover, the fundamental challenge is how to obtain a large-scale training dataset. To address this, we propose a synthetic data generation algorithm. Also, we propose a novel multi-task temporal information extraction model and investigate whether multi-task learning can contribute to performance improvement by exploiting additional training signals with the existing training data. For experiments, we collected a custom dataset containing unstructured texts with temporal information of sleep-related activities. Experimental results show that utilising synthetic data can improve the performance when the augmentation factor is 3. The results also show that when multi-task learning is used with an appropriate amount of synthetic data, the performance can significantly improve from 82. to 88.6 and from 83.9 to 91.9 regarding micro-and macro-average exact match scores of normalised time prediction, respectively.

Almost 30% of the adult population in the world is experiencing or has experience insomnia. Cognitive Behaviour Therapy for insomnia (CBT-I) is one of the most effective treatment, but it has limitations on accessibility and availability. Utilising technology is one of the possible solutions, but existing methods neglect conversational aspects, which plays a critical role in sleep therapy. To address this issue, we propose a PhD project exploring potentials of developing conversational artificial intelligence (AI) for a sleep coaching programme, which is motivated by CBT-I treatment. This PhD project aims to develop natural language processing (NLP) algorithms to allow the system to interact naturally with a user and provide automated analytic system to support human experts. In this paper, we introduce research questions lying under three phases of the sleep coaching programme: triaging, monitoring the progress, and providing coaching. We expect this research project’s outcomes could contribute to the research domains of NLP and AI but also the healthcare field by providing a more accessible and affordable sleep treatment solution and an automated analytic system to lessen the burden of human experts.

Recently proposed pre-trained language models can be easily fine-tuned to a wide range of downstream tasks. However, a large-scale labelled task-specific dataset is required for fine-tuning creating a bottleneck in the development process of machine learning applications. To foster a fast development by reducing manual labelling efforts, we propose a L abel- E fficient T raining S cheme (LETS). The proposed LETS consists of three elements: (i) task-specific pre-training to exploit unlabelled task-specific corpus data, (ii) label augmentation to maximise the utility of labelled data, and (iii) active learning to label data strategically. In this paper, we apply LETS to a novel aspect-based sentiment analysis (ABSA) use-case for analysing the reviews of the health-related program supporting people to improve their sleep quality. We validate the proposed LETS on a custom health-related program-reviews dataset and another ABSA benchmark dataset. Experimental results show that the LETS can reduce manual labelling efforts 2-3 times compared to labelling with random sampling on both datasets. The LETS also outperforms other state-of-the-art active learning methods. Furthermore, the experimental results show that LETS can contribute to better generalisability with both datasets compared to other methods thanks to the task-specific pre-training and the proposed label augmentation. We expect this work could contribute to the natural language processing (NLP) domain by addressing the issue of the high cost of manually labelling data. Also, our work could contribute to the healthcare domain by introducing a new potential application of NLP techniques.

This paper explores how Dutch diary fragments, written by family coaches in the social sector, can be analysed automatically using machine learning techniques to quantitatively measure the impact of social coaching. The focus lays on two tasks: determining which sentiment a fragment contains (sentiment analysis) and investigating which fundamental social rights (education, employment, legal aid, etc.) are addressed in the fragment. To train and test the new algorithms, a dataset consisting of 1715 Dutch diary fragments is used. These fragments are manually labelled on sentiment and on the applicable fundamental social rights. The sentiment analysis models were trained to classify the fragments into three classes: negative, neutral or positive. Fine-tuning the Dutch pre-trained Bidirectional Encoder Representations from Transformers (BERTje) (de Vries et al., 2019) language model surpassed the more classic algorithms by correctly classifying 79.6% of the fragments on the sentiment analysis, which is considered as a good result. This technique also achieved the best results in the identification of the fundamental rights, where for every fragment the three most likely fundamental rights were given as output. In this way, 93% of the present fundamental rights were correctly recognised. To our knowledge, we are the first to try to extract social rights from written text with the help of Natural Language Processing techniques.

User feedback is essential for understanding user needs. In this paper, we use free-text obtained from a survey on sleep-related issues to build a deep neural networks-based text classifier. However, to train the deep neural networks model, a lot of labelled data is needed. To reduce manual data labelling, we propose a method which is a combination of data augmentation and pseudo-labelling: data augmentation is applied to labelled data to increase the size of the initial train set and then the trained model is used to annotate unlabelled data with pseudo-labels. The result shows that the model with the data augmentation achieves macro-averaged f1 score of 65.2% while using 4,300 training data, whereas the model without data augmentation achieves macro-averaged f1 score of 68.2% with around 14,000 training data. Furthermore, with the combination of pseudo-labelling, the model achieves macro-averaged f1 score of 62.7% with only using 1,400 training data with labels. In other words, with the proposed method we can reduce the amount of labelled data for training while achieving relatively good performance.

표정은 사람이 감정을 표현하는 직접적인 방법 중 하나이다. 사람은 연속시간 상에서 표정의 변화를 통해 감정 상태를 나타내지만 기존의 기술은 대부분 정지영상에서만 감정 상태를 인식하였다. 본 연구는 연속 시간상에서 감정 상태를 추론하기 위한 딥러닝 시스템을 제안한다. 제안된 연속 시간 감정 상태 추론 시스템은 세 단계로 이루어져 있다. Convolutional Neural Network (CNN) 모델을 사용한 특징 추출 단계, 활성화 지도를 사용한 얼굴영역 검출 및 관심 영역 풀링 단계, 그리고 Recurrent Neural Network (RNN) 모델을 사용한 시계열 추론 단계. 본 논문에서는 짧은 표정 동영상을 통해 감정 상태를 추론하는 실험을 수행하였다. 실험을 통해 제안된 시스템이 기존의 CNN 기반 물체 검출 모델보다 빠르게 얼굴영역 검출을 수행하며, 연속 시간상에서 감정 상태 추론이 가능함을 보였다.

In this paper, we propose a facial expression recognition system using a deep convolutional network. Previous works used the facial action coding system (FACS) to classify emotions. Therefore, the system consists of a face detector, a feature extractor, a facial action classifier, and an emotional state classifier in series. In contrast, the proposed system is a simplified emotion recognition system that performs face detection and emotion classification in parallel. Moreover, the model was trained without any prior knowledge of FACS. We evaluated its performance on four different databases. Our main contributions are two folds: 1) Our simplified facial expression recognition system processes images in real-time. 2) Our model was trained to classify facial expressions without any action unit (AU) related information. The proposed method achieved a classification accuracy of 98.6% on six basic emotions and a neutral state from faces with five different angles. The experimental results showed that the deep convolutional network could classify emotional states from a multi-angle facial expressions database and various facial expression databases without the use of hand-crafted features.

본 연구는 노동집약적인 신발제조공정을 자동화하기 위한 기술 중 신발갑피 패턴분석 시스템을 제안한다. 제안 시스템은 심층 신경망으로 갑피패턴을 검출/인식하는 단계와 영상처리 알고리즘으로 패턴의 크기와 회전각과 같은 상태 정보를 분석하는 단계로 구성된다. 성능 평가를 위해 국제공인시험소에 시험을 의뢰 하였으며, 목표 성능인 95%의 분류율, 1000msec 이하의 비전 처리 속도, 3mm 이하의 비전 정밀도를 웃도는 결과를 보였다. 제안된 기술은 추후 열용융 신발갑피 재봉 자동화 로봇 시스템에서 로봇 제어를 위한 로봇 비전 시스템에 사용될 예정이다.

This paper proposes automatically generate a virtual guidance for teleoperation system using object detection. A virtual guidance that gives force feedback when performing a mission like a peg-in-hole using a slave robot with a haptic device is very helpful for the operator. The key point of our study is to automatically generate this virtual guidance using deep learning architecture. If the operator uses this information, there is no need to generate a virtual guidance one by one. In addition, our system can automatically and continuously generate virtual guidance in moving situations as well as stationary situations. The position of the target object is recognized using deep learning architecture and depth information. And visual information of the virtual guidance is visualized using the virtual environment visualization system. Our system helps the operator to recognize spatial information.

Human-Robot Interaction (HRI) technology plays a key role in social robotics. The main goal of HRI studies is to develop robots that can communicate with humans through an intuitive interface. In this paper, we focused on the Human Action Understanding (HAU) problem in HRI study fields. We propose an approach for HAU tasks using deep learning models. The proposed system consists of an object-human interaction pattern extraction module and a behaviour goal estimation module. Our main contributions are two-fold: 1) We applied a deep learning method for real-time object detection to extract object-human interaction patterns. 2) We also applied a recurrent neural network to complete the behaviour goal estimation task with a temporal contextual feature data sequence based on object-human interaction patterns. The experimental results from the benchmark data set for human behaviour recognition show that the proposed method performs well.

Facial expressions convey not only emotions but also communicative information. Therefore, facial expressions should be analysed to understand communication. The objective of this study is to develop an automatic facial expression analysis system for extracting nonverbal communicative information. This study focuses on specific communicative information: emotions expressed through facial movements and the direction of the expressions. We propose a multi-tasking deep convolutional network (DCN) to classify facial expressions, detect the facial regions, and estimate face angles. We reformulate facial region detection and face angle estimation as regression problems and add task-specific output layers in the DCN’s architecture. Experimental results show that the proposed method performs all tasks accurately. In this study, we show the feasibility of the multi-tasking DCN for extracting nonverbal communicative information from a human face.

The recognition and visualization of the situations in remote areas help the tele-operator work more effective and safe. This paper proposes the visualization system which can acquire and display 2D and 3D image-based environmental information for the operator. The proposed visualization system consists of egocentric viewer and exocentric viewer system. The egocentric viewer system shows VR information based on the object detection results using deep learning architecture. The exocentric viewer system shows the pose and space constraint information of the remote operation robot to be controlled on the 3D point cloud data scene that is capable of point-of view conversion.

Robots have been widely used in many industries, from manufacturing to services. In addition, Human Activity Recognition technology has become important as robots become smart thanks to recent development of artificial intelligence. Future robots will be able to collaborate and co-work with humans. The main purpose of this article is to make robots understand human activities with a machine vision system. Analysing human activities is not only a problem of detecting human motion but also interaction between humans with objects. Therefore, we propose an integrated object–human interaction pattern generating system consisting of object detector, skeletal tracker, and object–human interaction detector. This system is designed to detect objects, track humans’ skeletal movements in sequential images and analyse the interaction between object and human. Here, we focused on everyday human activities with related objects. The proposed system generates the object–human interaction patterns of each activity.