We address a challenging and practical task of labeling questions in speech in real time during telephone calls to emergency medical services in English, which embeds within a broader decision support system for emergency call-takers. We propose a novel multimodal approach to real-time sequence labeling in speech. Our model treats speech and its own textual representation as two separate modalities or views, as it jointly learns from streamed audio and its noisy transcription into text via automatic speech recognition. Our results show significant gains of jointly learning from the two modalities when compared to text or audio only, under adverse noise and limited volume of training data. The results generalize to medical symptoms detection where we observe a similar pattern of improvements with multimodal learning.

This paper provides an overview of the development of a self-oriented solar mirror (SOSM) project within the European Project Semester (EPS) at Instituto Superior de Engenharia do Porto (ISEP). While the main objective of the EPS@ISEP project-based educational framework is to foster teamwork, communication, interpersonal and problem solving skills in an international, multidisciplinary engineering environment, the goal of the SOSM is to track and reflect the Sun radiation onto a pre-defined area. In the spring of 2017 a group of five students chose to develop a proof-of-concept domestic SOSM called SUNO. The students undertook project supportive modules in Ethics, Sustainability, Marketing and Project Management together with project coaching meetings to assist the development of SUNO. The paper details this process, describing the initial project definition, the research of current technologies, the designing, the manufacturing and testing of the SUNO prototype, and discusses what the students gained from this learning experience.