The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call"direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of frameworks and $p$ is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).

Community question answering platforms need to automatically rank answers and questions with respect to a given question.In this paper, we present the approaches for the Answer Selection and Question Retrieval tasks of SemEval-2016 (task 3).We develop a bag-of-vectors approach with various vectorand text-based features, and different neural network approaches including CNNs and LSTMs to capture the semantic similarity between questions and answers for ranking purpose.Our evaluation demonstrates that our approaches significantly outperform the baselines.

Vector representations for language has been shown to be useful in a number of Natural Language Processing tasks. In this paper, we aim to investigate the effectiveness of word vector representations for the problem of Aspect Based Sentiment Analysis. In particular, we target three sub-tasks namely aspect term extraction, aspect category detection, and aspect sentiment prediction. We investigate the effectiveness of vector representations over different text data and evaluate the quality of domain-dependent vectors. We utilize vector representations to compute various vectorbased features and conduct extensive experiments to demonstrate their effectiveness. Using simple vector based features, we achieve F1 scores of 79.91% for aspect term extraction, 86.75% for category detection, and the accuracy 72.39% for aspect sentiment prediction.

Continuous word and phrase vectors have proven useful in a number of NLP tasks.Here we describe our experience using them as a source of features for the SemEval-2015 task 3, consisting of two community question answering subtasks: Answer Selection for categorizing answers as potential, good, and bad with regards to their corresponding questions; and YES/NO inference for predicting a yes, no, or unsure response to a YES/NO question using all of its good answers.Our system ranked 6th and 1st in the English answer selection and YES/NO inference subtasks respectively, and 2nd in the Arabic answer selection subtask.

The widespread acceptance of personal digital assistants (PDAs) has led to research into the interaction people have with these devices. Speech is a natural choice for this interaction. However, traditional speech recognition systems require an abundance of memory and processor cycles. On limited machines like an iPAQ, implementing an entire speech recognition system would be debilitating to the device. A solution to this problem is to allow the iPAQ to communicate with a server that does the actual recognition task. This method is better known as distributed speech recognition (DSR). This thesis examines the problems of implementing DSR on an iPAQ. We faced the challenge of reducing the bandwidth required by the system while maintaining reasonable recognition error rates. We examined using a fixed-point processing in reducing the computational demand put on the iPAQ. The word error rates for the baseline floating-point front-end system and our fixed-point front-end were 9.8% and 9.6% respectively. However, using the fixed-point front-end actually increased our bit rate. Next, we focused on the effects of quantizing Mel-Frequency Cepstral Coefficients (MFCCs) before sending them to a recognizer on the server side. We evaluated both scalar and vector quantizers using non-uniform bit allocation. Our optimal vector quantizer reached a word error rate of 9.8% at 6400 bps. Finally, because our recognizer further processes the MFCCs to arrive at boundary measurements, we explored the idea of quantizing these boundary measurements. The scalar boundary measurement quantizer reached a word error rate of 9.6% at 150 bits per hypothesized boundary. We averaged 21.1 hypothesize boundaries per second on our test data; thus, we could transmit boundary measurements at 3165 bps and maintain a 9.6% word error rate. Thesis Supervisor: I. Lee Hetherington Title: Research Scientist

This paper presents initial data collection and language understanding experiments conducted as part of a larger effort to create a nutrition dialogue system that automatically extracts food concepts from a user's spoken meal description. We first summarize the data collection and annotation of food descriptions performed via Amazon Mechanical Turk. We then present semantic labeling experiments using a semi-Markov conditional random field (CRF) that obtains an F1 test score of 85.1. Finally, we report food segmentation experiments that explored three methods for associating foods with their corresponding attributes: a generative Markov model, transformation-based learning, and a CRF classifier. The CRF performed best, achieving an F1 test score of 87.1.

This paper presents an empirical study on using syntactic and semantic information for Concept Segmentation and Labeling (CSL), a well-known component in spoken language understanding. Our approach is based on reranking N -best outputs from a state-of-the-art CSL parser. We perform extensive experimentation by comparing different tree-based kernels with a variety of representations of the available linguistic information, including semantic concepts, words, POS tags, shallow and full syntax, and discourse trees. The results show that the structured representation with the semantic concepts yields significant improvement over the base CSL parser, much larger compared to learning with an explicit feature vector representation. We also show that shallow syntax helps improve the results and that discourse relations can be partially beneficial.

This demonstration is an initial prototype of a larger effort to create a nutrition dialogue system that automatically extracts food concepts from a user’s spoken meal description. First, the user’s spoken input is recognized by a speech recognizer. Then, the language understanding component uses a semi-Markov conditional random field (CRF) to segment and assign semantic tags to each token in the recognized food diary, and a second CRF associates foods with their corresponding attributes. Finally, the segmented food concepts are shown in matrix form in a table along with potential matches to the USDA and other on-line nutritional databases.

Spoken dialogue systems have been studied for years, yet portability is still one of the biggest challenges in terms of language extensibility, domain scalability, and platform compatibility. In this work, we investigate the portability issue from the language understanding perspective and present the Asgard architecture, a CRF-based (Conditional Random Fields) and crowd-sourcing-centered framework, which supports expert-free development of multilingual dialogue systems and seamless deployment to mobile platforms. Combinations of linguistic and statistical features are employed for multilingual semantic understanding, such as n-grams, tokenization and part-of-speech. English and Mandarin systems in various domains (movie, flight and restaurant) are implemented with the proposed framework and ported to mobile platforms as well, which sheds lights on large-scale speech App development.

Query understanding has been well studied in the areas of information retrieval and spoken language understanding (SLU). There are generally three layers of query understanding: domain classification, user intent detection, and semantic tagging. Classifiers can be applied to domain and intent detection in real systems, and semantic tagging (or slot filling) is commonly defined as a sequence-labeling task - mapping a sequence of words to a sequence of labels. Various statistical features (e. g., n-grams) can be extracted from annotated queries for learning label prediction models; however, linguistic characteristics of queries, such as hierarchical structures and semantic relationships, are usually neglected in the feature extraction process. In this work, we propose an approach that leverages linguistic knowledge encoded in hierarchical parse trees for query understanding. Specifically, for natural language queries, we extract a set of syntactic structural features and semantic dependency features from query parse trees to enhance inference model learning. Experiments on real natural language queries show that augmenting sequence labeling models with linguistic knowledge can improve query understanding performance in various domains.

Spoken language systems often rely on static speech recognizers. When the underlying models are improved on-the-fly, training is usually performed using unsupervised methods. In this work, we explore an alternative approach that uses human computation to provide crowd-supervised training of a deployed system. Although the framework we describe is applicable to any stochastic model for which the training data can be generated by non-experts, we demonstrate its utility on the lexicon and language model of a speech recognizer in a cinema voicesearch domain. We show how an initially shaky system can achieve over a 10% absolute improvement in word error rate (WER) – entirely without expert intervention. We then analyze how these gains were made.

Online streaming companies such as Netflix have become dominant in the media distribution sector. However, such media delivery services often support very rudimentary search, especially for natural language queries. To provide a more natural search interface, we have developed a conversational movie search system, which parses the recognition hypothesis of a spoken query into semantic classes using conditional random fields (CRFs), and then searches an indexed database with the identified semantics. Topic modeling on user-generated content (e.g., movie reviews) is employed for query expansion. Thirteen searching schemas are supported (such as genre, plot, character and soundtrack search). A crowd-sourcing platform was utilized to automatically collect large-scale annotated data for incremental CRF training.

We describe a speech system for commanding robots in human-occupied outdoor military supply depots. To operate in such environments, the robots must be as easy to interact with as are humans, i.e. they must reliably understand ordinary spoken instructions, such as orders to move supplies, as well as commands and warnings, spoken or shouted from distances of tens of meters. These design goals preclude close-talking microphones and “push-to-talk” buttons that are typically used to isolate commands from the sounds of vehicles, machinery and non-relevant speech. We used multiple microphones to provide omnidirectional coverage. A novel voice activity detector was developed to detect speech and select the appropriate microphone to listen to. Finally, we developed a recognizer model that could successfully recognize commands when heard amidst other speech within a noisy environment. When evaluated on speech data in the field, this system performed significantly better than a more computationally intensive baseline system, reducing the effective false alarm rate by a factor of 40, while maintaining the same level of precision.

Academic and Research Staff Prof. K.N. Stevens, Prof. J. Allen, Prof. M. Halle, Prof. S.J. Keyser, Prof. V.W. Zue, A. Andrade1 1 , Dr. D. Bradley 1 , Dr. Martha Danly12 , Dr. F. Grosjean1 3 , Dr. S. Hawkins 14 , Dr. R.E. Hillman 5 , E.B. Holmberg , Dr. A.W.F. Huggins , C. Hume, Dr. H. Kawasaki, Dr. D.H. Klatt, Dr. L.S. Larkey, Dr. John Locke 18, Dr. B. Lyberg"1 , Dr. J.I. Makhou117, Dr. E. Maxwell, Dr. L. Menn1 9, Dr. P. Menyuk 20 , Dr. J.L. Miller 13, Dr. J.M. Pardo Munoz 11, Dr. J.S. Perkell, Dr. P.J. Price, Dr. S. Shattuck-Hufnagel, S.-Q. Wang", T. Watanabe1 1

We describe a multimodal dialogue system for interacting with a home entertainment center via a mobile device. In our working prototype, users may utilize both a graphical and speech user interface to search TV listings, record and play television programs, and listen to music. The developed framework is quite generic, potentially supporting a wide variety of applications, as we demonstrate by integrating a weather forecast application. In the prototype, the mobile device serves as the locus of interaction, providing both a small touchscreen display, and speech input and output; while the TV screen features a larger, richer GUI. The system architecture is agnostic to the location of the natural language processing components: a consistent user experience is maintained regardless of whether they run on a remote server or on the device itself.

In this paper we discuss our research activities in the area of spoken lecture processing. Our goal is to improve the access to on-line audio/visual recordings of academic lectures by developing tools for the processing, transcription, indexing, segmentation, summarization, retrieval and browsing of this media. In this paper, we provide an overview of the technology components and systems that have been developed as part of this project, present some experimental results, and discuss our ongoing and future research plans. Index Terms:spoken lecture processing, spoken document retrieval, audio browsing

We will demonstrate the MIT Spoken Lecture Processing Server and an accompanying lecture browser that students can use to quickly locate and browse lecture segments that apply to their query. We will show how lecturers can upload recorded lectures and companion text material to our server for automatic processing. The server automatically generates a time-aligned word transcript of the lecture which can be downloaded for use within a browser. We will also demonstrate a browser we have created which allows students to quickly locate and browse audio segments that are relevant to their query. These tools can provide students with easier access to audio (or audio/visual) lectures, hopefully improving their educational experience.

In the past decade, lower data storage costs and faster data transfer rates have made it feasible to provide on-line academic lecture material including audio-visual presentations. Such educational resources have the potential to eliminate many space and time constraints from the learning process by allowing people to access quality educational material irrespective of where they are or when they need it. Unlike text however, untranscribed audio data is tedious to browse, making it difficult to utilize the information to its full potential without time-consuming data preparation. In this work, we have developed a set of tools for automatically transcribing and indexing audio lectures as well as a prototype browser for searching and browsing these processed lectures.