Large Language Models (LLMs) have shown remarkable ability to converse with humans and solve a wide range of tasks. They have also been extended to make use of external tools or services through API calls. This is commonly achieved by fine-tuning the model, or with the use of in-context learning, where instructions and descriptions of those external APIs, along with examples of how to call them, are given to the LLM via its prompt. Given the limited context available in the LLM prompt and other latency constraints, scaling up to a large number of tools is challenging and requires the help of an external shortlisting process to prepare instructions and examples from a large number of APIs to a smaller set of relevant ones. In this work, we propose a new way for an LLM to generate the right API calls without the need to shortlist instructions or examples. Rather, we do this by allowing the LLM to hallucinate meaningful output while grounding the generation to an available set of APIs using a finite state machine-based constrained decoding algorithm. We call our approach FARS (FSM-Augmentation to make LLMs hallucinate the Right APIS). FARS allows us to ground LLMs to a large set of APIs with semantically meaningful names without using an external retriever or exemplars. We also demonstrate that with FARS, LLMs can seamlessly switch between conversation and API calling during multi-turn dialogs. We show that this can be achieved without any additional fine-tuning over the standard instruction tuning typically performed to train LLMs. This allows us to pave the way to build a truly powerful AI assistant using LLMs. We demonstrate the effectiveness of FARS for API calling on two public task-oriented API datasets: SNIPS and MultiWOZ, and a very challenging in-house Smart Home Control dataset.

Semantic parsing plays a key role in digital voice assistants such as Alexa, Siri, and Google Assistant by mapping natural language to structured meaning representations. When we want to improve the capabilities of a voice assistant by adding a new domain, the underlying semantic parsing model needs to be retrained using thousands of annotated examples from the new domain, which is time-consuming and expensive. In this work, we present an architecture to perform such domain adaptation automatically, with only a small amount of metadata about the new domain and without any new training data (zero-shot) or with very few examples (few-shot). We use a base seq2seq (sequence-to-sequence) architecture and augment it with a concept encoder that encodes intent and slot tags from the new domain. We also introduce a novel decoder-focused approach to pretrain seq2seq models to be concept aware using Wikidata and use it to help our model learn important concepts and perform well in low-resource settings. We report few-shot and zero-shot results for compositional semantic parsing on the TOPv2 dataset and show that our model outperforms prior approaches in few-shot settings for the TOPv2 and SNIPS datasets.

BACKGROUND:Veterans with a history of traumatic brain injury (TBI) and/or posttraumatic stress disorder (PTSD) may be at increased risk of suicide attempts and other forms of intentional self-harm as compared to veterans without TBI or PTSD.OBJECTIVE:Using administrative data from the US Veterans Health Administration (VHA), we studied associations between TBI and PTSD diagnoses, and subsequent diagnoses of intentional self-harm among US veterans who used VHA health care between 2008 and 2017.METHODS:All veterans with encounters or hospitalizations for intentional self-harm were assigned "index dates" corresponding to the date of the first related visit; among those without intentional self-harm, we randomly selected a date from among the veteran's health care encounters to match the distribution of case index dates over the 10-year period. We then examined the prevalence of TBI and PTSD diagnoses within the 5-year period prior to veterans' index dates. TBI, PTSD, and intentional self-harm were identified using International Classification of Diseases diagnosis and external cause of injury codes from inpatient and outpatient VHA encounters. We stratified analyses by veterans' average yearly VHA utilization in the 5-year period before their index date (low, medium, or high). Variations in prevalence and odds of intentional self-harm diagnoses were compared by veterans' prior TBI and PTSD diagnosis status (TBI only, PTSD only, and comorbid TBI/PTSD) for each VHA utilization stratum. Multivariable models adjusted for age, sex, race, ethnicity, marital status, Department of Veterans Affairs service-connection status, and Charlson Comorbidity Index scores.RESULTS:About 6.7 million veterans with at least two VHA visits in the 5-year period before their index dates were included in the analyses; 86,644 had at least one intentional self-harm diagnosis during the study period. During the periods prior to veterans' index dates, 93,866 were diagnosed with TBI only; 892,420 with PTSD only; and 102,549 with comorbid TBI/PTSD. Across all three VHA utilization strata, the prevalence of intentional self-harm diagnoses was higher among veterans diagnosed with TBI, PTSD, or TBI/PTSD than among veterans with neither diagnosis. The observed difference was most pronounced among veterans in the high VHA utilization stratum. The prevalence of intentional self-harm was six times higher among those with comorbid TBI/PTSD (6778/58,295, 11.63%) than among veterans with neither TBI nor PTSD (21,979/1,144,991, 1.92%). Adjusted odds ratios suggested that, after accounting for potential confounders, veterans with TBI, PTSD, or comorbid TBI/PTSD had higher odds of self-harm compared to veterans without these diagnoses. Among veterans with high VHA utilization, those with comorbid TBI/PTSD were 4.26 (95% CI 4.15-4.38) times more likely to receive diagnoses for intentional self-harm than veterans with neither diagnosis. This pattern was similar for veterans with low and medium VHA utilization.CONCLUSIONS:Veterans with TBI and/or PTSD diagnoses, compared to those with neither diagnosis, were substantially more likely to be subsequently diagnosed with intentional self-harm between 2008 and 2017. These associations were most pronounced among veterans who used VHA health care most frequently. These findings suggest a need for suicide prevention efforts targeted at veterans with these diagnoses.

Semantic parsing is an important NLP problem, particularly for voice assistants such as Alexa and Google Assistant. State-of-the-art (SOTA) semantic parsers are seq2seq architectures based on large language models that have been pretrained on vast amounts of text. To better leverage that pretraining, recent work has explored a reformulation of semantic parsing whereby the output sequences are themselves natural language sentences, but in a controlled fragment of natural language. This approach delivers strong results, particularly for few-shot semantic parsing, which is of key importance in practice and the focus of our paper. We push this line of work forward by introducing an automated methodology that delivers very significant additional improvements by utilizing modest amounts of unannotated data, which is typically easy to obtain. Our method is based on a novel synthesis of four techniques: joint training with auxiliary unsupervised tasks; constrained decoding; self-training; and paraphrasing. We show that this method delivers new SOTA few-shot performance on the Overnight dataset, particularly in very low-resource settings, and very compelling few-shot results on a new semantic parsing dataset.

For over thirty years, researchers have developed and analyzed methods for latent tree induction as an approach for unsupervised syntactic parsing. Nonetheless, modern systems still do not perform well enough compared to their supervised counterparts to have any practical use as structural annotation of text. In this work, we present a technique that uses distant supervision in the form of span constraints (i.e. phrase bracketing) to improve performance in unsupervised constituency parsing. Using a relatively small number of span constraints we can substantially improve the output from DIORA, an already competitive unsupervised parsing system. Compared with full parse tree annotation, span constraints can be acquired with minimal effort, such as with a lexicon derived from Wikipedia, to find exact text matches. Our experiments show span constraints based on entities improves constituency parsing on English WSJ Penn Treebank by more than 5 F1. Furthermore, our method extends to any domain where span constraints are easily attainable, and as a case study we demonstrate its effectiveness by parsing biomedical text from the CRAFT dataset.

Voice Assistants such as Alexa, Siri, and Google Assistant typically use a two-stage Spoken Language Understanding pipeline; first, an Automatic Speech Recognition (ASR) component to process customer speech and generate text transcriptions, followed by a Natural Language Understanding (NLU) component to map transcriptions to an actionable hypothesis. An end-to-end (E2E) system that goes directly from speech to a hypothesis is a more attractive option. These systems were shown to be smaller, faster, and better optimized. However, they require massive amounts of end-to-end training data and in addition, don't take advantage of the already available ASR and NLU training data. In this work, we propose an E2E system that is designed to jointly train on multiple speech-to-text tasks, such as ASR (speech-transcription) and SLU (speech-hypothesis), and text-to-text tasks, such as NLU (text-hypothesis). We call this the Audio-Text All-Task (AT-AT) Model and we show that it beats the performance of E2E models trained on individual tasks, especially ones trained on limited data. We show this result on an internal music dataset and two public datasets, FluentSpeech and SNIPS Audio, where we achieve state-of-the-art results. Since our model can process both speech and text input sequences and learn to predict a target sequence, it also allows us to do zero-shot E2E SLU by training on only text-hypothesis data (without any speech) from a new domain. We evaluate this ability of our model on the Facebook TOP dataset and set a new benchmark for zeroshot E2E performance. We release the audio data collected for the TOP dataset for future research.

Frailty is often cited as a factor influencing oral anticoagulation (OAC) prescription in patients with non-valvular atrial fibrillation (NVAF). We sought to determine the prevalence of frailty and its association with OAC prescription in older veterans with NVAF. We used ICD-9 codes in Veterans Affairs (VA) records and Medicare claims data to identify patients with NVAF and CHA2DS2VASC ≥2 receiving care between February 2010 and September 2015. We examined rates of OAC prescription, further stratified by direct oral anticoagulant (DOAC) or vitamin K antagonist (VKA). Participants were characterized into 3 categories: non-frail, pre-frail, and frail based on a validated 30-item EHR-derived frailty index. We examined relations between frailty and OAC receipt; and frailty and type of OAC prescribed in regression models adjusted for factors related to OAC prescription. Of 308,664 veterans with NVAF and a CHA2DS2VASC score ≥2, 121,839 (39%) were prescribed OAC (73% VKA). The mean age was 77.7 (9.6) years; CHA2DS2VASC and ATRIA scores were 4.6 (1.6) and 5.0 (2.9) respectively. Approximately a third (38%) were frail, another third (32%) were pre-frail, and the remainder were not frail. Veterans prescribed OAC were younger, had higher bleeding risk, and were less likely to be frail than participants not receiving OAC (all p’s<0.001). After adjustment for factors associated with OAC use, pre-frail (OR: 0.89, 95% CI: 0.87–0.91) and frail (OR: 0.66, 95% CI: 0.64–0.68) veterans were significantly less likely to be prescribed OAC than non-frail veterans. Of those prescribed OAC, pre-frail (OR:1.27, 95% CI: 1.22–1.31) and frail (OR: 1.75, 95% CI: 1.67–1.83) veterans were significantly more likely than non-frail veterans to be prescribed a DOAC than a VKA. There are high rates of frailty among older veterans with NVAF. Frailty using an EHR-derived index is associated with decreased OAC prescription.

The deep inside-outside recursive autoencoder (DIORA; Drozdov et al. 2019a) is a selfsupervised neural model that learns to induce syntactic tree structures for input sentences without access to labeled training data. In this paper, we discover that while DIORA exhaustively encodes all possible binary trees of a sentence with a soft dynamic program, its vector averaging approach is locally greedy and cannot recover from errors when computing the highest scoring parse tree in bottom-up chart parsing. To fix this issue, we introduce S-DIORA, an improved variant of DIORA that encodes a single tree rather than a softlyweighted mixture of trees by employing a hard argmax operation and a beam at each cell in the chart. Our experiments show that through fine-tuning a pre-trained DIORA with our new algorithm, we improve the state of the art in unsupervised constituency parsing on the English WSJ Penn Treebank by 2.2 - 6% F1, depending on the data used for fine-tuning.

Virtual assistants such as Amazon Alexa, Apple Siri, and Google Assistant often rely on a semantic parsing component to understand which action(s) to execute for an utterance spoken by its users. Traditionally, rule-based or statistical slot-filling systems have been used to parse "simple" queries; that is, queries that contain a single action and can be decomposed into a set of non-overlapping entities. More recently, shift-reduce parsers have been proposed to process more complex utterances. These methods, while powerful, impose specific limitations on the type of queries that can be parsed; namely, they require a query to be representable as a parse tree. In this work, we propose a unified architecture based on Sequence to Sequence models and Pointer Generator Network to handle both simple and complex queries. Unlike other works, our approach does not impose any restriction on the semantic parse schema. Furthermore, experiments show that it achieves state of the art performance on three publicly available datasets (ATIS, SNIPS, Facebook TOP), relatively improving between 3.3% and 7.7% in exact match accuracy over previous systems. Finally, we show the effectiveness of our approach on two internal datasets.

Pre-trained language models (LM) such as BERT, DistilBERT, and RoBERTa can be tuned for different domains (domain-tuning) by continuing the pre-training phase on a new target domain corpus. This simple domain tuning (SDT) technique has been widely used to create domain-tuned models such as BioBERT, SciBERT and ClinicalBERT. However, during the pretraining phase on the target domain, the LM models may catastrophically forget the patterns learned from their source domain. In this work, we study the effects of catastrophic forgetting on domain-tuned LM models and investigate methods that mitigate its negative effects. We propose continual learning (CL) based alternatives for SDT, that aim to reduce catastrophic forgetting. We show that these methods may increase the performance of LM models on downstream target domain tasks. Additionally, we also show that constraining the LM model from forgetting the source domain leads to downstream task models that are more robust to domain shifts. We analyze the computational cost of using our proposed CL methods and provide recommendations for computationally lightweight and effective CL domain-tuning procedures.

The current state-of-the-art task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders; these models have huge memory footprints. This poses a challenge to their deployment for voice assistants such as Amazon Alexa and Google Assistant on edge devices with limited memory budgets. We propose to learn compositional code embeddings to greatly reduce the sizes of BERT-base and RoBERTa-base. We also apply the technique to DistilBERT, ALBERT-base, and ALBERT-large, three already compressed BERT variants which attain similar state-of-the-art performances on semantic parsing with much smaller model sizes. We observe 95.15% ~ 98.46% embedding compression rates and 20.47% ~ 34.22% encoder compression rates, while preserving greater than 97.5% semantic parsing performances. We provide the recipe for training and analyze the trade-off between code embedding sizes and downstream performances.

Background Scalable and accurate health outcome prediction using electronic health record (EHR) data has gained much attention in research recently. Previous machine learning models mostly ignore relations between different types of clinical data (ie, laboratory components, International Classification of Diseases codes, and medications). Objective This study aimed to model such relations and build predictive models using the EHR data from intensive care units. We developed innovative neural network models and compared them with the widely used logistic regression model and other state-of-the-art neural network models to predict the patient’s mortality using their longitudinal EHR data. Methods We built a set of neural network models that we collectively called as long short-term memory (LSTM) outcome prediction using comprehensive feature relations or in short, CLOUT. Our CLOUT models use a correlational neural network model to identify a latent space representation between different types of discrete clinical features during a patient’s encounter and integrate the latent representation into an LSTM-based predictive model framework. In addition, we designed an ablation experiment to identify risk factors from our CLOUT models. Using physicians’ input as the gold standard, we compared the risk factors identified by both CLOUT and logistic regression models. Results Experiments on the Medical Information Mart for Intensive Care-III dataset (selected patient population: 7537) show that CLOUT (area under the receiver operating characteristic curve=0.89) has surpassed logistic regression (0.82) and other baseline NN models (<0.86). In addition, physicians’ agreement with the CLOUT-derived risk factor rankings was statistically significantly higher than the agreement with the logistic regression model. Conclusions Our results support the applicability of CLOUT for real-world clinical use in identifying patients at high risk of mortality.

We investigate methods to mitigate catastrophic forgetting during domain-specific pretraining of contextual embedding models such as BERT, DistilBERT, and RoBERTa. Recently proposed domain-specific models such as BioBERT, SciBERT and ClinicalBERT are constructed by continuing the pretraining phase on a domain-specific text corpus. Such pretraining is susceptible to catastrophic forgetting, where the model forgets some of the information learned in the general domain. We propose the use of two continual learning techniques (rehearsal and elastic weight consolidation) to improve domain-specific training. Our results show that models trained by our proposed approaches can better maintain their performance on the general domain tasks, and at the same time, outperform domain-specific baseline models on downstream domain tasks.

A lot of Natural Language Processing tasks currently consist of processing a few sentences or paragraphs of text to solve a certain downstream task. Question-Answering in the SQuAD dataset consists of a single question sentence and a paragraph of context. Tasks like named entity recognition and semantic parsing also mostly contains datasets where the inputs are fairly small. In most real world applications however, one would have to go through entire documents to be able to solve a problem. Take the problem of comparing two treatments for a disease that are mentioned in a clinical trial report. One would have to browse through the entire document to find the relevant text to be able to answer the question of which treatment is better. This problem of finding the evidence in large documents for downstream tasks exists in other scenarios as well. Going back to the bio-medical domain, one would have to search through entire clinician reports to find evidence of drug interactions. Researchers have released datasets and have started working on methods to tackle these tasks consisting of long documents. In this project, we aim to tackle one such problem, the task of the treatment comparison. Our task is defined as follows: Given a PubMed article about a randomized control trial, answer a structured question about treatments in the trial. The structured question is a prompt that contains an intervention, comparator, and an outcome. The intervention and comparator are some treatments that are described in the article. The outcome is a patient outcome that is of interest. The answer to the question is one of the three options: significantly increase, significantly decrease, and no significant difference. The task is described with an example in Figure 1. In the example, we Figure 1: Our task: Evidence selection to answer questions about different treatment options, structured as a prompt. We split this task into two steps, the evidence selection part, and the final prediction.

Coreference resolution is the task of identifying all mentions of entities and events in text and placing them into equivalence classes. It is a challenging and unsolved problem which is useful for numerous downstream tasks such as question answering, document summarization, and information retrieval. The goal of our project was two-fold. First, we wanted to obtain theoretical and practical knowledge of the state-of-the-art coreference resolution approaches. Second, we wanted to build our own coreference resolution system which is gender fair. Recently it has been shown (Webster et al., 2018; Zhao et al., 2018; Rudinger et al., 2018) that current state-of-the-art coreference resolution systems are biased by gender, which may impair their applicability to downstream tasks. The community has only just begun to address gender imbalance in coreference resolution accuracy, leaving significant room for improvement. However, (Webster et al., 2018) found that simple syntax-based baselines perform reasonably well on their newly introduced Gendered Ambiguous Pronouns (GAP) dataset. These syntactically motivated baselines also show improved gender balance. This indicated that neural models that incorporate syntax may hold promise. In addition, previous work found that attention-based models implicitly learn to do anaphora resolution (Vaswani et al., 2017; Voita et al., 2018), which suggests that these approaches could yield positive results in debiased coreference resolution. We explored these methods in our project, in addition to the baselines from GAP and neural baselines. We found that a reasonably simple task specific architecture added to BERT was able to outperform the baselines by a large margin. There is still room for improvement, especially in cases involving complicated narrative roles, domain-specific knowledge, or complicated syntactic constructions.

Predicting clinical outcomes using longitudinal electronic health record (EHRs) data is a clinically important and computationally challenging task. In this study, we report CLOUT, an LSTM-based predictive model, which uses a correlational neural network model to identify a latent space representation of three clinical features ICD codes, labs, and medications during a patient’s encounter. Experiments on the MIMIC-III dataset show that CLOUT improves performance on predicting patient mortality, surpassing previous state-of-the-art models.

With advances in solar photovoltaic technology and decreasing costs of panels, solar energy is considered as an economical and sustainable solution for growing energy demands, especially in countries such as India with high insolation levels. However, key challenges exist in large scale adoption of distributed solar generation. Firstly, there are concerns related to breach of voltage regulations and system instabilities caused by distributed generation. Secondly, there are challenges in providing robust communication infrastructure for utilities to actively control and dispatch distributed generation. We address these challenges by designing a decentralised controller iPlug, which involves voltage-sensitive back-off of the level of solar capacity injection into the grid and is inspired by the CSMA protocol from Networking literature. This is done in conjunction with other resources like local batteries and local demand ramp-up resources. iPlug functions are complementary to emerging microinverter technologies and could in principle be incorporated into either the inverter controls or battery and home energy management control systems. We present preliminary results from simulation-based evaluation of iPlug algorithms and compare it with static PV injection strategies.

District heating and cooling systems are becoming increasingly popular to serve the thermal demands of end consumers. In this paper, we investigate the problem of forecasting demand in district heating and cooling systems at the individual consumer level. We are driven by applications such as demand response, where understanding baseline consumption is required in order to set demand curtailment targets. In particular, we propose a data analytics based modeling framework, which uses a context vector based approach for forecasting energy consumption. Our proposed approach is effective in identifying the appropriate context combination that can help explain historical consumption as observed for a given consumer. We provide an evaluation of the methodology on an experimental data set obtained from households in Northern Sweden, where the resulting accuracy of prediction was up to 87%. We also demonstrate an application of the proposed approach to empower a utility company to allocate resources optimally.

This paper proposes an economic incentive strategy for encouraging the installation of solar photovoltaic cells amongst residential consumers in Brunei. Prior literature analyses several options in different geographical contexts, all of which focus on subsidising the effective prorated cost of energy from solar resources. It has also been noted that social effects such as the influence of neighbours and economic status can have a significant effect on the spatio-temporal rate of solar uptake. In this work, we estimate the impact of these factors using a model trained on an empirical data set. Subsequently, we develop a versatile policy architecture for the country of Brunei, based on (i) geographical location within the country, (ii) sociological meta data of residents, and (iii) the economic characteristics and drivers that are unique to Brunei. These characteristics, which include an abundance of fossil fuel reserves, subsidised energy costs, and the need to balance economic and social objectives, impart intriguing properties to the incentive structure.

We describe the problem of aggregating the label predictions of diverse classifiers using a class taxonomy. Such a taxonomy may not have been available or referenced when the individual classifiers were designed and trained, yet mapping the output labels into the taxonomy is desirable to integrate the effort spent in training the constituent classifiers. A hierarchical taxonomy representing some domain knowledge may be different from, but partially mappable to, the label sets of the individual classifiers. We present a heuristic approach and a principled graphical model to aggregate the label predictions by grounding them into the available taxonomy. Our model aggregates the labels using the taxonomy structure as constraints to find the most likely hierarchically consistent class. We experimentally validate our proposed method on image and text classification tasks.