Effective disease prediction based on electronic health records (EHR) is an important topic in health informatics. The current methods usually use common deep-learning models for disease prediction. However, it is difficult to fully learn the graphic encounter structure of EHR to improve prediction performance. Moreover, in prediction tasks, chronic kidney disease (CKD) has a poor prognosis due to excessive risk factors and complex comorbidities. Therefore, we propose a CKD onset prediction model called Graph Convolution Synthetic Transformer (GCST) based on EHR, using Fusion Attention Mechanism to solve these challenges. By modifying Transformer, GCST uses Factorized Dense Attention and Medical Local Attention to learn global and local attention, generating Synthetic Attention to learn the potential encounter structure and meaningful medical knowledge of EHR. In addition, we also propose a transfer learning strategy based on sample weighted correction to guide the prediction of GCST in specific low-resource EHR. We conduct sufficient experiments on three datasets to test the performance of GCST. Experiments show that GCST has significant improvement over state-of-the-art models.

Deep neural networks often have huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. Knowledge distillation is one approach to derive compact models from bigger ones. However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. In this paper, we propose proKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. Such projection is implemented by decomposing the training objective into local intermediate targets with approximate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result in a better local optima. Experiments on both image and text datasets show that our proposed proKT consistently achieves the state-of-the-art performance comparing to all existing knowledge distillation methods.

Deep neural networks often have huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. Knowledge distillation is one approach to derive compact models from bigger ones. However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. In this paper, we propose ProKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. Such projection is implemented by decomposing the training objective into local intermediate targets with approximate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result in a better local optima. Experiments on both image and text datasets show that our proposed ProKT consistently achieves superior performance comparing to other existing knowledge distillation methods.

Pre-trained contextual representations like BERT have achieved great success in natural language processing. However, the sentence embeddings from the pre-trained language models without fine-tuning have been found to poorly capture semantic meaning of sentences. In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We first reveal the theoretical connection between the masked language model pre-training objective and the semantic similarity task theoretically, and then analyze the BERT sentence embeddings empirically. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. To address this issue, we propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective. Experimental results show that our proposed BERT-flow method obtains significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks. The code is available at https://github.com/bohanli/BERT-flow.

文本矛盾是自然语言理解的一项基础性问题.目前的研究大多针对矛盾识别任务,而深入文本内部探究矛盾产生原因的工作较少,且缺乏专门的中文矛盾数据集.该文在前人矛盾研究基础上,提出矛盾语块的概念,将其划分为7种类型,并根据标注规范构建了包含16224条数据的中文矛盾语块(CCB)数据集.基于此数据集,利用序列标注及抽取式阅读理解类模型开展矛盾语块边界识别实验,以检验模型对矛盾内部语义信息的理解能力,结果显示阅读理解类模型在该任务上的性能优于序列标注模型.该文通过三个角度对影响语块边界识别的因素进行分析,为文本矛盾后续研究工作提供可靠的数据集和基线模型.

Pre-trained contextual representations like BERT have achieved great success in natural language processing. However, the sentence embeddings from the pre-trained language models without fine-tuning have been found to poorly capture semantic meaning of sentences. In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We first reveal the theoretical connection between the masked language model pre-training objective and the semantic similarity task theoretically, and then analyze the BERT sentence embeddings empirically. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. To address this issue, we propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective. Experimental results show that our proposed BERT-flow method obtains significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks. The code is available at https://github.com/bohanli/BERT-flow.

Recognizing human intentions from electroencephalographic (EEG) signals is attracting extraordinary attention from the artificial intelligence community because of its promise in providing non-muscular forms of communication and control to those with disabilities. So far, studies have explored correlations between specific segments of an EEG signal and an associated intention. However, there are still challenges to be overcome on the road ahead. Among these, vector representations suffer from the enormous amounts of noise that characterize EEG signals. Identifying the correlations between signals from adjacent sensors on a headset is still difficult. Further, research not yet reached the point where learning models can accept decomposed EEG signals to capture the unique biological significance of the six established frequency bands. In pursuit of a more effective intention recognition method, we developed DAMTRNN, a delta attention-based multi-task recurrent neural network, for human intention recognition. The framework accepts divided EEG signals as inputs, and each frequency range is modeled separately but concurrently with a series of LSTMs. A delta attention network fuses the spatial and temporal interactions across different tasks into high-impact features, which captures correlations over longer time spans and further improves recognition accuracy. Comparative evaluations between DAMTRNN and 14 state-of-the-art methods and baselines show DAMTRNN with a record-setting performance of 98.87% accuracy.

When trained effectively, the Variational Autoencoder (VAE) is both a powerful language model and an effective representation learning framework. In practice, however, VAEs are trained with the evidence lower bound (ELBO) as a surrogate objective to the intractable marginal data likelihood. This approach to training yields unstable results, frequently leading to a disastrous local optimum known as posterior collapse. In this paper, we investigate a simple fix for posterior collapse which yields surprisingly effective results. The combination of two known heuristics, previously considered only in isolation, substantially improves held-out likelihood, reconstruction, and latent representation learning when compared with previous state-of-the-art methods. More interestingly, while our experiments demonstrate superiority on these principle evaluations, our method obtains a worse ELBO. We use these results to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling.

In this work, we propose a method for neural dialogue response generation that allows not only generating semantically reasonable responses according to the dialogue history, but also explicitly controlling the sentiment of the response via sentiment labels. Our proposed model is based on the paradigm of conditional adversarial learning; the training of a sentiment-controlled dialogue generator is assisted by an adversarial discriminator which assesses the fluency and feasibility of the response generating from the dialogue history and a given sentiment label. Because of the flexibility of our framework, the generator could be a standard sequence-to-sequence (SEQ2SEQ) model or a more complicated one such as a conditional variational autoencoder-based SEQ2SEQ model. Experimental results using automatic and human evaluation both demonstrate that our proposed framework is able to generate both semantically reasonable and sentiment-controlled dialogue responses.

How to model distribution of sequential data, including but not limited to speech and human motions, is an important ongoing research problem. It has been demonstrated that model capacity can be significantly enhanced by introducing stochastic latent variables in the hidden states of recurrent neural networks. Simultaneously, WaveNet, equipped with dilated convolutions, achieves astonishing empirical performance in natural speech generation task. In this paper, we combine the ideas from both stochastic latent variables and dilated convolutions, and propose a new architecture to model sequential data, termed as Stochastic WaveNet, where stochastic latent variables are injected into the WaveNet structure. We argue that Stochastic WaveNet enjoys powerful distribution modeling capacity and the advantage of parallel training from dilated convolutions. In order to efficiently infer the posterior distribution of the latent variables, a novel inference network structure is designed based on the characteristics of WaveNet architecture. State-of-the-art performances on benchmark datasets are obtained by Stochastic WaveNet on natural speech modeling and high quality human handwriting samples can be generated as well.

This paper proposes a Knowledge Augmented Inference Network (K- AIN) that can effectively incorporate external knowledge into existing neural network models on Natural Language Inference (NLI) task. Different from previous works that use one-hot representations to describe external knowledge, we employ the TransE model to encode various semantic relations extracted from the external Knowledge Base (KB) as distributed relation features. We utilize these distributed relation features to construct knowledge augmented word embeddings and integrate them into the current neural network models. Experimental results show that our model achieves a better performance than the strong baseline on the SNLI dataset and we also surpass the current state-of-the-art models on the SciTail dataset.

Commonsense Reading Comprehension (CRC) is a significantly challenging task, aiming at choosing the right answer for the question referring to a narrative passage, which may require commonsense knowledge inference. Most of the existing approaches only fuse the interaction information of choice, passage, and question in a simple combination manner from a union perspective, which lacks the comparison information on a deeper level. Instead, we propose a Multi-Perspective Fusion Network (MPFN), extending the single fusion method with multiple perspectives by introducing the difference and similarity fusion. More comprehensive and accurate information can be captured through the three types of fusion. We design several groups of experiments on MCScript dataset [11] to evaluate the effectiveness of the three types of fusion respectively. From the experimental results, we can conclude that the difference fusion is comparable with union fusion, and the similarity fusion needs to be activated by the union fusion. The experimental result also shows that our MPFN model achieves the state-of-the-art with an accuracy of 83.52% on the official test set.

Graph sampling, simplying the networks while preserving primary graph characteristics, provides a convenient means for exploring large network. During the last few years a variety of graph sampling algorithms have been proposed, and the evaluation and comparison of the algorithms has witnessed a growing interest. Although different tests have been conducted, an important aspect of graph sampling, namely, uncertainty in graph sampling, has been ignored so far. Additionally, existing studies mainly rely on simple statistical analysis and a few relatively small datasets. They may not be applicable to other more complicated graphs with much larger numbers of nodes and edges. Furthermore, while graph clustering is becoming increasingly important, it is still unknown how different sampling algorithms and their associated uncertainty can impact the subsequent graph analysis, such as graph clustering. In this work, we propose an efficient visual analytics framework for measuring the uncertainty from different graph sampling methods and quantifying the influence of the uncertainty in general graph analysis procedures. A spreadsheet-style visualization with rich user interactions is presented to facilitate visual comparison and analysis of multiple graph sampling algorithms. Our framework helps users gain a better understanding of the graph sampling methods in producing uncertainty information. The framework also makes it possible for users to quickly evaluate graph sampling algorithms and select the most appropriate one for their applications.