Open-domain question answering has been used in a wide range of applications, such as web search and enterprise search, which usually takes clean texts extracted from various formats of documents (e.g., web pages, PDFs, or Word documents) as the information source. However, designing different text extraction approaches is time-consuming and not scalable. In order to reduce human cost and improve the scalability of QA systems, we propose and study an Open-domain Document Visual Question Answering (Open-domain DocVQA) task, which requires answering questions based on a collection of document images directly instead of only document texts, utilizing layouts and visual features additionally. To advance this task, we introduce the first Chinese Open-domain DocVQA dataset called DuReader(vis), containing about 15K question-answering pairs and 158K document images from the Baidu search engine. There are three main challenges in DuReadervis: (1) long document understanding, (2) noisy texts, and (3) multi-span answer extraction. The extensive experiments demonstrate that the dataset is challenging. Additionally, we propose a simple approach that incorporates the layout and visual features, and the experimental results show the effectiveness of the proposed approach. The dataset and code will be publicly available at https://github.com/baidu/DuReader/tree/master/ DuReader-vis.

Knowledge graph embedding has been proposed to embed entities and relations into continuous vector spaces, which can benefit various downstream tasks, such as question answering and recommender systems, etc. A common assumption of existing knowledge graph embedding models is that the relation is a translation vector connecting the embedded head entity and tail entity. However, based on this assumption, the same relation connecting multiple entities may form a circle and lead to mistakes during computing process. To solve this so-called circular relation problem that has been ignored previously, we propose a novel method called ContE ( Cont extualized E mbedding) for knowledge graphs by exploring collaborative relations. Specifically, each collaborative relation combines an explicit relation and a latent relation, where the explicit one is the original relation between two entities, and the latent one is introduced to capture the implicit interactions obtained via the context information of the two entities. With the collaborative relations, the same relations will be embedded varying across different contexts, and the sharing of similar semantics will be guaranteed as well. We conduct extensive experiments in link prediction and triple classification tasks on five benchmark datasets. The experimental results demonstrate that the proposed ContE achieves improvements over several baselines.

A recent discovery showed that large pre-001 trained language models (LM) are capable of 002 encoding knowledge into their parameters, as 003 a result, serving as powerful zero-shot/few-004 shot learners of knowledge-dependent tasks. 005 However, the commonsense knowledge of such 006 models is relatively under-explored despite 007 their importance on various natural language 008 tasks. In this paper, we propose Multi-Sense 009 model that is enriched with commonsense by 010 multi-task learning (MTL) on commonsense 011 question-answering and natural language in-012 ference (NLI) tasks. We hypothesize that su-013 pervision from tasks that require commonsense 014 reasoning ability will implicitly help strengthen 015 the commonsense representation within the 016 model parameters. Empirical results demon-017 strate that the multi-task commonsense enrich-018 ment step is helpful in downstream tasks (i.e., 019 reading comprehension, fact-checking). In ad-020 dition, we demonstrate the strength of Multi-021 Sense in low resource settings by conducting a 022 few-shot learning analysis. 023

Learning to reason in large-scale knowledge graphs has attracted much attention from research communities recently. This paper targets a practical task of multi-hop reasoning in knowledge graphs, which can be applied in various downstream tasks such as question answering, and recommender systems. A key challenge in multi-hop reasoning is to synthesize structural information (e.g., paths) in knowledge graphs to perform deeper reasoning. Existing methods usually focus on connection paths between each entity pair. However, these methods ignore predecessor paths before connection paths and regard entities and relations within every single path as equally important. With our observations, predecessor paths before connection paths can provide more accurate semantic representations. Furthermore, entities and relations in a single path contribute variously to the right answers. To this end, we propose a novel model HiAM (Hierarchical Attention based Model) for knowledge graph multi-hop reasoning. HiAM makes use of predecessor paths to provide more accurate semantics for entities and explores the effects of different granularities. Firstly, we extract predecessor paths of head entities and connection paths between each entity pair. Then, a hierarchical attention mechanism is designed to capture the information of different granularities, including entity/relation-level and path-level features. Finally, multi-granularity features are fused together to predict the right answers. We go one step further to select the most significant path as the explanation for predicted answers. Comprehensive experimental results demonstrate that our method achieves competitive performance compared with the baselines on three benchmark datasets.

Recently, dense passage retrieval has become a mainstream approach to finding relevant information in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder retriever. In order to capture more comprehensive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To implement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. Extensive experiments show that our approach significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions datasets.

Pre-training text representations has recently been shown to significantly improve the state-of-the-art in many natural language processing tasks. The central goal of pre-training is to learn text representations that are useful for subsequent tasks. However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling. In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps. The standard multi-task learning objective adopted in BERT is a special case of our learning algorithm where the depth of meta-train is zero. We study the problem in two settings: unsupervised pre-training and supervised pre-training with different pre-training objects to verify the generality of our approach.Experimental results show that our algorithm brings improvements and learns better initializations for a variety of downstream tasks.

Script learning aims to predict the subsequent event according to the existing event chain. Recent studies focus on event co-occurrence to solve this problem. However, few studies integrate external event knowledge to solve this problem. With our observations, external event knowledge can provide additional knowledge like temporal or causal knowledge for understanding event chain better and predicting the right subsequent event. In this work, we integrate event knowledge from ASER (Activities, States, Events and their Relations) knowledge base to help predict the next event. We propose a new approach consisting of knowledge retrieval stage and knowledge integration stage. In the knowledge retrieval stage, we select relevant external event knowledge from ASER. In the knowledge integration stage, we propose three methods to integrate external knowledge into our model and infer final answers. Experiments on the widely-used Multi- Choice Narrative Cloze (MCNC) task show our approach achieves state-of-the-art performance compared to other methods.

With information explosion on the Internet, only returning ranked documents by search engines cannot satisfy people’s requirements on news events understanding. A more intelligent news events search engine should not only retrieve all related documents about a specific event, but also provide a global view about how the event originates and evolves. In order to solve this challenge, two tasks, event news retrieval and eventline generation should be processed. For event news retrieval, existing approaches mainly focus on the document-level similarity to retrieve related news documents, while external knowledge is not effectively taken into consideration. To this end, we propose a similarity model named Event-Oriented Similarity combining the document-level with the knowledge-level similarity to retrieve news documents related to the specific event. For eventline generation, in order to outline the event structure more accurately, we construct an Event-Oriented Similarity Graph to represent the relationship among retrieved event news documents and develop a community detection algorithm to segment sub-events which are consequently chained into a cohesive eventline. Experimental results on real-world datasets demonstrate that the proposed approach outperforms existing methods.

Commonsense question answering aims to answer questions which require background knowledge that is not explicitly expressed in the question. The key challenge is how to obtain evidence from external knowledge and make predictions based on the evidence. Recent studies either learn to generate evidence from human-annotated evidence which is expensive to collect, or extract evidence from either structured or unstructured knowledge bases which fails to take advantages of both sources simultaneously. In this work, we propose to automatically extract evidence from heterogeneous knowledge sources, and answer questions based on the extracted evidence. Specifically, we extract evidence from both structured knowledge base (i.e. ConceptNet) and Wikipedia plain texts. We construct graphs for both sources to obtain the relational structures of evidence. Based on these graphs, we propose a graph-based approach consisting of a graph-based contextual word representation learning module and a graph-based inference module. The first module utilizes graph structural information to re-define the distance between words for learning better contextual word representations. The second module adopts graph convolutional network to encode neighbor information into the representations of nodes, and aggregates evidence with graph attention mechanism for predicting the final answer. Experimental results on CommonsenseQA dataset illustrate that our graph-based approach over both knowledge sources brings improvement over strong baselines. Our approach achieves the state-of-the-art accuracy (75.3%) on the CommonsenseQA dataset.

Scripts represent knowledge of event sequences that can help text understanding. Script event prediction requires to measure the relation between an existing chain and the subsequent event. The dominant approaches either focus on the effects of individual events, or the influence of the chain sequence. However, only considering individual events will lose much semantic relations within the event chain, and only considering the sequence of the chain will introduce much noise. With our observations, both the individual events and the event segments within the chain can facilitate the prediction of the subsequent event. This paper develops self attention mechanism to focus on diverse event segments within the chain and the event chain is represented as a set of event segments. We utilize the event-level attention to model the relations between subsequent events and individual events. Then, we propose the chain-level attention to model the relations between subsequent events and event segments within the chain. Finally, we integrate event-level and chain-level attentions to interact with the chain to predict what happens next. Comprehensive experiment results on the widely used New York Times corpus demonstrate that our model achieves better results than other state-of-the-art baselines by adopting the evaluation of Multi-Choice Narrative Cloze task.

Multi-turn retrieval-based conversation is an important task for building intelligent dialogue systems. Existing works mainly focus on matching candidate responses with every context utterance on multiple levels of granularity, which ignore the side effect of using excessive context information. Context utterances provide abundant information for extracting more matching features, but it also brings noise signals and unnecessary information. In this paper, we will analyze the side effect of using too many context utterances and propose a multi-hop selector network (MSN) to alleviate the problem. Specifically, MSN firstly utilizes a multi-hop selector to select the relevant utterances as context. Then, the model matches the filtered context with the candidate response and obtains a matching score. Experimental results show that MSN outperforms some state-of-the-art methods on three public multi-turn dialogue datasets.

Data augmentation methods are often applied to prevent overfitting and improve generalization of deep neural network models. Recently proposed contextual augmentation augments labeled sentences by randomly replacing words with more varied substitutions predicted by language model. Bidirectional Encoder Representations from Transformers (BERT) demonstrates that a deep bidirectional language model is more powerful than either an unidirectional language model or the shallow concatenation of a forward and backward model. We propose a novel data augmentation method for labeled sentences called conditional BERT contextual augmentation. We retrofit BERT to conditional BERT by introducing a new conditional masked language model (The term “conditional masked language model” appeared once in original BERT paper, which indicates context-conditional, is equivalent to term “masked language model”. In our paper, “conditional masked language model” indicates we apply extra label-conditional constraint to the “masked language model”.) task. The well trained conditional BERT can be applied to enhance contextual augmentation. Experiments on six various different text classification tasks show that our method can be easily applied to both convolutional or recurrent neural networks classifier to obtain improvement.

Opinion spam has become a widespread problem in social media, where hired spammers write deceptive reviews to promote or demote products to mislead the consumers for profit or fame. Existing works mainly focus on manually designing discrete textual or behavior features, which cannot capture complex global semantics of reviews. Although recent works apply deep learning methods to learn review-level semantic features, their models ignore the impact of the user-level and product-level information on learning review semantics and the inherent user-review-product relationship information. In this paper, we propose a Hierarchical Fusion Attention Network (HFAN) to automatically learn the semantics of reviews from user and product level. Specifically, we design a multiattention unit to extract user(product)-related review information. Then, we use orthogonal decomposition and fusion attention to learn a user, review, and product representation from the review information. Finally, we take the review as a relation between user and product entity and apply TransH to jointly encode this relationship into review representation. Experimental results obtained more than 10% absolute precision improvement over the state-of-the-art performances on four real-world datasets, which show the effectiveness and versatility of the model.

This paper proposes a novel approach to retrieve news articles related to a specific event and generate a storyline to help people understand the event evolution. First, a similarity calculation method is proposed to retrieve news articles related to the specific event, which combines textual similarity, temporal similarity and entity similarity. Then a multi-view attribute graph is constructed to represent the relationship between retrieved articles. Finally, a community detection algorithm is developed to segment and chain subevents in the graph. Experimental results on real-world datasets demonstrate that the proposed approach achieve better results than existing methods.0F0F

An entity on the web can be referred by numerous morphs that are always ambiguous, implicit and informal, which makes it challenging to accurately identify all the morphs corresponding to a specific entity. In this paper, we introduce a novel method based on knowledge graph, which takes advantage of both knowledge reasoning and statistic learning. First, we present a model to build a knowledge graph for the given entity. The knowledge graph integrates the fragmented knowledge on how humans create morphs. Then, the candidate morphs are generated based on the rules summarized from the knowledge graph. At last, we use a classification method to filter the useless candidates and identify the target morphs. The experiments conducted on real world dataset demonstrate efficiency of our proposed method in terms of precision and recall.