Constrained decoding approaches aim to control the meaning or style of text generated by the pre-trained large language models (LLMs or also PLMs) for various tasks at inference time. However, these methods often guide plausible continuations by greedily and explicitly selecting targets. Though fulfilling the task requirements, these methods may overlook certain general and natural logics that humans would implicitly follow towards such targets. Inspired by cognitive dual-process theory, in this work, we propose a novel decoding framework DECIDER where the base LLMs are equipped with a First-Order Logic (FOL) reasoner to express and evaluate the rules, along with a decision function that merges the outputs of both systems to guide the generation. Unlike previous constrained decodings, DECIDER transforms the encouragement of target-specific words into all words that satisfy several high-level rules, enabling us to programmatically integrate our logic into LLMs. Experiments on CommonGen and PersonaChat demonstrate that DECIDER effectively follows given FOL rules to guide LLMs in a more human-like and logic-controlled manner.

Lexicon-based constrained decoding approaches aim to control the meaning or style of the generated text through certain target concepts. Existing approaches over-focus the targets themselves, leading to a lack of high-level reasoning about how to achieve them. However, human usually tackles tasks by following certain rules that not only focuses on the targets but also on semantically relevant concepts that induce the occurrence of targets. In this work, we present DECIDER, a rule-controllable decoding strategy for constrained language generation inspired by dual-system cognitive theory. Specifically, in DECIDER, a pre-trained language model (PLM) is equiped with a logic reasoner that takes high-level rules as input. Then, the DECIDER allows rule signals to flow into the PLM at each decoding step. Extensive experimental results demonstrate that DECIDER can effectively follow given rules to guide generation direction toward the targets in a more human-like manner.

The task of Points of Interest (POI) description generation aims to generate an objective and informative description for a given POI based on POI-related information. High-quality descriptions can better guide users and improve the performance of POI-related recommendation systems. A practical POI description generation model should have effective multimodal fusion and information encoding methods suitable for various data forms. However, due to model structure and data utilization limitations, the previous method is challenging to meet the above requirements. We propose a novel Guide-Select multimodal fusion method that combines the guiding and selecting process to fuse various POI-related information efficiently. In addition, we propose a reasonable review encoding method and a category encoding method that has strong generalization ability. We integrate these methods into our Guide-Select Generation Model (GSGM). Experimental results demonstrate that our model significantly outperforms the state-of-the-art model while having a strong generalization ability on category information.

Visual storytelling is the task of generating a related story fora given image sequence, which has received significant attention. However, using general RNNs (such as LSTM and GRU) as the decoder limit the per-formance of the models in this task. This is because they can not differentiate different types of informa-tion representations. In addition, optimizing the probabilities of subsequent words conditioned on the previous ground-truth sequences can cause error accumulation during inference. Moreover, the existing method of alleviating error accumulation based on replacing reference words does not take into account the different effects of each word. To address the above problems, we propose a modified neural network named AOG-LSTM and a modified training strategy named ARS, respectively. AOG-LSTM can adaptatively pay appropriate attention to different information representations within it when predicting different words. During training, ARS replaces some words in the reference sentences with model predictions similar to the existing method. However, we utilize the selection network and selection strategy to select more appropriate words for the replacement to better improve the model. Experiments on the VIST Dataset demonstrate that our model outperforms several strong baselines on the most commonly used metrics.& COPY; 2023 Elsevier B.V. All rights reserved.

Recently, there has been a surge in the use of generated data to enhance the performance of downstream models, largely due to the advancements in pre-trained language models. However, most prevailing methods trained generative and discriminative models in isolation, which left them unable to adapt to changes in each other. These approaches lead to generative models that are prone to deviating from the true data distribution and providing limited benefits to discriminative models. While some works have proposed jointly training generative and discriminative language models, their methods remain challenging due to the non-differentiable nature of discrete data. To overcome these issues, we introduce a self-consistent learning framework in the text field that involves training a discriminator and generator cooperatively in a closed-loop manner until a scoring consensus is reached. By learning directly from selected samples, our framework are able to mitigate training instabilities such as mode collapse and non-convergence. Extensive experiments on four downstream benchmarks, including AFQMC, CHIP-STS, QQP, and MRPC, demonstrate the efficacy of the proposed framework.

Generating text from abstract meaning repre-sentation (AMR) is a challenging task. Graph-to-sequence (Graph2Seq-based) methods and pre-trained-based methods are proposed for this task. However, both methods have advantages and disadvantages. Graph2Seq-based methods can make use of the structural information of the graph but not the extra knowledge, while pre-trained-based methods have the advantage of the utilization of extra knowledge but may lose the structural information. In addition, both types of methods often suffer from the under- and over-translation problem. To address these prob-lems, we propose a graph structure reconstruction and coverage enhanced model for this task. The graph structure reconstruction uses two auxiliary objectives, relationship prediction and distance prediction of nodes in AMR graphs to enhance the information of graph structure. In addition, we design a coverage mechanism to solve the problem of information under-translation or over-translation in AMR-to-text generation. Experimental results on three datasets show that our proposed method outperforms the existing methods significantly.

Distractor generation, which aims to generate the wrong option part of multi-choice questions, has been proposed to assist educators to test the examinees’ reading comprehension and reasoning ability. Recently, some Seq2Seq-based models have been proposed to solve the task of automatic distractor generation. However, they did not make full use of context information to generate distractors. In order to overcome this shortcoming, we propose a context reasoning attention network for distractor generation. Experimental results show that our model outperforms state-of-the-art baselines and improves the distractive ability of the generated distractors in terms of automatic evaluation and human evaluation.

Maintaining a consistent persona is essential for building a human-like conversational model. However, the lack of attention to the partner makes the model more egocentric: they tend to show their persona by all means such as twisting the topic stiffly, pulling the conversation to their own interests regardless, and rambling their persona with little curiosity to the partner. In this work, we propose COSPLAY(COncept Set guided PersonaLized dialogue generation Across both partY personas) that considers both parties as a "team": expressing self-persona while keeping curiosity toward the partner, leading responses around mutual personas, and finding the common ground. Specifically, we first represent self-persona, partner persona and mutual dialogue all in the concept sets. Then, we propose the Concept Set framework with a suite of knowledge-enhanced operations to process them such as set algebras, set expansion, and set distance. Based on these operations as medium, we train the model by utilizing 1) concepts of both party personas, 2) concept relationship between them, and 3) their relationship to the future dialogue. Extensive experiments on a large public dataset, Persona-Chat, demonstrate that our model outperforms state-of-the-art baselines for generating less egocentric, more human-like, and higher quality responses in both automatic and human evaluations.

Automatic comment generation is a special and challenging task to verify the model ability on news content comprehension and language generation. Comments not only convey salient and interesting information in news articles, but also imply various and different reader characteristics which we treat as the essential clues for diversity. However, most of the comment generation approaches only focus on saliency information extraction, while the reader-aware factors implied by comments are neglected. To address this issue, we propose a unified reader-aware topic modeling and saliency information detection framework to enhance the quality of generated comments. For reader-aware topic modeling, we design a variational generative clustering algorithm for latent semantic learning and topic mining from reader comments. For saliency information detection, we introduce Bernoulli distribution estimating on news content to select saliency information. The obtained topic representations as well as the selected saliency information are incorporated into the decoder to generate diversified and informative comments. Experimental results on three datasets show that our framework outperforms existing baseline methods in terms of both automatic metrics and human evaluation. The potential ethical issues are also discussed in detail.

Recall the classical text generation works, the generation framework can be briefly divided into two phases: \textbf{idea reasoning} and \textbf{surface realization}. The target of idea reasoning is to figure out the main idea which will be presented in the following talking/writing periods. Surface realization aims to arrange the most appropriate sentence to depict and convey the information distilled from the main idea. However, the current popular token-by-token text generation methods ignore this crucial process and suffer from many serious issues, such as idea/topic drift. To tackle the problems and realize this two-phase paradigm, we propose a new framework named Sentence Semantic Regression (\textbf{SSR}) based on sentence-level language modeling. For idea reasoning, two architectures \textbf{SSR-AR} and \textbf{SSR-NonAR} are designed to conduct sentence semantic regression autoregressively (like GPT2/3) and bidirectionally (like BERT). In the phase of surface realization, a mixed-granularity sentence decoder is designed to generate text with better consistency by jointly incorporating the predicted sentence-level main idea as well as the preceding contextual token-level information. We conduct experiments on four tasks of story ending prediction, story ending generation, dialogue generation, and sentence infilling. The results show that SSR can obtain better performance in terms of automatic metrics and human evaluation.

Current approaches to empathetic response generation focus on learning a model to predict an emotion label and generate a response based on this label, and have achieved promis-ing results. However, the emotion cause, an essential factor for empathetic responding, is ignored. The emotion cause is a stimulus for human emotions. Recognizing the emotion cause is helpful to better understand human emotions to generate more empathetic responses. To this end, we propose a novel framework that improves empathetic response generation by recognizing emotion cause in conversations. Specifically, an emotion reasoner is designed to predict a context emotion label and a sequence of emotion cause-oriented labels, which indicate whether the word is related to the emotion cause. Then we devise both hard and soft gated attention mech-anisms to incorporate the emotion cause into response generation. Experiments show that incorporating emotion cause information improves the performance of the model on both emotion recognition and response generation.

Paraphrase generation aims to rewrite a text with different words while keeping the same meaning. Previous work performs the task based solely on the given dataset while ignoring the availability of external linguistic knowledge. However, it is intuitive that a model can generate more expressive and diverse paraphrase with the help of such knowledge. To fill this gap, we propose Knowledge-Enhanced Paraphrase Network (KEPN), a transformer-based framework that can leverage external linguistic knowledge to facilitate paraphrase generation. (1) The model integrates synonym information from the external linguistic knowledge into the paraphrase generator, which is used to guide the decision on whether to generate a new word or replace it with a synonym. (2) To locate the synonym pairs more accurately, we adopt an incremental encoding scheme to incorporate position information of each synonym. Besides, a multi-task architecture is designed to help the framework jointly learn the selection of synonym pairs and the generation of expressive paraphrase. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art approaches in terms of both automatic and human evaluation.

In this paper, we focus on essay generation, which aims at generating an essay (a paragraph) according to a set of topic words. Automatic essay generation can be applied to many scenarios to reduce human workload. Recently the recurrent neural networks (RNN) based methods are proposed to solve this task. However, the RNN-based methods suffer from incoherence problem and duplication problem. To overcome these shortcomings, we propose a self-attention and retrieval enhanced neural network for essay generation. We retrieve sentences relevant to topic words from corpus as material to assist in generation to alleviate the duplication problem. To improve the coherence of essays, the self-attention based encoders are applied to encode topic and material, and the self-attention based decoder are used to generate essay respectively. The final essay is generated under the guidance of topic and material. Experimental results on a real essay dataset show that our model outperforms state-of-the-art baselines according to automatic evaluation and human evaluation.

Abstractive summarization is a renewed and challenging task of document summarization. Recently, neural networks, especially attentional encoder-docoder architecture, have achieved impressive progress in abstractive document summarization. However, the saliency of summary, which is one of the key factors for document summarization, still needs improvement. In this paper, we propose Topic Attentional Neural Network (TANN) which incorporates topic information into neural networks to tackle this issue. Our model is based on attentional sequence-to-sequence structure but has paired encoders and paired attention mechanisms to deal with original document and topic information in parallel. Moreover, we propose a novel selection method called topic selection. This method uses topic information to improve the standard selection method of beam search and chooses a better candidate as the final summary. We conduct experiments on the CNN/Daily Mail dataset. The results show our model obtains higher ROUGE scores and achieves a competitive performance compared with the state-of-the-art abstractive and extractive models. Human evaluation also demonstrates our model is capable of generating summaries with more informativeness and readability.

Many existing rating-based recommendation algorithms have achieved relative success. However, the real-world datasets are extremely sparse and most rating-based algorithms are still suffering from the data sparsity problem. Along with integer-valued ratings, we consider that the user-generated review is also an important user feedback. Furthermore, compared with the traditional recommendation algorithms which have the limited ability to learn the distributions of ratings and reviews simultaneously, the generative adversarial networks can learn better representations for data. In this paper, we propose Rating and Review Generative Adversarial Networks (RRGAN), an innovative framework for recommendation, in which the generative model and discriminative model play a minimax game. Specifically, the generative model predicts the ratings of topN list for users or items based on reviews, while the discriminative model aims to distinguish the predicted ratings from real ratings. With the competition between these two models, RRGAN improves the ability of understanding users and items based on ratings and reviews. We introduce the user profiles, item representations and ratings into a matrix factorization model to predict the top-N list for the users. In addition, we study three different architectures to learn reasonable user profiles and item representations based on ratings and reviews to achieve better recommendations. To evaluate the performance of our model, we conduct the extensive experiments on three real-world amazon datasets in three parts, which are top-N recommendation analysis, case study and long-tail users analysis. The experimental results show that our method significantly outperforms various state-of-the-art methods, including LFM, LambdaFM, HFT, DeepCoNN and IRGAN methods.

There are more and more online sites that allow users to express their sentiments by writing reviews. Recently, researchers have paid attention to review generation. They generate review text under specific contexts, such as rating, user ID or product ID. The encoder-attention-decoder based methods achieve impressive performance in this task. However, these methods do not consider user preference when generating reviews. Only considering numeric contexts such as user ID or product ID, these methods tend to generate generic and boring reviews, which results in a lack of diversity when generating reviews for different users or products. We propose a user preference-aware review generation model to take account of user preference. User preference reflects the characteristics of the user and has a great impact when the user writes reviews. Specifically, we extract keywords from users’ reviews using a score function as user preference. The decoder generates words depending on not only the context vector but also user preference when decoding. Through considering users’ preferred words explicitly, we generate diverse reviews. Experiments on a real review dataset from Amazon show that our model outperforms state-of-the-art baselines according to two evaluation metrics.

With the development of recurrent neural networks (RNN), various natural language generation (NLG) tasks have boomed in the past few years, such as response generation in conversation and poetry generation. However, automatic generation of news comments is anew, challenging and not well-studied task in NLG. Different from other NLG tasks, this task requires the contextual relevance between comments and news. In addition, we need to generate diversified comments, because different people usually have different opinions on the same news in the real world. In this paper, we propose a gated attention neural network model (GANN) to generate news comments. To address the problem of contextual relevance, we introduce the gated attention mechanism to use news context self-adaptively and selectively. To ensure the diversity of comments, we use random sample and relevance control to generate comments with different topics and degrees of relevance. Moreover, we apply generative adversarial nets to improve GANN. Automatic evaluation with perplexity score reveals that GANN outperforms the existing comment generation methods. Human evaluation proves that the generated news comments are close to human comments.

Relation detection is a key step in Knowledge Base Question Answering (KBQA), but far from solved due to the significant differences between questions and relations. Previous studies usually treat relation detection as a text matching task, and mainly focus on reducing the detection error with better representations of KB relations. However, the understanding of questions is also important since they are generally more varied. And the text pair representation requires improvement because KB relations are not always counterparts of questions. In this paper, we propose a novel system with enhanced question understanding and representation processes for KB relation detection (QURRD). We design a KBQA-specific slot filling module based on Bi-LSTM-CRF for question understanding. Besides, with two CNNs for modeling and matching text pairs respectively, QURRD obtains richer question-relation representations for semantic analysis, and achieves better performance through learning from multiple tasks. We conduct experiments on both single-relation (Simple-Questions) and multi-relation (WebQSP) benchmarks. Results show that QURRD is robust against the diversity of questions and outperforms the state-of-the-art system on both tasks.

SummaryRecently, microblog sites such as Twitter attract a great deal of attention as an information resource for topic detection task. Most of existing feature‐pivot topic detection algorithms in Twitter just take a single feature into account rather than multiple features. Thus, these methods always only detect the topics related to the single feature and miss some important topics, which causes a relatively low performance. In this paper, we build a flexible term representation framework for feature‐pivot topic detection based on four features. A Learning‐based Topic Detection using Multiple Features (LTDMF) method is proposed to improve the performance of topic detection. We define a correlation function based on a specific neural network to integrate various features. A Hierarchical Agglomerative Clustering (HAC) algorithm is applied to cluster terms as topics. Based on multiple features, LTDMF detects all types of topics and improves the accuracy of topic detection to solve the problem of missing topics. Experiments show that LTDMF gets a better performance compared with several baseline methods in terms of precision and recall.