Most scholars focus on the prevalence and democratic effects of (partisan) news exposure. This focus misses large parts of online activities of a majority of politically disinterested citizens. Although political content also appears outside of news outlets and may profoundly shape public opinion, its prevalence and effects are under-studied at scale. This project combines three-wave panel survey data from three countries (total N = 7,266) with online behavioral data from the same participants (over 106M visits). We create a multi-lingual classifier to identify political content both in news and outside (e.g. in shopping or entertainment sites). We find that news consumption is infrequent: just 3.4% of participants' online browsing comprised visits to news sites. Only between 14% (NL) and 36% (US) of these visits were to news about politics. The overwhelming majority of participants' visits were to non-news sites. Although only 1.6\% of those visits related to politics, in absolute terms, citizens encounter politics more frequently outside of news than within news. Out of every 10 visits to political content, 3.4 come from news and 6.6 from non-news sites. Furthermore, exposure to political content outside news domains had the same - and in some cases stronger - associations with key democratic attitudes and behaviors as news exposure. These findings offer a comprehensive analysis of the online political (not solely news) ecosystem and demonstrate the importance of assessing the prevalence and effects of political content in non-news sources.

Kai-Hui Liang, Sam Davidson, Xun Yuan, Shehan Panditharatne, Chun-Yen Chen, Ryan Shea, Derek Pham, Yinghua Tan, Erik Voss, Luke Fryer. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). 2023.

Few studies have analyzed the writing features of second language (L2) alongside heritage language (HL) learners of Spanish. Focusing on the development of writing complexity, we collected data from students at the beginning and at the end of a composition sequence. Comparing the groups at the outset revealed HL students' greater mood complexity. After 14 weeks, both groups improved in tense complexity, mean essay length, and lexical sophistication. Our findings suggest that, given sufficient time, both HL and L2 students can progress in similar ways in at least some measures of linguistic complexity. Nevertheless, they might gain from instruction that facilitates new and diverse vocabulary, and the construction of longer sentences. L2 students specifically would benefit from greater emphasis on the subjunctive. Our study supports the claim that achieving writing complexity takes time and comes in stages, contributing to a more nuanced understanding of L2 and HL writing development.

Although significant progress has been made in developing methods for Grammatical Error Correction (GEC), addressing word choice improvements has been notably lacking and enhancing sentence expressivity by replacing phrases with advanced expressions is an understudied aspect. In this paper, we focus on this area and present our investigation into the task of incorporating the usage of idiomatic expressions in student writing. To facilitate our study, we curate extensive training sets and expert-annotated testing sets using real-world data and evaluate various approaches and compare their performance against human experts.

One of the major impediments to the development of new task-oriented dialogue (TOD) systems is the need for human evaluation at multiple stages and iterations of the development process. In an effort to move toward automated evaluation of TOD, we propose a novel user simulator built using recently developed large pretrained language models (LLMs). In order to increase the linguistic diversity of our system relative to the related previous work, we do not fine-tune the LLMs used by our system on existing TOD datasets; rather we use in-context learning to prompt the LLMs to generate robust and linguistically diverse output with the goal of simulating the behavior of human interlocutors. Unlike previous work, which sought to maximize goal success rate (GSR) as the primary metric of simulator performance, our goal is a system which achieves a GSR similar to that observed in human interactions with TOD systems. Using this approach, our current simulator is effectively able to interact with several TOD systems, especially on single-intent conversational goals, while generating lexically and syntactically diverse output relative to previous simulators that rely upon fine-tuned models. Finally, we collect a Human2Bot dataset of humans interacting with the same TOD systems with which we experimented in order to better quantify these achievements.

This paper reports on progress towards building an online language learning tool to provide learners with conversational experience by using dialog systems as conversation practice partners. Our system can adapt to users' language proficiency on the fly. We also provide automatic grammar error feedback to help users learn from their mistakes. According to our first adopters, our system is entertaining and useful. Furthermore, we will provide the learning technology community a large-scale conversation dataset on language learning and grammar correction. Our next step is to make our system more adaptive to user profile information by using reinforcement learning algorithms.

Currently available grammatical error correction (GEC) datasets are compiled using essays or other long-form text written by language learners, limiting the applicability of these datasets to other domains such as informal writing and conversational dialog. In this paper, we present a novel GEC dataset consisting of parallel original and corrected utterances drawn from open-domain chatbot conversations; this dataset is, to our knowledge, the first GEC dataset targeted to a human-machine conversational setting. We also present a detailed annotation scheme which ranks errors by perceived impact on comprehension, making our dataset more representative of realworld language learning applications. To demonstrate the utility of the dataset, we use our annotated data to fine-tune a state-of-the-art GEC model. Experimental results show the effectiveness of our data in improving GEC model performance in a conversational scenario.

This study examines the impact of two topic-related variables (i.e., valence polarity and everyday-life closeness) on the lexical diversity scores (i.e., MTLD) of learners of L2 Spanish at different proficiency levels. The analysis included 3,045 texts written in response to two pairs of prompts by 1,165 students enrolled in an L2 Spanish program. The first pair of prompts asked learners to narrate an event: prompt 1 focused on a perfect vacation (positive event), while prompt 2 asked participants to tell a terrible story (negative event). The second pair asked to describe a person: prompt 1 required that the subject be famous, thus not close to the writer, whereas prompt 2 required that the subject be special and close to the writer. Results indicate that lexical diversity scores were higher for the texts written about the positive event and the famous subject across all proficiency levels.

Incivility in social media has become a major concern of the public, who perceive uncivil online interactions to be both widespread and increasing. This study provides a descriptive account of incivility dynamics over the past 11 years by examining the trends of incivility in three main categories of social media interactions: political, mixed, and non-political. Using longitudinal data from Reddit that accounts for 95% of the entire Reddit universe across 11 years and relying on the combination of supervised machine learning models and traditional statistical inference, the study found that incivility consistently represents around 10% of total Reddit comments. Additionally, political groups tend to be more uncivil, and discussions in mixed groups that are not overtly political but nevertheless discuss politics are less uncivil than in political groups. We also found that the fluctuations of incivility correspond to offline events and platform-specific policies.

We explore the application of state-of-the-art NER algorithms to ASR-generated call center transcripts. Previous work in this domain focused on the use of a BiLSTM-CRF model which relied on Flair embeddings; however, such a model is unwieldy in terms of latency and memory consumption. In a production environment, end users require low-latency models which can be readily integrated into existing pipelines. To that end, we present two different models which can be utilized based on the latency and accuracy requirements of the user. First, we propose a set of models which utilize state-of-the-art Transformer language models (RoBERTa) to develop a high-accuracy NER system trained on custom annotated set of call center transcripts. We then use our best-performing Transformer-based model to label a large number of transcripts, which we use to pretrain a BiLSTM-CRF model and further fine-tune on our annotated dataset. We show that this model, while not as accurate as its Transformer-based counterpart, is highly effective in identifying items which require redaction for privacy law compliance. Further, we propose a new general annotation scheme for NER in the call-center environment.

Gunrock 2.0 is built on top of Gunrock with an emphasis on user adaptation. Gunrock 2.0 combines various neural natural language understanding modules, including named entity detection, linking, and dialog act prediction, to improve user understanding. Its dialog management is a hierarchical model that handles various topics, such as movies, music, and sports. The system-level dialog manager can handle question detection, acknowledgment, error handling, and additional functions, making downstream modules much easier to design and implement. The dialog manager also adapts its topic selection to accommodate different users' profile information, such as inferred gender and personality. The generation model is a mix of templates and neural generation models. Gunrock 2.0 is able to achieve an average rating of 3.73 at its latest build from May 29th to June 4th.

In this paper we present an NLP-based approach for tracking the evolution of written language competence in L2 Spanish learners using a wide range of linguistic features automatically extracted from students' written productions. Beyond reporting classification results for different scenarios, we explore the connection between the most predictive features and the teaching curriculum, finding that our set of linguistic features often reflects the explicit instruction that students receive during each course.

Incivility is not only prevalent on online social media platforms, but also has concrete effects on individual users, online groups, and the platforms themselves. Given the prevalence and effects of online incivility, and the challenges involved in human-based incivility detection, it is urgent to develop validated and versatile automatic approaches to identifying uncivil posts and comments. This project advances both a neural, BERT-based classifier as well as a logistic regression classifier to identify uncivil comments. The classifier is trained on a dataset of Reddit posts, which are annotated for incivility, and further expanded using a combination of labeled data from Reddit and Twitter. Our best performing model achieves an F1 of 0.802 on our Reddit test set. The final model is not only applicable across social media platforms and their distinct data structures, but also computationally versatile, and - as such - ready to be used on vast volumes of online data. All trained models and annotated data are made available to the research community.

This paper presents the Corpus of Written Spanish of L2 and Heritage Speakers (COWS-L2H), a large corpus of compositions written by North American university students learning Spanish. The goals of this work are to (1) build a large corpus of Spanish learner writing that provides samples of written data from Spanish learners in the context of a North American university, (2) to contribute corpus data collected not only from second language (L2) learners of Spanish but also from learners of Spanish as a heritage language (SHL), and (3) to develop one of the few Spanish learner corpora to provide longitudinal data.

The development of effective NLP tools for the L2 classroom depends largely on the availability of large annotated corpora of language learner text. While annotated learner corpora of English are widely available, large learner corpora of Spanish are less common. Those Spanish corpora that are available do not contain the annotations needed to facilitate the development of tools beneficial to language learners, such as grammatical error correction. As a result, the field has seen little research in NLP tools designed to benefit Spanish language learners and teachers. We introduce COWS-L2H, a freely available corpus of Spanish learner data which includes error annotations and parallel corrected text to help researchers better understand L2 development, to examine teaching practices empirically, and to develop NLP tools to better serve the Spanish teaching community. We demonstrate the utility of this corpus by developing a neural-network based grammatical error correction system for Spanish learner writing.

Dependency parsing of conversational input can play an important role in language understanding for dialog systems by identifying the relationships between entities extracted from user utterances. Additionally, effective dependency parsing can elucidate differences in language structure and usage for discourse analysis of human-human versus human-machine dialogs. However, models trained on datasets based on news articles and web data do not perform well on spoken human-machine dialog, and currently available annotation schemes do not adapt well to dialog data. Therefore, we propose the Spoken Conversation Universal Dependencies (SCUD) annotation scheme that extends the Universal Dependencies (UD) (Nivre et al., 2016) guidelines to spoken human-machine dialogs. We also provide ConvBank, a conversation dataset between humans and an opendomain conversational dialog system with SCUD annotation. Finally, to demonstrate the utility of the dataset, we train a dependency parser on the ConvBank dataset. We demonstrate that by pre-training a dependency parser on a set of larger public datasets and finetuning on ConvBank data, we achieved the best result, 85.05% unlabeled and 77.82% labeled attachment accuracy.

Gunrock is the winner of the 2018 Amazon Alexa Prize, as evaluated by coherence and engagement from both real users and Amazon-selected expert conversationalists. We focus on understanding complex sentences and having in-depth conversations in open domains. In this paper, we introduce some innovative system designs and related validation analysis. Overall, we found that users produce longer sentences to Gunrock, which are directly related to users' engagement (e.g., ratings, number of turns). Additionally, users' backstory queries about Gunrock are positively correlated to user satisfaction. Finally, we found dialog flows that interleave facts and personal opinions and stories lead to better user satisfaction.