This tutorial will describe various aspects of scaling up language technologies to many of the world’s languages by describing the latest research in Massively Multilingual Language Models (MMLMs). We will cover topics such as data collection, training and fine-tuning of models, Responsible AI issues such as fairness, bias and toxicity, linguistic diversity and evaluation in the context of MMLMs, specifically focusing on issues in non-English and low-resource languages. Further, we will also talk about some of the real-world challenges in deploying these models in language communities in the field. With the performance of MMLMs improving in the zero-shot setting for many languages, it is now becoming feasible to use them for building language technologies in many languages of the world, and this tutorial will provide the computational linguistics community with unique insights from the latest research in multilingual models.
Wenting Zhao, Mor Geva, Bill Yuchen Lin, Michihiro Yasunaga, Aman Madaan, Tao Yu. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts). 2023.
This tutorial targets researchers and practitioners who are interested in ML technologies for NLP from indirect supervision. In particular, we will present a diverse thread of indirect supervision studies that try to answer the following questions: (i) when and how can we provide supervision for a target task T, if all we have is data that corresponds to a “related” task T′? (ii) humans do not use exhaustive supervision; they rely on occasional feedback, and learn from incidental signals from various sources; how can we effectively incorporate such supervision in machine learning? (iii) how can we leverage multi-modal supervision to help NLP? To the end, we will discuss several lines of research that address those challenges, including (i) indirect supervision from T ′ that handles T with outputs spanning from a moderate size to an open space, (ii) the use of sparsely occurring and incidental signals, such as partial labels, noisy labels, knowledge-based constraints, and cross-domain or cross-task annotations—all having statistical associations with the task, (iii) principled ways to measure and understand why these incidental signals can contribute to our target tasks, and (iv) indirect supervision from vision-language signals. We will conclude the tutorial by outlining directions for further investigation.
Conversational systems are envisioned to provide social support or functional service to human users via natural language interactions. Conventional conversation researches mainly focus on the responseability of the system, such as dialogue context understanding and response generation, but overlooks the design of an essential property in intelligent conversations, i.e., goal awareness. The awareness of goals means the state of not only being responsive to the users but also aware of the target conversational goal and capable of leading the conversation towards the goal, which is a significant step towards higher-level intelligence and artificial consciousness. It can not only largely improve user engagement and service efficiency in the conversation, but also empower the system to handle more complicated conversation tasks that involve strategical and motivational interactions. In this tutorial, we will introduce the recent advances on the design of agent’s awareness of goals in a wide range of conversational systems.
An increasingly large percentage of natural language processing (NLP) tasks center around the generation of text from probabilistic language models.Despite this trend, techniques for improving or specifying preferences in these generated texts rely mostly on intuition-based heuristics.Further, there lacks a unified presentation of their motivations, practical implementation, successes and pitfalls.Practitioners must, therefore, choose somewhat blindly between generation algorithms-like top-p sampling or beam search-which can lead to wildly different results.At the same time, language generation research continues to criticize and improve the standard toolboxes, further adding entropy to the state of the field.In this tutorial, we will provide a centralized and cohesive discussion of critical considerations when choosing how to generate from a language model.We will cover a wide range of empirically-observed problems (like degradation, hallucination, repetition) and their corresponding proposed algorithmic solutions from recent research (like topp sampling and its successors).We will then discuss a subset of these algorithms under a unified light; most stochastic generation strategies can be framed as locally adapting the probabilities of a model to avoid failure cases.Finally, we will then cover methods in controlled generation, that go beyond just ensuring coherence to ensure text exhibits specific desired properties.We aim for NLP practitioners and researchers to leave our tutorial with a unified framework which they can use to evaluate and contribute to the latest research in language generation.
Retrieval-based language models (LMs) have shown impressive performance on diverse NLP tasks. In this tutorial, we will provide a comprehensive and coherent overview of recent advances in retrieval-based LMs. We will start by providing preliminaries covering the foundation of LMs (e.g., masked LMs, autoregressive LMs) and retrieval systems (e.g., nearest-neighbor search). We will then detail recent progress in retrieval-based models, focusing on their model architectures and learning approaches. Finally, we will show how retrieval-based LMs are adapted to downstream applications, and extended to multilingual and multi-modal settings. Finally, we will use an exercise to showcase the effectiveness of retrieval-based LMs.
How to efficiently interconnect ubiquitous wireless sensor networks (WSNs) and the Internet becomes an important challenge of Internet of Things. In this paper, we explore a route of information-centric networking (ICN) to solve this challenge and propose an open interconnection framework named GlobeSen. To overcome the problem that traditional ICN solution (such as NDN) is not suitable for resource-constrained WSNs, we present a new implementation of NDN, NDNs, for WSNs. Specifically, by extracting the spatio-temporal information and data type information of interest, we construct a globally unique name structure, and exploit the spatio-temporal relation operation as the matching method. Based on the new naming strategy and matching method, we further design packet forwarding and routing schemes. Moreover, we also develop a gateway system, NDNt, for protocol translating and an application, SenBrowser, to provide a user-friendly interface for generating interests and illustrating the returned sensory data. We implement a proof-of-concept prototype based on TelosB sensor nodes and an ARM development board, and conduct a series of experiments to evaluate the performance of GlobeSen.
This paper presents Botzone, a competitive and interactive platform for game AI education, aiming to simplify the teaching process of game AI courses and inspire self-study AI learners. Botzone is a universal online game AI platform, designed to evaluate different implementations of game AI by applying them to agents in a variety of games and compete with each other. It has been successfully used in various AI competition and courses in practice, and has the expandability to support more games and languages, as well as further usages such as researching machine learning on game AI. In this paper, we describe the structure and features of Botzone, and provide case studies of instances of applications.
Calculating fine-grained link metrics by using aggregated path measurements, known as network tomography, is an effective and efficient way to facilitate various network operations, such as network monitoring, load balancing, and fault diagnosis. Recently, there is a growing interest in the monitor placement problem that ensures link identifiability in a network with link failures. Unfortunately, existing work either assumes an ideal failure prediction model where all failures can be predicted perfectly, or makes pessimistic assumptions that all failures are unpredictable. In this paper, we study the problem of placing a minimum number of monitors to identify additive link metrics from end-to-end measurements among monitors with considering both predictable and unpredictable link failures. We propose a set of robust monitor placement algorithms with different performance-complexity tradeoffs to solve this tomography problem. In particular, we show that the optimal (i.e., minimum) monitor placement is the solution to a hitting set problem, for which we provide a polynomial-time algorithm to construct the input. We formally prove that the proposed algorithms can guarantee network identifiability against failures based on graph theory. Trace-driven evaluation results show the effectiveness of our algorithms.
Keyword search is a widely popular mechanism for query processing that alleviates users from understanding complex data structures and learning query languages. Existing keyword search systems are designed and tuned for one specific data model. In big data era, data is usually resident in heterogeneous data sources including unstructured data, semi-structured data and structured data. In order to obtain complete and meaningful results, a system is required to perform keyword search queries upon diverse data sources rather than just one type of data sources. One possible approach is to send the keyword query to all the systems simultaneously, then integrate the returned answers from each data source to obtain final answers. However, answers from different data sources have different formats and may contain duplicates, which yields the heterogeneous search challenge. In this paper we introduce a heterogeneous keyword search system that facilitates integrating answers from diverse data sources. We present several detailed design challenges and also share our preliminary thoughts to the challenges. In particular, we (i) propose a unified result format entity-relationship pattern (ERP), (ii) define new ranking functions, (iii) build native index structures, (iv) provide an efficient global top-k processing algorithm, and (v) introduce the fuzzy entity mapping problem. We have built a prototype HKSearch based on our current solutions to support heterogeneous keyword search.
Flash memory has been used as storage media on a wide spectrum of devices, such as sensor nodes, mobile devices, portable computers and enterprise servers. However, due to the unique characteristics of flash memory, especially the erase-before-write limitation, the design of flash-optimized index structures becomes an important and challenging problem. In this paper, we propose new index structure and optimization techniques to address this problem. Our key contributions include: (1) An innovative indexing framework which combines a write-optimized component with a read-optimized component to improve both write and read performance; (2) A flash-optimized index structure, called FlashSkipList, which can efficiently support dynamic updates and adapt to various workloads and flash devices; (3) Several optimization techniques for flash-friendly memory and storage management. Experimental evaluation shows FlashSkipList achieves high efficiency across a wide range of workloads, flash devices and memory constraints, significantly outperforms state-of-the-art flash-optimized indexing schemes.
As is widely known, the electricity industry is closely connected to the economic development. Lots of work has focused on the regression relationship between the economic metrics and electricity-related indexes. Studies on this relationship have great significance on economic predication and regulation. Obviously, the economic and electricity data can be treated as time series data, which is called trajectory in this paper. The similarity of two trajectories could reflect how much two metrics change in the same tendency. We proposed a new trajectory similarity, which based on the area enclosed by two trajectories. Different to the traditional polygon area, we keep one trajectory stay, and move another trajectory vertically seeking for the least area. Meanwhile, some economic activities would last for some time. So, there exists some time lagging between the economic activities and the electricity consumption. We could move the economic indicators data horizontally so that the most similarity could be found. Therefore, we could get time offset between the economic consumption and economic indicators. In the experiment, we used three similarity functions to compare the results. Our algorithm outperforms state-of-the-art techniques in terms of both effectiveness and efficiency.
Named Data Networking (NDN) improves the data delivery efficiency by caching contents in routers. To prevent corrupted and faked contents be spread in the network, NDN routers should verify the digital signature of each published content. Since the verification scheme in NDN applies the asymmetric encryption algorithm to sign contents, the content verification overhead is too high to satisfy wire-speed packet forwarding. In this paper, we propose two schemes to improve the verification performance of NDN routers to prevent content poisoning. The first content verification scheme, called “user-assisted”, leads to the best performance, but can be bypassed if the clients and the content producer collude. A second scheme, named “Router-Cooperation”, prevents the aforementioned collusion attack by making edge routers verify the contents independently without the assistance of users and the core routers no longer verify the contents. The Router-Cooperation verification scheme reduces the computing complexity of cryptographic operation by replacing the asymmetric encryption algorithm with symmetric encryption algorithm. The simulation results demonstrate that this Router-Cooperation scheme can speed up 18.85 times of the original content verification scheme with merely extra 80 Bytes transmission overhead.
The search engine is tightly coupled with social networks and is primarily designed for users to acquire interested information. Specifically, the search engine assists the information dissemination for social networks, i.e., enabling users to access interested contents with keywords-searching and promoting the process of contents-transferring from the source users directly to potential interested users. Accompanying such processes, the social network evolves as new links e-merge between users with common interests. However, there is no clear understanding of such a "chicken-and-egg" problem, namely, new links encourage more social interactions, and vice versa. In this paper, we aim to quantitatively characterize the social network evolution phenomenon driven by a search engine. First, we propose a search network model for social network evolution. Second, we adopt two performance metrics, namely, degree distribution and network diameter. Theoretically, we prove that the degree distribution follows an intensified power-law, and the network diameter shrinks. Third, we quantitatively show that the search engine accelerates the rumor propagation in social networks. Finally, based on four real-world data sets (i.e., CDBLP, Facebook, Weibo, and P2P), we verify our theoretical findings. Furthermore, we find that the search engine dramatically increases the speed of rumor propagation.
Recently, neural-network based word embedding models have been shown to produce high-quality distributional representations capturing both semantic and syntactic information. In this paper, we propose a grouping-based context predictive model by considering the interactions of context words, which generalizes the widely used CBOW model and Skip-Gram model. In particular, the words within a context window are split into several groups with a grouping function, where words in the same group are combined while different groups are treated as independent. To determine the grouping function, we propose a relatedness hypothesis stating the relationship among context words and propose several context grouping methods. Experimental results demonstrate better representations can be learned with suitable context groups.
This paper presents MagSpider, a unique localization-free indoor navigational map construction approach. It only utilizes the unlabeled and noisy measurements by inertial sensors on the smartphones to construct the navigation space; and hence can avoid the labor-intensive indoor floor plan calibration. The current approaches for indoor floor plan construction pursue accurately characterize the indoor structures, which require the support of indoor localization techniques. However, accurate localizing the indoor users is non-trivial, rather to say the labor-intensive process of extracting all the walkable paths from the floor plan. We present a novel idea with MagSpider for constructing a localization-free indoor navigational map. The main idea is to mesh the isolated trajectories with their inherent temporal and spatial characteristics, and ease the burden of localization. We conduct extensive experiments in both an office building and an urban shopping mall over 4,000 m2. We believe our approach can greatly ease the burden of constructing the indoor navigational maps, and pave the way for the rapid deployment and adoption of indoor navigation services.
With the prevalence1 of the computational thinking, Chinese colleges and universities adopt this concept into their elementary computer teaching for cultivating students' thinking and application ability, which strongly support the students' development. Therefore, it is very important to evaluate the effectiveness of computational thinking in the elementary computer courses. Through such an evaluation, we can obtain the objective data which reflects the teaching effect of elementary computer courses, establish scientific teaching evaluation matrices, and emphasize the importance of those courses. In this paper, we investigate the teaching effectiveness for achieve accurate data and decision support on the computational-thinking-oriented computer curriculum reform. Our research is based on recent evaluations and data acquisition. We analyze the test database, categories of questions, and scores of students. We find several problems, such as the over-emphasizing the programming, incomplete knowledge components, limited background, large deviation of evaluation results among universities. Finally, we give several improvement recommendations.
With the increasing pace of education informationization, conventional education mode in China is suffering unprecedented challenges. On one side, new thoughts and new approaches are continuously putting forward, on the other side, old conventions and customs are following the beaten track. Confronted with the fast-changing new technologies which bring totally different ideas to us, how should we deal with the status and what method should we take are the most urgent problems we should seriously think about. This paper begins with the origin of education revolution in China, discusses the impact of modern educational methods to the conventional teaching method, and analyzes whether MOOC and 'Flipped Classroom' can be the catalyst to the Chinese high education reform. Finally, we make an overall summary and outlook about the typical Chinese education reform methods.
Online advertisement is a significant element of the Web browsing experience. A good advertising can not only bring benefits to publisher but also improve user satisfaction and extends advertiser's product marketing. To satisfy the desire of all three parties, the click through rate (CTR) prediction of a user to a specified ad in a specific context is of great importance. This challenging problem plays a key role in online advertising system and has to deal with several hard issues. Firstly, the model must process very high dimensional features from frequently changing ad, user and context, most of which are category features having large cardinality and sparse nature extending the dimensionality by two orders of magnitude. Secondly, nonlinear features such as conjunction information must be integrated into the model for a better prediction accuracy. Finally, the model must be able to parallelized efficiently to train from very large scale data sets. To address these problems, we proposed a novel model called Coupled Logistic Regression (CLR), for accurate and efficient CTR prediction. CLR can exploit all features from ad, user, context and nonlinear features among them by seamlessly integrate the conjunction information by employing factorization machine to achieve precise prediction result. And the high-dimensional problem is avoided by decomposing the decision function into two sub ones. Scalability of CLR is ensured through a newly invited MapReduce parallelization strategy, which can reduce communication and waiting time between nodes. Experimental results on real-world data set show that our CLR model can guarantee both accuracy and efficiency on large scale CTR prediction problems.
This paper presents a MOOC-ready online FPGA laboratory platform which targets computer system experiments. Goal of design is to provide user with highly approximate experience and results as offline experiments. Rich functions are implemented by utilizing SoC FPGA as the controller of lab board. The design details and effects are discussed in this paper.
