Self-correction techniques have recently emerged as a promising framework to improve the quality of responses generated by large language models (LLMs). Few-shot prompted LLMs act as critics to produce feedback for an input, which is further fed to a refiner (also an LLM) to produce an output. However, these critique-refine steps require multiple expensive LLM calls. To circumvent this large inference cost, we borrow inspiration from prior work on knowledge distillation and propose the use of critique distillation to train critic models. These are smaller sequence-to-sequence models that are trained on input-critique pairs generated by an LLM. We focus on the problem of text simplification for three Indian languages: Hindi, Bengali and Marathi. This task is a good fit for self-correction style techniques. It also has not been systematically explored for Indian languages before. We train two separate critics that focus on lexical and structure complexity, and show that it is surprisingly more effective than using an LLM directly as a critic in both 0-shot and few-shot settings. We also show the benefits of training multilingual critics, as opposed to monolingual critics. Extensive human evaluations show that on average, raters find 80% of DIMSIM's output to be simple and easy to read.

Code-switching has seen growing interest in recent years as an important multilingual NLP phenomenon. Generating code-switched text for data augmentation has been sufficiently well-explored. However, there is no prior work on generating code-switched text with fine-grained control on the degree of code-switching and the lexical choices used to convey formality. We present CoCoa, an encoder-decoder translation model that converts monolingual Hindi text to Hindi-English code-switched text with both encoder-side and decoder-side interventions to achieve fine-grained controllable generation. CoCoa can be invoked at test-time to synthesize code-switched text that is simultaneously faithful to syntactic and lexical attributes relevant to code-switching. CoCoa outputs were subjected to rigorous subjective and objective evaluations. Human evaluations establish that our outputs are of superior quality while being faithful to desired attributes. We show significantly improved BLEU scores when compared with human-generated code-switched references. Compared to competitive baselines, we show 10% reduction in perplexity on a language modeling task and also demonstrate clear improvements on a downstream code-switched sentiment analysis task.

Webscale services dealing with sensitive content are increasingly being deployed in public and hybrid cloud environments. At the same time, the impact of security breaches have also increased manifold averaging at USD 3.86M per data breach. To tackle such increasing risks, regulations and security frameworks are defined that an organization must comply with. Most of these frameworks are published in natural language text that run into hundreds of pages resulting into thousands of requirements and controls. When these frameworks undergo revisions, understanding the changes, and interpreting their impact consumes huge amount of time, effort and resources. In this paper, we propose a change management system that supports SMEs with AI-assisted automation of this extremely manual and time consuming activity. Specifically, we introduce the concept of live crosswalks – a framework that models complex relationships among security and compliance documents along with associated operations to manage the change. It uses natural language processing (NLP) and algorithmic techniques to transform the current document-driven, highly manual process into a data-driven interactive intelligent system. We present the overall design and demonstrate its efficacy over several hundreds of diversified controls through experimental evaluation.

In this paper we present our experience in the design, modelling, implementation and evaluation of a conversational medical school tutor (MST), employing AI on the cloud. MST combines case-based tutoring with competency based curriculum review, using a natural language interface to enable an adaptive and rich learning experience. It is designed both to engage and tutor medical students through Digital Virtual Patient (DVP) interactions built around clinical reasoning activities and their application of foundational knowledge. DVPs in MST are realistic clinical cases authored by subject matter experts in natural language text. The context of each clinical case is modelled as a set of complex concepts with their associated attributes and synonyms using the UMLS ontology. The MST conversational engine understands the intent of the user’s natural language inputs by training Watson Assistant service and drives a meaningful dialogue relevant to the clinical case under investigation. The curriculum content is analysed using NLP techniques and represented as a related and cohesive graph with concepts as its nodes. The runtime application is modelled as a dynamic and adaptive flow between the case and student characteristics. We describe in detail the various challenges encountered in the design and implementation of this intelligent tutor and also present evaluation of the tutor through two field trials with third and fourth year students comprising of 90 medical students.

A typical medical curriculum is organized as a hierarchy of learning outcomes (LOs), each LO is a short text that describes a medical concept. Machine learning models have been applied to predict relatedness between LOs. These models are trained on examples of LO-relationships annotated by experts. However, medical curricula are periodically reviewed and revised, resulting in changes to the structure and content of LOs. This work addresses the problem of model adaptation under curriculum drift. First, we propose heuristics to generate reliable annotations for the revised curriculum, thus eliminating dependence on expert annotations. Second, starting with a model pre-trained on the old curriculum, we inject a task-specific transformation layer to capture nuances of the revised curriculum. Our approach makes significant progress towards reaching human-level performance.

A typical medical curriculum is organized in a hierarchy of instructional objectives called Learning Outcomes (LOs); a few thousand LOs span five years of study. Gaining a thorough understanding of the curriculum requires learners to recognize and apply related LOs across years, and across different parts of the curriculum. However, given the large scope of the curriculum, manually labeling related LOs is tedious, and almost impossible to scale. In this paper, we build a system that learns relationships between LOs, and we achieve up to human-level performance in the LO relationship extraction task. We then present an application where the proposed system is employed to build a map of related LOs and Learning Resources (LRs) pertaining to a virtual patient case. We believe that our system can help medical students grasp the curriculum better, within classroom as well as in Intelligent Tutoring Systems (ITS) settings.

This paper studies the social networking and community question answering aspects of Khan Academy, a popular yet largely uninvestigated online educational forum. We start with a brief description of our dataset and data collection methodology. We then proceed to construct the underlying network and study its topology based on degree distribution and degree correlation. We examine the performance of different ranking algorithms vis-a-vis user-provided expertise ranking, and explain the observed high correlation with PageRank. Furthermore, we empirically observe how interactions evolve as a course advances, and note that while the network progressively shrinks because low-performing nodes drop out, it also becomes a more tight-knit community. We infer that users who drop out are possibly novice learners, who ask several questions but lack the required expertise to answer many questions themselves. Throughout our work, we draw parallels with existing studies on other web-based question-answering forums which are primarily targeted towards an adult population.

The problem of maximizing the spread of influence with a limited budget is central to social networks research. Most solution approaches available in the existing literature devote the entire budget towards triggering diffusion at seed nodes. This paper investigates the effect of splitting the budget across two different, sequential phases. In phase 1, we adopt the classical approach of initiating diffusion at a selected seed-set. In phase 2, we use the remaining budget to offer referral incentives. We formulate this problem and explore suitable ways to split the budget between the two phases, with detailed experiments on synthetic and real-world datasets. The principal findings from our study are: (a) when the budget is low, it is prudent to use the entire budget for phase 1; (b) when the budget is moderate to high, it is preferable to use much of the budget for phase 1, while allocating the remaining budget to phase 2; (c) in the presence of moderate to strict temporal constraints, phase 2 is not warranted; (d) if the temporal constraints are low or absent, phase 2 yields a decisive improvement in influence spread.