We study visual question answering in a setting where the answer has to be mined from a pool of relevant and irrelevant images given as a context. For such a setting, a model must first retrieve relevant images from the pool and answer the question from these retrieved images. We refer to this problem as retrieval-based visual question answering (or RETVQA in short). The RETVQA is distinctively different and more challenging than the traditionally-studied Visual Question Answering (VQA), where a given question has to be answered with a single relevant image in context. Towards solving the RETVQA task, we propose a unifiedMulti Image BART (MI-BART) that takes a question and retrieved images using our relevance encoder for free-form fluent answer generation. Further, we introduce the largest dataset in this space, namely RETVQA, which has the following salient features: multi-image and retrieval requirement for VQA, metadata-independent questions over a pool of heterogeneous images, expecting a mix of classification-oriented and open-ended generative answers. Our proposed framework achieves an accuracy of 76.5% and a fluency of 79.3% on the proposed dataset, namely RETVQA and also outperforms state-of-the-art methods by 4.9% and 11.8% on the image segment of the publicly available WebQA dataset on the accuracy and fluency metrics, respectively.

Table-to-text generation has been widely studied in the Natural Language Processing community in the recent years. We give a new perspective to this problem by incorporating signals from both tables as well as associated images to generate relevant text. While tables contain a structured list of facts, images are a rich source of unstructured visual information. For example, in the tourism domain, images can be used to infer knowledge such as the type of landmark (e.g., church), its architecture (e.g., Ancient Roman), and composition (e.g., white marble). Therefore, in this paper, we introduce the novel task of Vision-augmented Table-To-Text Generation (VisToT, defined as follows: given a table and an associated image, produce a descriptive sentence conditioned on the multimodal input. For the task, we present a novel multimodal table-to-text dataset, WikiLandmarks, covering 73,084 unique world landmarks. Further, we also present a competitive architecture, namely, VT3 that generates accurate sentences conditioned on the image and table pairs. Through extensive analyses and experiments, we show that visual cues from images are helpful in (i) inferring missing information from incomplete or sparse tables, and (ii) strengthening the importance of useful information from noisy tables for natural language generation. We make the code and data publicly available.

Shopping experience on any e-commerce website is largely driven by the content customers interact with. The large volume of diverse content on e-commerce platforms, and the advances in machine learning, pose unique opportunities for gathering insights through content understanding and applying these insights to generate content better shopper experience. The purpose of the first edition of this workshop was to bring together researchers from industry and academia on questions surrounding e-commerce content understanding and generation.

According to Bangladesh Demographic Health Survey (BDHS 2014) only 55% infants aged 0-6 months are exclusively breastfed mainly due to traditional norms and lack of knowledge of mothers / family members about recommended practices (Formative Research Report, Ministry of Information & UNICEF, 2014). Moreover, according to the Bangladesh national labor force survey in 2010, around 3.2 million women are working in Garments sector. Most of them reside in urban slums where malnutrition rates are high. Despite having major contribution to the national growth, they have very limited access and rights to enjoy benefits like maternity leave, baby-friendly policies etc. Female workers in lactating stage often face challenges to ensure EBF. Through Mothers@Work initiative of UNICEF, BRAC is implementing this programme to improve EBF support and practice for working mothers in RMG factories by engaging public, private and civil society stakeholders. The objective of the programme is to improve quality of breastfeeding counseling and support for pregnant and lactating mothers (PLW) during ANC and PNC as well as to improve EBF practices in the workplace. This project aims to address the bottlenecks to practice EBF in workplaces. Two RMG factories were selected randomly of Dhaka district to implement seven minimum standards (such as maternity leave and benefits, day-care & breastfeeding centers, breastfeeding supportive environment, maternity health protection, breastfeeding breaks etc.) based on the Global and Bangladesh policy framework and guidelines to enhance breastfeeding support among working women through targeting senior, mid management and general workers of factories. The project after running almost one and half years could increase EBF rates from 17% (October 2016) to 72% (August 2018) in both factories. Biography: Mithun Gupta has completed her Masters of Public Health from National Institute of Preventive and Social Medicine (NIPSOM), Bangladesh. She has been working in public health since last seven years. She is now working as a Senior Sector Specialist and leading programmer focusing early childhood nutrition of Health, Nutrition and Population department of BRAC, which is world’s number one development organization. Speaker Publications: 1. Kramer MS, Kakuma R. The optimal duration of exclusive breastfeeding. Protecting infants through human milk: Springer; 2004. p. 63–77. 2. Zar JH. Biostatistical analysis: Pearson Education India; 1999. 3. Kawachi I, Berkman LF. Neighborhoods and health: Oxford University Press; 2003. 5th World Congress on Public Health and Nutrition; London, UK- February 24-25, 2020. Abstract Citation: Mithun Gupta, Promoting Exclusive Breast Feeding (EBF) among working mothers of Ready-Made Garment (RMG) sector of Bangladesh to ensure proper nutrition for their children, Public Health 2020, 5th World Congress on Public Health and Nutrition; London, UK- February 24-25, 2020. (https://publichealth.healthconferences.org/abstract/2020/promoting-exclusive-breast-feeding-ebf-among-working-mothers-of-ready-made-garment-rmg-sector-of-bangladesh-to-ensure-proper-nutrition-for-their-children)

We present the problem of Visually Precise Query (VPQ) generation which enables a more intuitive match between a user's information need and an e-commerce site's product description. Given an image of a fashion item, what is the most optimum search query that will retrieve the exact same or closely related product(s) with high probability. In this paper we introduce the task of VPQ generation which takes a product image and its title as its input and provides aword level extractive summary of the title, containing a list of salient attributes, which can now be used as a query to search for similar products. We collect a large dataset of fashion images and their titles and merge it with an existing research dataset which was created for a different task. Given the image and title pair, VPQ problem is posed as identifying a non-contiguous collection of spans within the title. We provide a dataset of around 400K image, title and corresponding VPQ entries and release it to the research community. We provide a detailed description of the data collection process as well as discuss the future direction of research for the problem introduced in this work. We provide the standard text as well as visual domain baseline comparisons and also provide multi-modal baseline models to analyze the task introduced in this work. Finally, we propose a hybrid fusion model which promises to be the direction of research in the multi-modal community.

We present a determinantal point process (DPP) inspired alternative to non-maximum suppression (NMS) which has become an integral step in all state-of-the-art object detection frameworks. DPPs have been shown to encourage diversity in subset selection problems. We pose NMS as a subset selection problem and posit that directly incorporating DPP like framework can improve the overall performance of the object detection system. We propose an optimization problem which takes the same inputs as NMS, but introduces a novel sub-modularity based diverse subset selection functional. Our results strongly indicate that the modifications proposed in this paper can provide consistent improvements to state-of-the-art object detection pipelines.

Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. ADEM (Lowe et al. 2017) formulated the automatic evaluation of dialogue systems as a learning problem and showed that such a model was able to predict responses which correlate significantly with human judgements, both at utterance and system level. Their system was shown to have beaten word-overlap metrics such as BLEU with large margins. We start with the question of whether an adversary can game the ADEM model. We design a battery of targeted attacks at the neural network based ADEM evaluation system and show that automatic evaluation of dialogue systems still has a long way to go. ADEM can get confused with a variation as simple as reversing the word order in the text! We report experiments on several such adversarial scenarios that draw out counterintuitive scores on the dialogue responses. We take a systematic look at the scoring function proposed by ADEM and connect it to linear system theory to predict the shortcomings evident in the system. We also devise an attack that can fool such a system to rate a response generation system as favorable. Finally, we allude to future research directions of using the adversarial attacks to design a truly automated dialogue evaluation system.

We present a conditional generative adversarial model to draw realistic samples from paired fashion clothing distribution and provide real samples to pair with arbitrary fashion units. More concretely, given an image of a shirt, obtained from a fashion magazine, a brochure or even any random click on ones phone, we draw realistic samples from a parameterized conditional distribution learned as a conditional generative adversarial network (c^+GAN) to generate the possible pants which can go with the shirt. We start with a classical cGAN model as proposed by Mirza and Osindero [arXiv:1411.1784] and modify both the generator and discriminator to work on captured-in-the-wild data with no human alignment. We gather a dataset from web crawled data, systematically develop a method which counters the problems inherent to such data, and finally present plausible results based on our technique. We propose simple ideas to evaluate how these techniques can conquer the cognitive gap that exists when arbitrary clothing articles need to be paired with another relevant article, based on similarity of search results.

In developing countries like India agriculture plays an extremely important role in the lives of the population. In India, around 80\% of the population depend on agriculture or its by-products as the primary means for employment. Given large population dependency on agriculture, it becomes extremely important for the government to estimate market factors in advance and prepare for any deviation from those estimates. Commodity arrivals to market is an extremely important factor which is captured at district level throughout the country. Historical data and short-term prediction of important variables such as arrivals, prices, crop quality etc. for commodities are used by the government to take proactive steps and decide various policy measures. In this paper, we present a framework to work with short timeseries in conjunction with remote sensing data to predict future commodity arrivals. We deal with extremely high dimensional data which exceed the observation sizes by multiple orders of magnitude. We use cascaded layers of dimensionality reduction techniques combined with regularized regression models for prediction. We present results to predict arrivals to major markets and state wide prices for `Tur' (red gram) crop in Karnataka, India. Our model consistently beats popular ML techniques on many instances. Our model is scalable, time efficient and can be generalized to many other crops and regions. We draw multiple insights from the regression parameters, some of which are important aspects to consider when predicting more complex quantities such as prices in the future. We also combine the insights to generate important recommendations for different government organizations.

The simple approach of retrieving a closest match of a query image from one in the gallery, compares an image pair using sum of absolute difference in pixel or feature space. The process is computationally expensive, ill-posed to illumination, background composition, pose variation, as well as inefficient to be deployed on gallery sets with more than 1000 elements. Hashing is a faster alternative which involves representing images in reduced dimensional simple feature spaces. Encoding images into binary hash codes enables similarity comparison in an image-pair using the Hamming distance measure. The challenge, however, lies in encoding the images using a semantic hashing scheme that lets subjective neighbors lie within the tolerable Hamming radius. This work presents a solution employing adversarial learning of a deep neural semantic hashing network for fashion inventory retrieval. It consists of a feature extracting convolutional neural network (CNN) learned to (i) minimize error in classifying type of clothing, (ii) minimize hamming distance between semantic neighbors and maximize distance between semantically dissimilar images, (iii) maximally scramble a discriminator's ability to identify the corresponding hash code-image pair when processing a semantically similar query-gallery image pair. Experimental validation for fashion inventory search yields a mean average precision (mAP) of 90.65% in finding the closest match as compared to 53.26% obtained by the prior art of deep Cauchy hashing for hamming space retrieval.

Search queries issued over the Web increasingly look like questions, especially as the domain becomes more specific. Finding good response to such queries amounts to finding relevant passages from Web documents. Traditional information retrieval based Web search still matches the query to the words in the entire document. With the advent of machine reading comprehension techniques, Web search is moving more towards identifying the best sentence / group of sentences in the document. We present AQuPR an A ttention based Qu ery P assage R etrieval system to find human acceptable answer containing passages to technology queries issued over the Web. We train character level embeddings for the query and passage pairs, train a deep recurrent network with a novel simplified attention mechanism and incorporate additional signals present in Web documents to improve the performance of such a system. We collect a database of human issued queries along with their answer passages and learn an end to end system to enable automated query resolution. We present results for answering human issued search queries which show considerable promise against basic versions of current generation question answering systems.

Text classification is a fundamental task in NLP applications. Latest research in this field has largely been divided into two major sub-fields. Learning representations is one sub-field and learning deeper models, both sequential and convolutional, which again connects back to the representation is the other side. We posit the idea that the stronger the representation is, the simpler classifier models are needed to achieve higher performance. In this paper we propose a completely novel direction to text classification research, wherein we convert text to a representation very similar to images, such that any deep network able to handle images is equally able to handle text. We take a deeper look at the representation of documents as an image and subsequently utilize very simple convolution based models taken as is from computer vision domain. This image can be cropped, re-scaled, re-sampled and augmented just like any other image to work with most of the state-of-the-art large convolution based models which have been designed to handle large image datasets. We show impressive results with some of the latest benchmarks in the related fields. We perform transfer learning experiments, both from text to text domain and also from image to text domain. We believe this is a paradigm shift from the way document understanding and text classification has been traditionally done, and will drive numerous novel research ideas in the community.

Chat bots are becoming ubiquitous in our day to day life. The advent of the summer of AI has brought us all in close contact with intelligent agents such as Cortana, Siri and Alexa. We envisage a world, where these bots have there physical existence within the realm of Mixed Reality (MR). We present the first 3D chit-chat bot called the HoloCompanion. This bot has a personality, can chat with anyone and about any topic and has articulated lip, eye and head movements.

In the theory of compressed sensing (CS), the sparsity x_0 of the unknown signal 𝐱∈ℛ^n is of prime importance and the focus of reconstruction algorithms has mainly been either x_0 or its convex relaxation (via x_1). However, it is typically unknown in practice and has remained a challenge when nothing about the size of the support is known. As pointed recently, x_0 might not be the best metric to minimize directly, both due to its inherent complexity as well as its noise performance. Recently a novel stable measure of sparsity s(𝐱) := 𝐱_1^2/𝐱_2^2 has been investigated by Lopes , which is a sharp lower bound on 𝐱_0. The estimation procedure for this measure uses only a small number of linear measurements, does not rely on any sparsity assumptions, and requires very little computation. The usage of the quantity s(𝐱) in sparse signal estimation problems has not received much importance yet. We develop the idea of incorporating s(𝐱) into the signal estimation framework. We also provide a three step algorithm to solve problems of the form 𝐀𝐱=𝐛 with no additional assumptions on the original signal 𝐱.

Techniques for automating the generation and analysis of fundus drawings are described. Captured images undergo image processing to extract information about image features. Fundus images are generated and recommended labels for the fundus drawing are generated. Fundus drawings can be analyzed and undergo textual processing to extract existing labels. Machine learning models and co-occurrence analysis can be applied to collections of fundus images and drawings to gather information about commonly associated labels, label locations, and user information. The most frequently used labels associated with the image can be identified to improve recommendations and personalize labels.

Psoriasis Area and Severity Index, or PASI score [7] is one of the most prevalent scoring indices for Psoriasis. Erythema or redness of skin is an important identifier for evaluation of PASI score. Extra subjectiveness in the evaluation of erythema has been observed, since the perception of redness can be influenced by the skin tone, ambient lighting and many other such factors which are difficult to control in a clinical setting. We propose a novel colorimetric feature for erythema grading by extending the tissue-photon interaction model [12] to make it skin tone independent. We propose to use Skellam distribution statistics as feature vectors for erythema grading. We present a random forest based technique for classification of erythema regions in Psoriasis into severe, moderate and slight categories.

— Content Based Image Retrieval (CBIR) coupled with Case Based Reasoning (CBR) is a paradigm that is becoming increasingly popular in the diagnosis and therapy planning of medical ailments utilizing the digital content of medical images. This paper presents a survey of some of the promising approaches used in the detection of abnormalities in retina images as well in mammographic screening and detection of regions of interest in MRI scans of the brain. We also describe our proposed algorithm to detect hard exudates in fundus images of the retina of Diabetic Retinopathy patients.

In this paper we present a symmetric KL divergence based agglomerative clustering framework to segment multiple levels of depigmentation in Vitiligo images. The proposed framework starts with a simple merge cost based on symmetric KL divergence. We extend the recent body of work related to Bregman divergence based agglomerative clustering and prove that the symmetric KL divergence is an upper-bound for uni-modal Gaussian distributions. This leads to a very powerful yet elegant method for bottom-up agglomerative clustering with strong theoretical guarantees. We introduce albedo and reflectance fields as features for the distance computations. We compare against other established methods to bring out possible pros and cons of the proposed method.