Automatic fashion attribute recognition enables retailers to address an array of applications. Usually, fashion attributes are manually input in the system by retailers, which is a time-consuming and an error prone process. To alleviate this, several existing works use traditional CNN-based backbones to recognize attributes. These backbones generate attribute embeddings that are entangled in the feature space. Existing methods that generate disentangled attribute embedding do not explicitly specify the location of attributes, and often extract features from irrelevant regions. This directly impacts the quality of downstream tasks. To alleviate this problem, we have proposed a novel framework to extract location-aware attribute representation using localization maps created from fashion landmarks. These localization maps highlight regions of interest in an image, aiding localized attribute feature extraction. Moreover, we have proposed a novel fusion module to effectively select important features from the global representation of an image to enhance the local features of the attribute. These attribute embeddings are then used in downstream applications such as attribute recognition, hierarchical taxonomy classification, and retrieval with two large-scale datasets. Using the proposed model, we observe improvement in performance from the state-of-the-art by a significant margin.

Glaucoma remains one of the leading causes of irreversible blindness, its timely detection being imperative to avoiding permanent visual impairment. Deep learning methods offer a solution for early detection of Glaucoma by reducing the need for manual labor at screening stages. Hence, numerous automated methods have been proposed to assist experts in diagnosing Glaucoma from fundus images. However, the sole focus on increasing the accuracy of predictions has resulted in a lack of trust due to the blackbox nature of such models. Similar sentiment across multiple high-stakes decision domains has led to a growing demand for replacing black-box models with glass-box ones. In this work, we propose an inherently explainable model that 1.) learns class-specific prototypes, which capture the general characteristics or concepts of the pathology, 2.) uses the actual visualized prototypes in the decision-making process by computing the similarity between them and the query image, as a result revealing the underlying model’s reasoning process, 3) is end-to-end optimizable. Moreover, the proposed approach does not require joint training of the classification models with decoders for visualization of the prototypes, simplifying the overall training process. Experimental results demonstrate that our proposed approach achieves comparable performance with its black-box counterparts and outperforms the state-of-the-art baseline, both quantitatively and qualitatively, on the benchmark RIMONE DL dataset.

Recommendation systems improve users' online shopping experience by recommending relevant items from a large pool of items in different categories. Fashion recommendation systems apart from recommending individual fashion items also recommend fashion outfits. In this work, we consider the problem of the outfit compatibility prediction task, an integral part of the fashion outfit recommendation system. A compatibility prediction module determines whether all the items in an outfit are visually compatible with each other and match the user's preferences. Existing approaches can be grouped based on the representation scheme: (i) pair-wise and (ii) set or sequence. Pair-wise representation does not consider the outfit as a whole, and the sequence representation approaches are sensitive to the ordering of the items. Further, these methods do not explicitly capture the visual relationship between the items. We propose a novel method for the personalized outfit-compatible prediction task. The proposed method represents the outfit as a graph and uses a dot-attention graph neural network to capture the visual relationship between items. The graph read-out layer generates the final outfit embedding. A novel approach is proposed to model the user's preference for different styles. The final outfit compatibility score is generated by computing the similarity between outfit embedding and user embedding. Experimental results and ablation study on the Polyvore-U dataset, highlight the effectiveness of the proposed method.

Identification of bias and its mitigation in a classifier is a fundamental sanity check required in trustworthy AI systems. There have been many methods for mitigation of bias in literature that use bias as apriori information. In this work, we propose a system that can detect the low-level bias (e.g., color, texture) and mitigate the same. A novel auto-encoder architecture to explain the predictions made by a deep neural network is built that helps in identification of the bias. The auto-encoder is trained to produce a generalized representation of the input image by decomposing it into a set of latent embeddings. These embeddings are learned by specializing the group of higher dimensional feature maps to learn the disentangled color and shape concepts. The shape embeddings are trained to reconstruct discrete wavelet transform components of an image and the color embeddings are trained to capture the color information. The feature specialization is done by reconstructing the RGB image using the shape embeddings modulated by color embeddings. We have shown that these representations can be used to detect low level bias in a classification task. Post detection of bias, we also propose a method to de-bias the classifier by training it with counterfactual images generated by manipulating the representations learned by the auto-encoder. We have shown that our proposed method of bias discovery and mitigation is able to achieve state-of-the-art results on ColorMNIST and the newly proposed BiasedShape dataset.

Scene graph captures rich semantic information of an image by representing objects and their relationships as nodes and edges of a graph. Recent works have demonstrated that scene graph representation improves the performance of various computer vision tasks such as image retrieval, action recognition, visual question answering. Computationally efficient scene graph generation methods are required to leverage scene graphs in various real-world applications (e.g., autonomous driving, robotics). A typical scene graph generation model consists of two modules: (i) object detector and (ii) scene graph classifier. The scene graph classifier module predicts the object category and object-object relationships. The presence of a quadratic number of potential edges poses a major challenge in the scene graph classification task. Detecting the relationship between each object pair using the traditional approach is computationally intensive and non-scalable. To address this issue, we propose a novel module named EdgeNet that directly predicts the set of relevant edges and helps to prune out a significant number of unrelated object pairs, thereby improving the effectiveness and efficiency of the scene graph classifier. The proposed EdgeNet is a generic module and can be plugged into an existing scene graph classifier. Experimental results highlight the effectiveness and efficiency of the proposed approach on the Visual Genome dataset.

Over the past few decades, minimally invasive surgical techniques have gained wide acceptance due to multiple benefits it offers. In these surgeries, a camera with a light source is inserted via a small incision. The video feed from the camera is the only source for visualization of internal organs. Certain procedures produce fumes that severely degrade the video feed. Various image processing based de-smoking systems are proposed to provide a continuous, good quality video feed. However, most of the existing approaches perform de-smoking at the frame level and fail to exploit the dynamic properties of the smoke. We propose a novel de-smoking model that harnesses both spatial and temporal properties of smoke. We evaluate the performance of the proposed model on the Cholec-80 dataset and observe a superior performance in terms of MS-SSIM and PSNR metrics compared to existing works.

As humans, we inherently perceive images based on their predominant features, and ignore noise embedded within lower bit planes. On the contrary, Deep Neural Networks are known to confidently misclassify images corrupted with meticulously crafted perturbations that are nearly imperceptible to the human eye. In this work, we attempt to address this problem by training networks to form coarse impressions based on the information in higher bit planes, and use the lower bit planes only to refine their prediction. We demonstrate that, by imposing consistency on the representations learned across differently quantized images, the adversarial robustness of networks improves significantly when compared to a normally trained model. Present state-of-the-art defenses against adversarial attacks require the networks to be explicitly trained using adversarial samples that are computationally expensive to generate. While such methods that use adversarial training continue to achieve the best results, this work paves the way towards achieving robustness without having to explicitly train on adversarial samples. The proposed approach is therefore faster, and also closer to the natural learning process in humans.

Deep learning models have shown impressive performance across a spectrum of computer vision applications including medical diagnosis and autonomous driving. One of the major concerns that these models face is their susceptibility to adversarial attacks. Realizing the importance of this issue, more researchers are working towards developing robust models that are less affected by adversarial attacks. Adversarial training method shows promising results in this direction. In adversarial training regime, models are trained with mini-batches augmented with adversarial samples. In order to scale adversarial training to large networks and datasets, fast and simple methods (e.g., single-step gradient ascent) are used for generating adversarial samples. It is shown that models trained using single-step adversarial training method (adversarial samples are generated using non-iterative method) are pseudo robust. Further, this pseudo robustness of models is attributed to the gradient masking effect. However, existing works fail to explain when and why gradient masking effect occurs during single-step adversarial training. In this work, (i) we show that models trained using single-step adversarial training method learns to prevent the generation of single-step adversaries, and this is due to over-fitting of the model during the initial stages of training, and (ii) to mitigate this effect, we propose a single-step adversarial training method with dropout scheduling to learn robust models. Unlike models trained using single-step adversarial training method, models trained using the proposed single-step adversarial training method are robust against both single-step and multi-step adversarial attacks, and achieve on-par results compared to the computationally expensive state-of-the-art multi-step adversarial training method, in white-box and black-box settings.

Deep learning models have shown impressive performance across a spectrum of computer vision applications including medical diagnosis and autonomous driving. One of the major concerns that these models face is their susceptibility to adversarial attacks. Realizing the importance of this issue, more researchers are working towards developing robust models that are less affected by adversarial attacks. Adversarial training method shows promising results in this direction. In adversarial training regime, models are trained with mini-batches augmented with adversarial samples. Fast and simple methods (e.g., single-step gradient ascent) are used for generating adversarial samples, in order to reduce computational complexity. It is shown that models trained using single-step adversarial training method (adversarial samples are generated using non-iterative method) are pseudo robust. Further, this pseudo robustness of models is attributed to the gradient masking effect. However, existing works fail to explain when and why gradient masking effect occurs during single-step adversarial training. In this work, (i) we show that models trained using single-step adversarial training method learn to prevent the generation of single-step adversaries, and this is due to over-fitting of the model during the initial stages of training, and (ii) to mitigate this effect, we propose a single-step adversarial training method with dropout scheduling. Unlike models trained using existing single-step adversarial training methods, models trained using the proposed single-step adversarial training method are robust against both single-step and multi-step adversarial attacks, and the performance is on par with models trained using computationally expensive multi-step adversarial training methods, in white-box and black-box settings.

Adversarial Training (AT) is a straight forward solution to learn robust models by augmenting the training mini-batches with adversarial samples. Adversarial attack methods range from simple non-iterative (single-step) methods to computationally complex iterative (multi-step) methods. Although the single-step methods are efficient, the models trained using these methods merely appear to be robust, due to the masked gradients. In this work, we propose a novel regularizer named Plug-And-Pipeline (PAP) for single-step AT. The proposed regularizer attenuates the gradient masking effect by promoting the model to learn similar representations for both single-step and multi-step adversaries. Further, we present a novel pipelined approach that allows an efficient implementation of the proposed regularizer. Plug-And-Pipeline yields robustness comparable to multi-step AT methods, while requiring a low computational overhead, similar to that of single-step AT methods.

Deep Neural Networks trained using human-annotated data are able to achieve human-like accuracy on many computer vision tasks such as classification, object recognition and segmentation. However, they are still far from being as robust as the human visual system. In this paper, we demonstrate that even models that are trained to be robust to random perturbations do not necessarily learn robust representations. We propose to address this by imposing a perception based prior on the learned representations to ensure that perceptually similar images have similar representations. We demonstrate that, although this training method does not use adversarial samples during training, it significantly improves the network’s robustness to single-step and multi-step adversarial attacks, thus validating our hypothesis that the network indeed learns more robust representations. Our proposed method provides a means of achieving adversarial robustness at no additional computational cost when compared to normal training.

Neural networks are susceptible to adversarial samples: samples with imperceptible noise, crafted to manipulate network's prediction. In order to learn robust models, a training procedure, called Adversarial Training has been introduced. During adversarial training, models are trained with mini-batch containing adversarial samples. In order to scale adversarial training for large datasets and networks, fast and simple methods (e.g., FGSM:Fast Gradient Sign Method) of generating adversarial samples are used while training. It has been shown that models trained using single-step adversarial training methods (i.e., adversarial samples generated using non-iterative methods such as FGSM) are not robust, instead they learn to generate weaker adversaries by masking the gradients. In this work, we propose a regularization term in the training loss, to mitigate the effect of gradient masking during single-step adversarial training. The proposed regularization term causes training loss to increase when the distance between log its (i.e., pre-softmax output of a classifier) for FGSM and R-FGSM (small random noise is added to the clean sample before computing its FGSM sample) adversaries of a clean sample becomes large. The proposed single-step adversarial training is faster than computationally expensive state-of-the-art PGD adversarial training method, and also achieves on par results.

Though Deep Neural Networks (DNN) show excellent performance across various computer vision tasks, several works show their vulnerability to adversarial samples, i.e., image samples with imperceptible noise engineered to manipulate the network's prediction. Adversarial sample generation methods range from simple to complex optimization techniques. Majority of these methods generate adversaries through optimization objectives that are tied to the pre-softmax or softmax output of the network. In this work we, (i) show the drawbacks of such attacks, (ii) propose two new evaluation metrics: Old Label New Rank (OLNR) and New Label Old Rank (NLOR) in order to quantify the extent of damage made by an attack, and (iii) propose a new attack FDA: Feature Disruptive attack, to address the drawbacks of existing attacks. FDA works by generating image perturbation that disrupts features at each layer of the network and causes deep-features to be highly corrupt. This allows FDA adversaries to severely reduce the performance of deep networks. We experimentally validate that FDA generates stronger adversaries than other state-of-the-art methods for Image classification, even in the presence of various defense measures. More importantly, we show that FDA disrupts feature-representation based tasks even without access to the task-specific network or methodology.

Adversarial samples are perturbed inputs crafted to mislead the machine learning systems. A training mechanism, called adversarial training, which presents adversarial samples along with clean samples has been introduced to learn robust models. In order to scale adversarial training for large datasets, these perturbations can only be crafted using fast and simple methods (e.g., gradient ascent). However, it is shown that adversarial training converges to a degenerate minimum, where the model appears to be robust by generating weaker adversaries. As a result, the models are vulnerable to simple black-box attacks. In this paper we, (i) demonstrate the shortcomings of existing evaluation policy, (ii) introduce novel variants of white-box and black-box attacks, dubbed “gray-box adversarial attacks” based on which we propose novel evaluation method to assess the robustness of the learned models, and (iii) propose a novel variant of adversarial training, named “Gray-box Adversarial Training” that uses intermediate versions of the models to seed the adversaries. Experimental evaluation demonstrates that the models trained using our method exhibit better robustness compared to both undefended and adversarially trained models.

In this paper we present the design, architecture and implementation of KNADIA, a conversational dialogue system for intra-enterprise use, providing knowledge-assisted question answering and transactional assistance to employees of a large organization. KNADIA has been deployed in production in TCS, a large organization with over 380,000 employees distributed globally; the system is currently supporting a few thousand active users making hundreds of queries per day. We identify, define and distinguish two distinct classes of use-cases: virtual assistance and knowledge synthesis, which we have found to cover a variety of enterprise needs. KNADIA supports both types of conversational agents, with multiple instances of each, while presenting a common digital persona. KNADIA disambiguates which agent should respond to each query. Further, since individual components use deep learning algorithms giving probabilistic outputs, the confidence with which different components answer is often required before answering or acting. The user's dialogue context also needs to be maintained judiciously. Due to these and many other challenges, the overall architecture of KNADIA is non-trivial. We present instances of HR-assistance and technical knowledge synthesis that are in production use in TCS along with accuracy figures and key performance metrics. Finally, we suggest that many elements of our architecture are also generally applicable to other complex deep learning systems.