Knowledge graphs have been commonly used to represent relationships between entities and utilized in the industry to enhance service qualities. As knowledge graphs integrate data from a variety of sources, they can also be useful references for human users. However, there is a lack of effective tools for data analysts to make the most of the rich information in knowledge graphs. Existing knowledge graph exploration systems are ineffective because they didn’t consider various users’ needs and the characteristics of knowledge graphs. Exploratory approaches specifically designed for uncovering and summarizing insights in knowledge graphs have not been well studied yet. In this paper, we propose KGScope that supports interactive visual explorations and provides embedding-based guidance to derive insights from knowledge graphs. We demonstrate KGScope with a usage scenario and assess its efficacy in supporting knowledge graph exploration with a user study. The results show that KGScope supports knowledge graph exploration effectively by providing useful information and aiding comprehensive exploration.

While existing visualization systems excel in exploring datasets and discovering data patterns and insights, challenges remain in automatically generating infographics from exploration-derived visualizations. We propose VisCollage, a computational pipeline that automatically organizes and renders charts from an exploration in a “visual collage”, which is inspired by data journalism and can be viewed as a kind of “partitioned poster infographic”. By analyzing the relation (e.g., drill-down or comparison) between charts established during exploration, VisCollage groups and merges them to reduce data redundancy. In addition, VisCollage automatically identifies a main chart of the exploration and arranges annotations and background charts around it. User studies evaluated from the perspectives of creators, professional data journalists, and general readers indicate that our system assists creators in generating satisfactory visualization summaries of data events, enables the general audience to extract insights from the data through visual collages, and are well received by professionals.

Data exploration systems have become popular tools with which data analysts and others can explore raw data and organize their observations. However, users of such systems who are unfamiliar with their datasets face several challenges when trying to extract data events of interest to them. Those challenges include progressively discovering informative charts, organizing them into a logical order to depict a meaningful fact, and arranging one or more facts to illustrate a data event. To alleviate them, we propose VisGuide—a data exploration system that generates personalized recommendations to aid users’ discovery of data events in breadth and depth by incrementally learning their data exploration preferences and recommending meaningful charts tailored to them. As well as user preferences, VisGuide’s recommendations simultaneously consider sequence organization and chart presentation. We conducted two user studies to evaluate 1) the usability of VisGuide and 2) user satisfaction with its recommendation system. The results of those studies indicate that VisGuide can effectively help users create coherent and user-oriented visualization trees that represent meaningful data events.

Current practices to apply temperature scaling assume either a fixed, or a manually-crafted dynamically changing schedule. However, our studies indicate that the individual optimal trajectory for each class can change with the context. To this end, we propose context-aware temperature, a generalized approach to provide an individual optimal temperature trajectory over the context for each vocabulary, while allowing the temperature to be learned along with the remaining model parameters during training. Experiment results confirm that the proposed method significantly improves state-of-the-art language models, achieving a perplexity of 19.90 on Penn Treebank, 33.88 on WikiText-2, and 4.7 on WikiText-103.

Deep image classifiers often perform poorly when training data are heavily class-imbalanced. In this work, we propose a new regularization technique, Remix, that relaxes Mixup's formulation and enables the mixing factors of features and labels to be disentangled. Specifically, when mixing two samples, while features are mixed in the same fashion as Mixup, Remix assigns the label in favor of the minority class by providing a disproportionately higher weight to the minority class. By doing so, the classifier learns to push the decision boundaries towards the majority classes and balance the generalization error between majority and minority classes. We have studied the state-of-the art regularization techniques such as Mixup, Manifold Mixup and CutMix under class-imbalanced regime, and shown that the proposed Remix significantly outperforms these state-of-the-arts and several re-weighting and re-sampling techniques, on the imbalanced datasets constructed by CIFAR-10, CIFAR-100, and CINIC-10. We have also evaluated Remix on a real-world large-scale imbalanced dataset, iNaturalist 2018. The experimental results confirmed that Remix provides consistent and significant improvements over the previous methods.

Despite recent success in neural task-oriented dialogue systems, developing such a real-world system involves accessing large-scale knowledge bases (KBs), which cannot be simply encoded by neural approaches, such as memory network mechanisms. To alleviate the above problem, we propose , an end-to-end trainable text-to-SQL guided framework to learn a neural agent that interacts with KBs using the generated SQL queries. Specifically, the neural agent first learns to ask and confirm the customer’s intent during the multi-turn interactions, then dynamically determining when to ground the user constraints into executable SQL queries so as to fetch relevant information from KBs. With the help of our method, the agent can use less but more accurate fetched results to generate useful responses efficiently, instead of incorporating the entire KBs. We evaluate the proposed method on the AirDialogue dataset, a large corpus released by Google, containing the conversations of customers booking flight tickets from the agent. The experimental results show that significantly improves over previous work in terms of accuracy and the BLEU score, which demonstrates not only the ability to achieve the given task but also the good quality of the generated dialogues.

Analog computing hardwares, such as Processing-in-memory (PIM) accelerators, have gradually received more attention for accelerating the neural network computations. However, PIM accelerators often suffer from intrinsic noise in the physical components, making it challenging for neural network models to achieve the same performance as on the digital hardware. Previous works in mitigating intrinsic noise assumed the knowledge of the noise model, and retraining the neural networks accordingly was required. In this paper, we propose a noise-agnostic method to achieve robust neural network performance against any noise setting. Our key observation is that the degradation of performance is due to the distribution shifts in network activations, which are caused by the noise. To properly track the shifts and calibrate the biased distributions, we propose a "noise-aware" batch normalization layer, which is able to align the distributions of the activations under variational noise inherent in the analog environments. Our method is simple, easy to implement, general to various noise settings, and does not need to retrain the models. We conduct experiments on several tasks in computer vision, including classification, object detection and semantic segmentation. The results demonstrate the effectiveness of our method, achieving robust performance under a wide range of noise settings, more reliable than existing methods. We believe that our simple yet general method can facilitate the adoption of analog computing devices for neural networks.

Temperature scaling has been widely used as an effective approach to control the smoothness of a distribution, which helps the model performance in various tasks. Current practices to apply temperature scaling assume either a fixed, or a manually-crafted dynamically changing schedule. However, our studies indicate that the individual optimal trajectory for each class can change with the context. To this end, we propose contextual temperature, a generalized approach that learns an optimal temperature trajectory for each vocabulary over the context. Experimental results confirm that the proposed method significantly improves state-of-the-art language models, achieving a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. In-depth analyses show that the behaviour of the learned temperature schedules varies dramatically by vocabulary, and that the optimal schedules help in controlling the uncertainties. These evidences further justify the need for the proposed method and its advantages over fixed temperature schedules.

Hierarchical label structures widely exist in many machine learning tasks, ranging from those with explicit label hierarchies such as image classification to the ones that have latent label hierarchies such as semantic segmentation. Unfortunately, state-of-the-art methods often utilize cross-entropy loss which in-explicitly assumes the independence among class labels. Motivated by the fact that class members from the same hierarchy need to be similar to each others, we design a new training diagram called Hierarchical Complement Objective Training (HCOT). In HCOT, in addition to maximizing the probability of the ground truth class, we also neutralize the probabilities of rest of the classes in a hierarchical fashion, making the model take advantage of the label hierarchy explicitly. We conduct our method on both image classification and semantic segmentation. Results show that HCOT outperforms state-of-the-art models in CIFAR100, Imagenet, and PASCAL-context. Our experiments also demonstrate that HCOT can be applied on tasks with latent label hierarchies, which is a common characteristic in many machine learning tasks.

A visualization sequence is an effective representation of meaningful data stories.Existing visualization sequencing approaches use heuristics to arrange charts in a meaningful order.While they perform well in specific scenarios, they do not customize the generated sequences to individual users' preferences. In this work, we present VisGuide, an assistive data exploration system that helps a user create contextual visualization sequence trees by sequentially recommending meaningful charts tailoring to the user's preference on data exploration. Our results show that VisGuide can recommend chart sequences that interest users and are also considered meaningful by domain experts.

Temperature scaling has been widely used to improve performance for NLP tasks that utilize Softmax decision layer. Current practices in using temperature either assume a fixed value or a dynamically changing temperature but with a fixed schedule. Little has been known on an optimal trajectory of temperature that can change with the context. In this paper, we propose contextual temperature, a mechanism that allows temperatures to change over the context for each vocabulary, and to co-adopt with model parameters during training. Experimental results illustrated that contextual temperature improves over state-of-the-art language models significantly. Our model CT-MoS achieved a perplexity of 55.31 in the test set of Penn Treebank and a perplexity of 62.89 in the test set of WikiText-2. The in-depth analysis showed that the behavior of temperature schedule varies dramatically by vocabulary. The optimal temperature trajectory drops as the context becomes longer to suppress uncertainties in language modeling. These evidence further justified the need for contextual temperature and explained its performance advantage over fixed temperature or scheduling.

Label hierarchies widely exist in many vision-related problems, ranging from explicit label hierarchies existed in image classification to latent label hierarchies existed in semantic segmentation. Nevertheless, state-of-the-art methods often deploy cross-entropy loss that implicitly assumes class labels to be exclusive and thus independence from each other. Motivated by the fact that classes from the same parental category usually share certain similarity, we design a new training diagram called Hierarchical Complement Objective Training (HCOT) that leverages the information from label hierarchy. HCOT maximizes the probability of the ground truth class, and at the same time, neutralizes the probabilities of rest of the classes in a hierarchical fashion, making the model take advantage of the label hierarchy explicitly. The proposed HCOT is evaluated on both image classification and semantic segmentation tasks. Experimental results confirm that HCOT outperforms state-of-the-art models in CIFAR-100, ImageNet-2012, and PASCAL-Context. The study further demonstrates that HCOT can be applied on tasks with latent label hierarchies, which is a common characteristic in many machine learning tasks.

“Is there any pattern in location-based, mobile check-in activities?” “If yes, is it possible to accurately predict the intention of a user’s next check-in, given his/her check-in history?” To answer these questions, we study and analyze probably the largest mobile check-in datasets, containing 20 millions check-in activities from 0.4 million users. We provide two observations: “work-n-relax” and “diurnal-n-nocturnal” showing that the intentions of users’ check-ins are strongly associated with time. Furthermore, the category of each check-in venue, which reveals users’ intentions, has structure and forms taxonomy. In this paper, we propose Hierarchical LSTM that takes both (a) check-in time and (b) taxonomy structure of venues from check-in sequences into consideration, providing accurate predictions on the category of a user’s next check-in location. Hierarchical LSTM also projects each category into an embedding space, providing a new representation with stronger semantic meanings. Experimental results are poised to demonstrate the effectiveness of the proposed Hierarchical LSTM: (a) Hierarchical LSTM improves Accuracy@5 by 4.22% on average, and (b) Hierarchical LSTM learns a better taxonomy embedding for clustering categories, which improves Silhouette Coefficient by 1.5X.

The First International Workshop on Multimodal Understanding and Learning for Embodied Applications is held in Nice, France, in conjunction with ACM Multimedia 2019. Embodied applications require the learning and knowledge discovery process involving an agent, the environment, and actions, as well as the understanding and grounding of multiple modalities of input signals. Being one of the frontiers in AI research, it covers many of the applications in AI, such as robotics, autonomous driving, multimodal chatbots, or simulated games. The Workshop brings an exciting program with invited speeches, original research papers, and lively discussions on this new and exciting research area.

Adversarial robustness has emerged as an important topic in deep learning as carefully crafted attack samples can significantly disturb the performance of a model. Many recent methods have proposed to improve adversarial robustness by utilizing adversarial training or model distillation, which adds additional procedures to model training. In this paper, we propose a new training paradigm called Guided Complement Entropy (GCE) that is capable of achieving "adversarial defense for free," which involves no additional procedures in the process of improving adversarial robustness. In addition to maximizing model probabilities on the ground-truth class like cross-entropy, we neutralize its probabilities on the incorrect classes along with a "guided" term to balance between these two terms. We show in the experiments that our method achieves better model robustness with even better performance compared to the commonly used cross-entropy training objective. We also show that our method can be used orthogonal to adversarial training across well-known methods with noticeable robustness gain. To the best of our knowledge, our approach is the first one that improves model robustness without compromising performance.

Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the groundtruth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.

Recent studies on neural architecture search have shown that automatically designed neural networks perform as good as expert-crafted architectures. While most existing works aim at finding architectures that optimize the prediction accuracy, these architectures may have complexity and is therefore not suitable being deployed on certain computing environment (e.g., with limited power budgets). We propose MONAS, a framework for Multi-Objective Neural Architectural Search that employs reward functions considering both prediction accuracy and other important objectives (e.g., power consumption) when searching for neural network architectures. Experimental results showed that, compared to the state-ofthe-arts, models found by MONAS achieve comparable or better classification accuracy on computer vision applications, while satisfying the additional objectives such as peak power.

In this paper, we define a new problem related to social media, namely, the data-driven engineering of social dynamics. More precisely, given a set of observations from the past, we aim at finding the best short-term intervention that can lead to predefined long-term outcomes. Toward this end, we propose a general formulation that covers two useful engineering tasks as special cases, namely, pattern matching and profit maximization. By incorporating a deep learning model, we derive a solution using convex relaxation and quadratic-programming transformation. Moreover, we propose a data-driven evaluation method in place of the expensive field experiments. Using a Twitter dataset, we demonstrate the effectiveness of our dynamics engineering approach for both pattern matching and profit maximization, and study the multifaceted interplay among several important factors of dynamics engineering, such as solution validity, pattern-matching accuracy, and intervention cost. Finally, the method we propose is general enough to work with multi-dimensional time series, so it can potentially be used in many other applications.