Visual dialog is a task of answering a question given an input image, a historical dialog about the image and often requires to retrieve visual and textual facts about the question. This problem is different from visual question answering (VQA), which only relies on visual grounding estimated from an image and question pair, while visual dialog task requires interactions among a question, an input image and a historical dialog. Most methods rely on one-turn attention network to obtain facts w.r.t. a question. However, the information transition phenomenon which exists in these facts restricts these methods to retrieve all relevant information. In this paper, we propose a multi-turn attentional memory network for visual dialog. Firstly, we propose a attentional memory network that maintains image regions and historical dialog in two memory banks and attends the question to be answered to both the visual and textual banks to obtain multi-model facts. Further, considering the information transition phenomenon, we design a multi-turn attention architecture which attend to memory banks multiple turns to retrieve more facts in order to produce a better answer. We evaluate the proposed model in on VisDial v0.9 dataset and the experimental results prove the effectiveness of the proposed model.
Local feature descriptors play an important role in facial expression recognition. Local Binary Pattern (LBP) only considers the signal information of the difference between the gray value of the center pixel and the neighbor pixel. It does not take the magnitude information into consideration and has poor robustness. Local Mapped Pattern (LMP) is not ideal for discriminating differences between different textures and experimental result is not excellent. This paper proposes a novel feature descriptor based on gray-level difference mapping, called Local Double Binary Mapped Pattern (LDBMP). This new approach is an improvement over the previous LBP and LMP, not only retains the advantages of LBP and LMP but also preserves the information of magnitude and captures nuances that occur in the image. In our experiments, the new descriptor performs favorably.
VQA is a multimodal joint learning task of AI-complete. The goal of our work is to present a text-guided dual-branch attention network (TDAN) for visual question answer. The focus of different attention models is different, and merging multiple models can produce better answers. So TDAN has two branches (i.e., two sub-models) and it first separately generates predictions for the answers through dual branch, and then generates weight to dual branch through question guidance. Thus, the key layers of two branches are merged into one and produce final output. We also exploit a one-dimensional gated convolutional neural network (1D-GCNN) encoding question texts and text embedding method with position information for high efficiency. In experiments, ours model is superior to the previous models, on the VQA 2.0 dataset from 61.89% to 63.94%, and on the COCOQA dataset from 62.5% to 63.98%.
Creativity is a fundamental feature of human intelligence, and a challenge for artificial intelligence (AI). In recent years, AI has gained tremendous development in many single tasks with single models, such as classification, detection and parsing. As the development continued, AI has been increasingly used for more complex tasks, multitasking for example, and then research in multimodal fusion naturally became a new trend. In this paper, we propose a multimodal fusion framework and system to generate traditional Chinese paintings. We select suitable existing networks for different elements generation in this oldest continuous artistic traditions artwork, and finally fusion these networks and elements to create a complete new painting. Meanwhile, we propose a divide-and-conquer strategy to generate large images with limited GPU resources. In our end-to-end system, a large image becomes a traditional Chinese painting in minutes automatically. It shows that our multimodal fusion framework works well and AI methods has good performance in traditional Chinese painting creation.
Residual Networks (ResNet) and Dense Convolutional Networks (DenseNet) have shown great success in lots of high-level computer vision applications. In this paper, we propose a novel network with Local Dense and Adaptive Global Residual (LD+AGR) frameworks for fast and accurate image denoising. More precisely, we combine local residual/dense with global residual/dense to investigate the best performance dealing with image denoising problem. In particular, local/global residual/dense means the connection way of inner/outer recursive blocks. And residual/dense represents combining layers by summation/concatenation. Furthermore, when combining skip connections, we add some adaptive and trainable scaling parameters, which could adjust automatically during training to balance the importance of different layers. Numerous experiments demonstrate that the proposed network performs favorably against the state-of-the-art methods in terms of quality and speed.
