Is a “happy dog” more “happy” than “dog”?-Adjective and Noun Contributions for Adjective-Noun Pair prediction

Page 1. Is a “happy dog” image more “happy” than “dog”? Analyzing Adjective and Noun Visual Contributions Dèlia Fernández Víctor Campos Brendan Jou Xavier Giró-i-Nieto Shih-Fu Chang Introduction 2. The Adjective-Noun Pairs (ANPs) CLASSIFIER 1. Objective: Predict Sentiment/Emotions HAPPY DOG Solution: Mid-level Affective Representation Foggy Morning Bright Day Cute Dog Happy Kids 3. Hypothesis: Studied Architectures Adjective and Noun contribute differently depending on the ANP. Visual Features Semantic Features Class Probability Vector 0.2 0.1 0.25 0.00 0.01 … 0.12 0.011 Visual-ANPNet Semantic-ANPNet Noun Visual Features Adjective Visual Features Noun Semantic Features Adjective Semantic Features % Adjective contribution % Noun contribution Backpropagate Contributions CLASSIFIER Problem: Affective Gap 4. Traditional Method: Cute Dog Fine-tuning Single Branch Architectures …