How Do Your Friends on Social Media Disclose Your Emotions?

Introduction

Extracting emotions from images has attracted much interest, in particular with the rapid development of social networks. The emotional impact is very important for understanding the intrinsic meanings of images. Despite many studies having been done, most existing methods focus on image content, but ignore the emotion of the user who published the image. One interesting question is: How does social effect correlate with the emotion expressed in an image? Specifically, can we leverage friends interactions (e.g., discussions) related to an image to help extract the emotions? In this paper, we formally formalize the problem and propose a novel emotion learning method by jointly modeling images posted by social users and comments added by their friends. One advantage of the model is that it can distinguish those comments that are closely related to the emotion expression for an image from the other irrelevant ones. Experiments on an open Flickr dataset show that the proposed model can significantly improve (+37.4% by F1) the accuracy for inferring user emotions. More interestingly, we found that half of the improvements are due to interactions between 1.0% of the closest friends.

Problem formulation

We are given a set of images M. For each image m \in M, we have the user vm who posts m, and a set of comments Dm which are posted about m. Also, for each comment d \in Dm, we know the user vd who posts d. Our goal is to determine the emotional status of user vm when she posted the image m.

More precisely, we use a T dimensional vector xm = (\forall_t xmt \in \mathbb{R}) to represent the image m, where each dimension indicates one of m's visual features (e.g., saturation, cool color ratio, etc.). Each comment d is regarded as a Nd-sized bag of words wd, where each word is chosen from a vocabulary of size W. For users' emotional status, in this work, we mainly consider Ekman's six emotions: happiness, surprise, anger, disgust, fear, sadness.

The users who has posted either an image or a comment are grouped as a user set V. All comments are denoted as a set D. We incorporate images, comments, and social network information in a single heterogeneous social network.

Codes

We develop a novel emotion learning method. Figure below demonstrate the graphical representation of our model. The purple block can be regarded as a mixture Gaussian, which describes the visual features of images. The yellow block can be seen as a LDA, which describes the comment information. The green block models how likely a comment will be influenced by the relevant image, which integrates images and comments into an uniform model. The green block considers social tie information. Particularly, one is more easily to understand her closed friends' emotional status, and will be more influenced.

Figure 1. Graphical representation of the proposed model.

Please find the generative process of our model, and a Gibbs-sampling based learning algorithm to estimate unknown parameters in our paper.

Visual Features. Table 1 describes the visual features, which are proposed by Wang et al.~\cite{xiaohui:13}, we used in this paper. Traditional features include SIFT~\cite{lowe1999object}. The features we utilized are mainly aesthetics based and have much stronger artistic evidences and nuanced connections with human feelings, as revealed by various art theories. Thus the features used in this work are more appropriate to depict emotions, especially on highly subjective abstract paintings. As~\cite{xiaohui:13} shows, based on the same classifier, these features improve the performance on the ABSTRACT dataset~\cite{machajdik2010affective} by 18% averagely, compared with the visual features proposed in~\cite{machajdik2010affective}.

Table 1. Summary of visual features utilized in this work.

C++ Code Download

Dataset

On this page, we share our Flickr dataset, which consists of 354,192 randomly downloaded images posted by 4,807 users. For each image, we also collect its tags and all comments. Thus we get 557,177 comments posted by 6,735 users in total. We also record the authors' profiles, including the authors' id, alias and their contact list. Furthermore, the contact list is a list of ids which the user follows on Flickr, so we are able to figure out the relationship between two users.

For training and evaluating the proposed model for inferring emotions, we firstly need to know the primary emotions of the images. Manually acquiring a large labeled image dataset for evaluation is time-consuming and tedious. Thus we follow a comprised method of using tags supplied by users. This method was also used in~\cite{xiepicture}~\cite{chen2013neil}~\cite{hwang2013discriminative}. Particularly, we first manually define the word list for each of the six emotion categories based on WordNet (http://wordnet.princeton.edu/) and HowNet (http://www.keenage.com/). Next we compare the adjective words in images' tags with the word lists. The emotion category whose word list has the most same words as the tag words is finally assigned to the image. This has left us six sets of images, consisting of 145,946, 33,854, 22,040, 9,491, 54,637 and 35,935 images each.

Data Download

Results

We compare the following baseline methods in our experiments:

SVM. This method simply regards the visual features of images as inputs and uses a SVM as a classifier. It then uses the trained classifier to infer the emotions. \hide{Notice that neither comment information or social tie information is not considered in this method.} We use LIBSVM~\cite{chang:libsvm} in this work.

PFG. This method is used in~\cite{jia2012can} to infer images' emotions. More specifically, it considers both the color features and social correlations among images and utilizes a partially-labeled factor graph model~\cite{tang2011learning} as a classifier.

LDA+SVM. This method first uses LDA~\cite{blei:lda} to extract hidden topics from user comments. It then uses the visual features of images, topic distributions of comments, and relationships between users as features to train SVM as a classifier. We use this method to compare the effectiveness of the joint modeling part of the proposed model and this alternative method.

EL+SVM. This method employs the proposed emotion learning method (EL) to learn the topic distributions of images. It then uses SVM as a classifier. For parameter configuration, we empirically set

K=6

\alpha=0.1

\gamma=0.1

\tau=0.01

b_0=1

, and

b_1=2

Table 2 shows the evaluation results.

Table 2. Performance of emotion inference

Parameter Analysis. We study how the parameters of the proposed model influence the performance. Figure 2(a) shows the performance of the proposed model by varying the number of latent variables K from 2 to 10. In general, increasing K often obtain a performance improvement. However, when K is large enough (K > 5), the performance tends to be stable, which implies that K should not less than the number of categories in the emotion inference task. We then study how the hyper-parameter \alpha influences the performance. Figure 2(b) shows the performance of the proposed method with changing \alpha. We see the performance is stable, which indicates the proposed method is not sensitive to the hyper-parameter \alpha. Figure 2(c) shows how the performance is influenced by b1:b0. Overall, too small ratio of b1 and b0 hurts the performance by not leveraging comment information sufficiently, while too large valued ratio causes over-fitting. Indeed, how to find a best combination of b0 and b1 efficiently is a remaining challenge needed to be solved in our future work. We finally study the convergence of the proposed method. As Figure 2(d) shows, the proposed method converges within 80 iterations in all tasks.

Figure 2. Parameter analysis.Y-axis in all figures denote the F1-score in emotion inference.

Image Interpretations. Compared with traditional methods, a more clear interpretation of the correlations between images and emotions is one of the advantages of the proposed model. Table 3 demonstrates how each visual feature xt distributes over different emotions by the learned ut of the proposed model.

Table 3. Image interpretations. We demonstrate how each visual feature distributes over each category of images by

\mu_{et}

learned by the proposed model. The visual features include saturation (SR), saturation contrast (SRC), bright contrast (BRC), cool color ratio (CCR), figure-ground color difference (FGC), figure-ground area difference (FGA), background texture complexity (BTC), and foreground texture complexity (FTC). We also show a user's friends response after the user publishes different categories of images.

For example, in the Happiness category, images tend to have high saturation and high bright contrast, which both bring out a sense of peace and joy. On the contrary, images in Sadness category tend to have lower saturation and saturation contrast, which both convey a sense of dullness and obscurity. Sad images also have low texture complexity, which gives a feeling of pithiness and coherence.

Social Response column of Table 3 elucidate how a user's friends response after the user publishes a certain category of images. Specifically, the red x-axises indicate the number of comments a friend leaves. And the red y-axises denote the number of friends who leave certain number of comments. Bars colored by yellow show the proportion of a user's friends with different emotions when leaving the comments. From the figure, we see an interesting phenomenon: the positive emotion (happiness) attracts more response (+4.4 times), and more easily to influences others to have the same emotion. On the contrast, when a user publishes a sad image, her friends' emotions distribute more uniformly, which implies sadness has less influential compared with happiness.

How Do Your Friends on Social Media Disclose Your Emotions?

Introduction

Codes

Dataset

Results

References

Yang Yang, Jia Jia, Shumei Zhang, Boya Wu, Juanzi Li, and Jie Tang. How Do Your Friends on Social Media Disclose Your Emotions? In AAAI'14, 2014.