Mining Structural Hole Spanners Through Information Diffusion in Social Networks.

Introduction

The theory of structural holes suggests that individuals would benefit from filling the ``holes'' (called as structural hole spanners) between people or groups that are otherwise disconnected. A few empirical studies have verified that structural hole spanners play a key role in the information diffusion. However, there is still lack of a principled methodology to detect structural hole spanners from a given social network.

On this page, we share three data sets, i.e., Coauthor, Twitter, and Inventor, for studying the problem of minining structural hole spanners through information diffusion in large social networks. They can be also used for many other purposes such as social influence analysis, opinion leader finding, and community detection.

Data set	#users	#Relationships
Coauthor	53,442	127,969
Twitter	465,023	833,590
Inventor	2,455,351	5,841,940

When you use the data sets for publications, please kindly cite the following paper.

@inproceedings{LouTang:13WWW,

author = {Tiancheng Lou and Jie Tang},

title = {Mining Structural Hole Spanners Through Information Diffusion in Social Networks},

booktitle = {WWW'13},

year = {2013},

} [PDF Download]

Abstract: In this paper, we precisely define the problem of mining top-k structural hole spanners in large-scale social networks and provide an objective (quality) function to formalize the problem. Two instantiation models have been developed to implement the objective function. For the first model, we present an exact algorithm to solve it and prove its convergence. As for the second model, the optimization is proved to be NP-hard, and we design an efficient algorithm with provable approximation guarantees.

We test the proposed models on three different networks: Coauthor, Twitter, and Inventor. Our study provides evidence for the theory of structural holes, e.g., 1% of Twitter users who span structural holes control 25% of the information diffusion on Twitter. We compare the proposed models with several alternative methods and the results show that our models clearly outperform the comparison methods. Our experiments also demonstrate that the detected structural hole spanners can help other social network applications, such as community kernel detection and link prediction. To the best of our knowledge, this is the first attempt to address the problem of mining structural hole spanners in large social networks.

Problem definition: Mining Top-k Stuctural Hole Spanners

We assume a setting in which a set V of n distinct users form l groups C = {C1, ..., Cl} (called communities). A utility function Q(v,C) is defined for each node to measure its degree to span structural holes.

Data Sets

Coauthor is a network of authors, collected by ArnetMiner\footnote{https://www.aminer.cn, an academic search system.}. The data set is obtained from~\cite{Tang:08KDD}. The network consists of 815,946 authors and 2,792,833 coauthorships. For the evaluation purpose, we create a sub-network, which contains coauthorships extracted from papers published at 28 major computer science conferences. These conferences cover six research areas: Artificial Intelligence (AI), Databases (DB), Data Mining (DM), Distributed Parallel Computing (DP), Graphics, Vision and HCI (GV), as well as Networks, Communications and Performance (NC). \footnote{AI: IJCAI, AAAI, ICML, UAI, UMAP, NIPS, and AAMAS; DB: VLDB, SIGMOD, PODS, ICDE, ICDT, and EDBT; DM: SIGKDD and ICDM; DP: PPoPP, PACT, IPDPS, ICPP, and Euro-Par; GV: SIGGRAPH, CVPR, and ICCV; NC: SIGCOMM, PERFORMANCE, SIGMETRICS, INFOCOM, and MOBICOM.}. Each conference has a group of program committee (PC) members. We extracted the PC member information from respective conference websites from 2008 to 2010. Computer scientists who served as PC members at conferences of different areas are considered as spanning structural holes across those areas. In total, we extracted 1,718 PC members, among whom 107 PC members span structural holes. Our goal is to identify those PC members who span structural holes from the coauthor network.

The data set contains the following files: coauthor graph and communities, [coauthor graph] [Communities] [Map ID to author name].

Ground truth: [PC Members: AI, DB, DM, DP, GV, NC]

Twitter is crawled from Twitter.com, a widely used microblogging system. The data set is obtained from~\cite{Hopcroft:11CIKM,LouTang:13TKDD}. The sub-network is comprised of 112,044 users, 468,238 following links among them, and all tweets (2,409,768 tweets) posted by these users. Here, we examine the role of structural hole spanners in the information diffusion process on Twitter.

There are two versions: Large: [Following graph]; Small: [Following graph] [Communities] [Cross Communities]

Inventor is a network of inventors, extracted from a large patent data set from USPTO\footnote{http://uspto.gov/, the US patent and trademark office.}. The data set is obtain from~\cite{Tang:12KDDPMiner}. The inventor network contains 2,445,351 inventors and 5,841,940 co-inventing relationships. Each company is considered as a community. We study how technologies spread across different companies via inventors who span structural holes.
The data set contains three files [Patent graph] [Inventor list] [Communities]

Related publications of the three data sets:

J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su. Arnetminer: Extraction and mining of academic social networks. In KDD’08, pages 990–998, 2008.
J. Hopcroft, T. Lou, and J. Tang. Who will follow you back? reciprocal relationship prediction. In CIKM’11, pages 1137–1146, 2011.
T. Lou, J. Tang, J. Hopcroft, Z. Fang, and X. Ding. Learning to predict reciprocity and triadic closure in social networks. TKDD, 2013, (accepted).
J. Tang, B. Wang, Y. Yang, P. Hu, Y. Zhao, X. Yan, B. Gao, M. Huang, P. Xu, W. Li, and A. K. Usadi. Patentminer: Topic-driven patent analysis and mining. In KDD’2012, pages 1366–1375, 2012.
Y. Yang, J. Tang, J. Keomany, Y. Zhao, Y. Ding, J. Li, and L. Wang. Mining Competitive Relationships by Learning across Heterogeneous Networks. In CIKM'2012, pages 1432-1441, 2012.

Codes

We develop two models for mining structural hole spanners from large social networks.

The HIS Algorithm
[C++ Code Download]

The first model, referred to as HIS considers the connectivity between opinion leaders and structural hole spanners. The user’s importance (authority) and the degree of spanning structural hole are defined in terms of one another in a mutual recursion.

where Ci is one community in social netowrk G=(V, E), v is a user, euv is a relationship between user u and v. S represents any subset of communities containing Ci in social network G. I(v,Ci) denotes the importance of user v in community Ci and H(v, S) represents the degree of user v spanning structural holes in the subset of communities S. Refer to our WWW'13 paper for more details.

The algorithm is summarized in Algorithm 1.

The Max-D Algorithm
[C++ Code Download]

The second model is based on the idea that users who span structural holes play an important role in information diffusion between different communities. Following this, we formalize the problem of structural hole spanner detection by minimal cut. Given this, the structural hole spanner detection problem can be cast as finding top-k nodes such that after removing these nodes, the minimal cut in G will be significantly reduced, i.e., the decrease of the minimal cut after removing will be maximized. The idea is natural, as structural hole spanners play bridging roles between communities. Without these structural hole nodes, the connections between different communities would be minimized.

where MC(G, C) denotes the minimal cut to separate all communities C in social netowrk G.

The algorithm is summarized in Algorithm 2.

Results

We have some interesting results on the three data sets:

1% of Twitter users who span structural holes control 25% of the information diffusion (retweeting). This provides a strong evidence for the theory of structural hole.
Opinion leaders control information diffusion within a community, while structural hole spanners control information diffusion across communities.
We apply the detected structural hole spanners to help communities detection and link prediction, two important applications in social networks. Results demonstrate that the structural hole information can significantly improve the quality (+10% in terms of F1-score) of communities detection and improve the performance (+3-4% by F1-score) of predicting the type of relationships in two different networks.

Some results on the Twitter network. For more results, please refer to our WWW 2013 paper.

References

Tiancheng Lou and Jie Tang. Mining Structural Hole Spanners Through Information Diffusion in Social Networks. WWW'13, 2013. [PDF]