In this paper we propose a Structural Deep Network Embedding method, namely SDNE
Structural Deep Network Embedding
KDD, pp.1225-1234, (2016)
Network embedding is an important method to learn low-dimensional representations of vertexes in networks, aiming to capture and preserve the network structure. Almost all the existing network embedding methods adopt shallow models. However, since the underlying network structure is complex, shallow models cannot capture the highly non-li...更多
下载 PDF 全文
- Networks are ubiquitous and many real-world applications need to mine the information within these networks.
- 15.00 DOI: http://dx.doi.org/10.1145/2939672.2939753 ment targeting often needs to cluster the users into communities in the social network.
- An effective way is to embed networks into a low-dimensional space, i.e. learn vector representations for each vertex, with the goal of reconstructing the network in the learned embedding space.
- As a result, mining information in networks, such as information retrieval , classification , and clustering , can be directly conducted in the low-dimensional space
- Nowadays, networks are ubiquitous and many real-world applications need to mine the information within these networks
- With the first-order and second-order proximity, we can well characterize the local and global network structure, respectively. To preserve both the local and global network structure in the deep model, we propose a semi-supervised architecture, in which the unsupervised component reconstructs the second-order proximity to preserve the global network structure while the supervised component exploits the first-order proximity as the supervised information to preserve the local network structure
- We report the results of the generalization of the network representations generated by different embedding methods on three classic data mining and machine learning applications, i.e. multilabel classification, link prediction and visualization
- Before proceeding to evaluate the generalization of the proposed method on real-world applications, we first provide a basic evaluation on different network embedding methods with respect to their capability of network reconstruction. The reason for this experiment is that a good network embedding method should ensure that the the learned embeddings can preserve the original network structure
- The result on Mean Average Precision is shown in Table 4
- The authors evaluate the proposed method on several realworld datasets and applications.
- In order to comprehensively evaluate the effectiveness of the representations, the authors use five networked datasets, including three social networks, one citation network and one language network, for three real-world applications, i.e. multi-label classification, link prediction and visualization.
- There are overall 39 different categories for BLOGCATALOG, 195 categories for FLICKR and 47 categories for categories
- These categories can be used as the ground-truth of each vertex.
- They can be evaluated on the multi-label classification task
- Before proceeding to evaluate the generalization of the proposed method on real-world applications, the authors first provide a basic evaluation on different network embedding methods with respect to their capability of network reconstruction.
- The reason for this experiment is that a good network embedding method should ensure that the the learned embeddings can preserve the original network structure.
- The authors propose a Structural Deep Network Embedding, namely SDNE, to perform network embedding.
- To capture the highly non-linear network structure, the authors design a semi-supervised deep model, which has multiple layers of nonlinear functions.
- To further address the structure-preserving and sparsity problem, the authors jointly exploit the first-order proximity and second-order proximity to characterize the local and global network structure.
- By jointly optimizing them in the semi-supervised deep model, the learned representations are local-global structurepreserved and are robust to sparse networks.
- The results demonstrate substantial gains of the method compared with state-of-the-art
- Table1: Terms and Notations
- Table2: Statistics of the dataset
- Table3: Neural Network Structures
- Table4: MAP on ARXIV-GRQC and BLOGCATALOG on reconstruction task
- Table5: Table 5
- Table6: KL-divergence for the 20-NEWSGROUP dataset
- 2.1 Deep Neural Network
Representation learning has long been an important problem of machine learning and many works aim at learning representations for samples [3, 35]. Recent advances in deep neural networks have witnessed that they have powerful representations abilities  and can generate very useful representations for many types of data. For example,  proposed a seven-layer convolutional neural network to generate image representations for classification.  proposed a multimodal deep model to learn image-text unified representations to achieve cross-modality retrieval task.
However, to the best of our knowledge, there have been few deep learning works handling networks, especially learning network representations. In , Restricted Boltzmann Machines were adopted to do collaborative filtering.  adopted deep autoencoder to do graph clustering.  proposed a heterogeneous deep model to do heterogeneous data embedding. We differ from these works in two aspects. Firstly, the goals are different. Our work focuses on learning low-dimensional structure-preserved network representations which can be utilized among tasks. Secondly, we consider both the first-order and second-order proximity between vertexes to preserve the local and global network structure. But they only focus on one-order information.
- This work was supported by National Program on Key Basic Research Project, No 2015CB352300; National Natural Science Foundation of China, No 61370022, No 61531006, No 61472444 and No 61210008
- M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
- Y. Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1–127, 2009.
- Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1798–1828, 2013.
- S. Cao, W. Lu, and Q. Xu. Grarep: Learning graph representations with global structural information. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 891–900. ACM, 2015.
- S. Chang, W. Han, J. Tang, G.-J. Qi, C. C. Aggarwal, and T. S. Huang. Heterogeneous network embedding via deep architectures. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 119–128. ACM, 2015.
- N. S. Dash. Context and contextual word meaning. SKASE Journal of Theoretical Linguistics, 5(2):21–31, 2008.
- D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research, 11:625–660, 2010.
- R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. Liblinear: A library for large linear classification. The Journal of Machine Learning Research, 9:1871–1874, 2008.
- K. Georgiev and P. Nakov. A non-iid framework for collaborative filtering with restricted boltzmann machines. In ICML-13, pages 1148–1156, 2013.
- G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29(6):82–97, 2012.
- G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527–1554, 2006.
- G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.
- M. Jamali and M. Ester. A matrix factorization technique with trust propagation for recommendation in social networks. In Proceedings of the fourth ACM conference on Recommender systems, pages 135–142. ACM, 2010.
- E. M. Jin, M. Girvan, and M. E. Newman. Structure of growing social networks. Physical review E, 64(4):046132, 2001.
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
- J. Leskovec, J. Kleinberg, and C. Faloutsos. Graph evolution: Densification and shrinking diameters. ACM Transactions on Knowledge Discovery from Data (TKDD), 1(1):2, 2007.
- D. Liben-Nowell and J. Kleinberg. The link-prediction problem for social networks. Journal of the American society for information science and technology, 58(7):1019–1031, 2007.
- T. Liu and D. Tao. Classification with noisy labels by importance reweighting. TPAMI, (1):1–1.
- D. Luo, F. Nie, H. Huang, and C. H. Ding. Cauchy graph embedding. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 553–560, 2011.
- A. Y. Ng, M. I. Jordan, Y. Weiss, et al. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 2:849–856, 2002.
- B. Perozzi, R. Al-Rfou, and S. Skiena. Deepwalk: Online learning of social representations. In SIGKDD, pages 701–710. ACM, 2014.
- S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323–2326, 2000.
- R. Salakhutdinov and G. Hinton. Semantic hashing. International Journal of Approximate Reasoning, 50(7):969–978, 2009.
- B. Shaw and T. Jebara. Structure preserving embedding. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 937–944. ACM, 2009.
- R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642.
- J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web, pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015.
- L. Tang and H. Liu. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2009.
- L. Tang and H. Liu. Scalable learning of collective behavior based on sparse social dimensions. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 1107–1116. ACM, 2009.
- J. B. Tenenbaum, V. De Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319–2323, 2000.
- F. Tian, B. Gao, Q. Cui, E. Chen, and T.-Y. Liu. Learning deep representations for graph clustering. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 1293–1299, 2014.
- L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85, 2008.
- S. V. N. Vishwanathan, N. N. Schraudolph, R. Kondor, and K. M. Borgwardt. Graph kernels. The Journal of Machine Learning Research, 11:1201–1242, 2010.
- D. Wang, P. Cui, M. Ou, and W. Zhu. Deep multimodal hashing with orthogonal regularization. In Proceedings of the 24th International Conference on Artificial Intelligence, pages 2291–2297. AAAI Press, 2015.
- Y. Weiss, A. Torralba, and R. Fergus. Spectral hashing. In Advances in neural information processing systems, pages 1753–1760, 2009.
- C. Xu, D. Tao, and C. Xu. Multi-view intact space learning. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 37(12):2531–2544, 2015.
- J. Zhuang, I. W. Tsang, and S. Hoi. Two-layer multiple kernel learning. In International conference on artificial intelligence and statistics, pages 909–917, 2011.