AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We propose a unified Contextual Self-Attention Network to address the three properties

CSAN: Contextual Self-Attention Network for User Sequential Recommendation.

MM '18: ACM Multimedia Conference Seoul Republic of Korea October, 2018, pp.447-455, (2018)

Cited by: 25|Views83
EI

Abstract

The sequential recommendation is an important task for online user-oriented services, such as purchasing products, watching videos, and social media consumption. Recent work usually used RNN-based methods to derive an overall embedding of the whole behavior sequence, which fails to discriminate the significance of individual user behavior...More

Code:

Data:

0
Introduction
  • With the rapid development of the Internet, some applications of sequential scenario have become pervasive and multilateral, such “Using Data Mining Technology to Analyze the Law of Population Migration”.
  • Big data, Sociology, Dynamics, Demographics Machine learning AI.
  • TensorFlow reinforcement learning Li Kaifu Deep Learning Timeline.
  • Matrilineal system “Using Data Mining Technology to Analyze the Law of Population Migration” Data mining, Big data, Sociology of Scientific Knowledge, SSK
Highlights
  • With the rapid development of the Internet, some applications of sequential scenario have become pervasive and multilateral, such “Using Data Mining Technology to Analyze the Law of Population Migration”

    Data mining, Big data, Sociology, Dynamics, Demographics Machine learning AI

    TensorFlow reinforcement learning Li Kaifu Deep Learning Timeline

    Statistics Sociology

    Economic sociology Demography

    Matrilineal system “Using Data Mining Technology to Analyze the Law of Population Migration” Data mining, Big data, Sociology of Scientific Knowledge, SSK
  • We propose a novel contextual self-attention network for the sequential recommendation, which can leverage user historical behaviors in a more effective manner and have high computational efficiency
  • We propose to employ embedding network, self-attention mechanism and position encoding to deal with the heterogeneity, polysemy, and dynamic contextual dependency of user sequential behaviors
  • Our work differs from the above approaches in that we introduce a unified Recurrent Neural Network (RNN)/Convolutional Neural Network (CNN)-free user behavior modeling framework based solely on self-attention for sequential recommendation whose attention mechanism works on the feature level instead of element level, and use position encoding matrices to model dynamic contextual dependency instead of time encoding
  • We introduce a contextual self-attention network, Contextual Self-Attention Network (CSAN), for modeling the sequential behaviors in recommendation tasks
  • CSAN is a unified framework which can model with multi-type actions and multi-modal content based solely on attention mechanism
Methods
  • 4.1 Evaluation Datasets

    The authors evaluate the proposed method on two real-world datasets: Amazon product dataset and Zhihu activity dataset.
  • Amazon3 is an e-commerce website where users interacts with the commodity.
  • We take a series of large categories including ‘Automotive’, ‘Office Products’, ‘Toys and Games’, ‘Clothing, Shoes and Jewelry’, and ‘Video Games’ for experiment.
  • This set of data is notable for its high sparsity and variability.
  • The authors retain some of the characteristics used to construct the user sequential behaviors
Results
  • AUC [25], the area under the ROC curve, is a commonly used metric for evaluating the quality of a ranking list.
  • The authors report the performance of each method on the test set on both Amazon datasets and.
  • Zhizhu dataset in terms of the following ranking metrics: AUC = 1 U u ∈U 1 |J ||J′|.
  • Pu, j is the predicted probability that a user u ∈ U may act on i in the test set.
  • A higher value of AUC indicates better performance for ranking performance.
  • AUC from random guess is 0.5 and the best result is 1
Conclusion
  • The authors introduce a contextual self-attention network, CSAN, for modeling the sequential behaviors in recommendation tasks.
  • CSAN is a unified framework which can model with multi-type actions and multi-modal content based solely on attention mechanism.
  • The authors analyze the proposed model on both single-type behaviors datasets (Amazon) and multitype multi-modal behaviors dataset (Zhihu).
  • The experiment results show that CSAN achieves promising performances over the existing highly optimized individual models, and demonstrates its suitability for modeling complex behavior patterns
Tables
  • Table1: Statistics of Amazon product dataset
  • Table2: Statistics of Zhihu activity dataset
  • Table3: Ranking results on Amazon datasets and Zhihu dataset (higher is better). The best performance in each case is highlighted
Download tables as Excel
Related work
  • Sequential Recommendation Sequential recommendation problem is usually cast as sequence prediction problem. Most existing approaches focus on Markov Chain (MC) based methods and Neural network-based methods. Scalable sequential models usually rely on MC to capture sequential patterns [7, 26], where an L-order Markov chain makes recommendations based on L previous actions. However, a major problem of MC based models is that all the components are independently combined, indicating that it makes strong independence assumptions among multiple factors [31]. Recently, a Matrix Factorization (MF) based approach factorizes the matrix of transition probability from the current item to the next one into the latent factors [3]. However, MF easily suffers from sparsity issues due to the power-law distributed data in the real word [11]. Inspired by the great power of matrix factorization, Factorized Personalized Markov Chain (FPMC) [26] combines the power of MF and MC to factorize the transition matrix over underlying MC to model personalized sequential behaviors for the next-basket recommendation. FPMC and its variant [2] improve this method by factorizing this transition matrix into two latent and low-rank sub-matrices. All the MC-based methods have the same deficiency that these recommenders only obtain the local sequential behaviors between every two adjacent items.
Funding
  • This work was supported in part by the National Key Research and Development Program of China (No 2017YFB1002804), the National Natural Science Foundation of China under Grants 61432019, 61720106006, 61572503 and 61702509, the Key Research Program of Frontier Sciences, CAS, Grant NO
Study subjects and analysis
users: 10458
Zhihu activity dataset To collect multi-type behaviors dataset for the sequential recommendation, we crawl Zhihu users’ dynamic activities data. Starting with a specific user, we finally collect 10458 users through their following and follower lists layer by layer. For each user, we crawl his/her dynamic activities for one year, including multi-type actions and multi-modal content

Reference
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473 (2014).
    Findings
  • Chen Cheng, Haiqin Yang, Michael R Lyu, and Irwin King. 2013. Where You Like to Go Next: Successive Point-of-Interest Recommendation.. In IJCAI, Vol. 13. 2605–2611.
    Google ScholarLocate open access versionFindings
  • Szu-Yu Chou, Yi-Hsuan Yang, Jyh-Shing Roger Jang, and Yu-Ching Lin. 2016. Addressing cold start for next-song recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 115–118.
    Google ScholarLocate open access versionFindings
  • Junyoung Chung, ÃĞaglar GülÃğehre, Kyunghyun Cho, and Yoshua Bengio. 201Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/1412.3555 (2014).
    Findings
  • Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-Attention Neural Networks for Reading Comprehension. In ACL.
    Google ScholarFindings
  • Yong Du, Wei Wang, and Liang Wang. 2015. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1110–1118.
    Google ScholarLocate open access versionFindings
  • Shanshan Feng, Xutao Li, Yifeng Zeng, Gao Cong, Yeow Meng Chee, and Quan Yuan. 2015. Personalized Ranking Metric Embedding for Next New POI Recommendation.. In IJCAI. 2069–2075.
    Google ScholarFindings
  • Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional Sequence to Sequence Learning. In ICML.
    Google ScholarFindings
  • Ruining He, Wang-Cheng Kang, and Julian McAuley. 2017. Translation-based Recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 161–169.
    Google ScholarLocate open access versionFindings
  • Ruining He and Julian McAuley. 2016. Fusing similarity models with markov chains for sparse sequential recommendation. In Data Mining (ICDM), 2016 IEEE 16th International Conference on. IEEE, 191–200.
    Google ScholarLocate open access versionFindings
  • Liang Hu, Longbing Cao, Jian Cao, Zhiping Gu, Guandong Xu, and Dingyu Yang. 2016. Learning Informative Priors from Heterogeneous Domains to Improve Recommendation in Cold-Start User Domains. ACM Transactions on Information Systems (TOIS) 35, 2 (2016), 13.
    Google ScholarLocate open access versionFindings
  • Minghao Hu, Yuxing Peng, and Xipeng Qiu. 2017. Reinforced mnemonic reader for machine comprehension. CoRR, abs/1705.02798 (2017).
    Findings
  • Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1725–1732.
    Google ScholarLocate open access versionFindings
  • Yoon Kim. 20Convolutional Neural Networks for Sentence Classification. In EMNLP.
    Google ScholarFindings
  • Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.
    Google ScholarFindings
  • Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning. 1188–1196.
    Google ScholarLocate open access versionFindings
  • Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 20A Structured Self-attentive Sentence Embedding. CoRR abs/1703.03130 (2017).
    Findings
  • Duen-Ren Liu, Chin-Hui Lai, and Wang-Jung Lee. 2009. A hybrid of sequential rules and collaborative filtering for product recommendation. Information Sciences 179, 20 (2009), 3505–3519.
    Google ScholarLocate open access versionFindings
  • Qiang Liu, Shu Wu, and Liang Wang. 2017. Multi-behavioral sequential prediction with recurrent log-bilinear model. IEEE Transactions on Knowledge and Data Engineering 29, 6 (2017), 1254–1267.
    Google ScholarLocate open access versionFindings
  • Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. Predicting the Next Location: A Recurrent Model with Spatial and Temporal Contexts.. In AAAI. 194–200.
    Google ScholarLocate open access versionFindings
  • Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 43–52.
    Google ScholarLocate open access versionFindings
  • Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association.
    Google ScholarLocate open access versionFindings
  • Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 5528–5531.
    Google ScholarLocate open access versionFindings
  • Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. Rnnlm-recurrent neural network language modeling toolkit. In Proc. of the 2011 ASRU Workshop. 196–201.
    Google ScholarLocate open access versionFindings
  • Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence. AUAI Press, 452–461.
    Google ScholarLocate open access versionFindings
  • Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web. ACM, 811–820.
    Google ScholarLocate open access versionFindings
  • Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16126
    Locate open access versionFindings
  • Huan Song, Deepta Rajan, Jayaraman J. Thiagarajan, and Andreas Spanias. 2018. Attend and Diagnose: Clinical Time Series Analysis Using Attention Models. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018. https://www.aaai.org/ocs/index.php/ AAAI/AAAI18/paper/view/16325
    Locate open access versionFindings
  • Jiaxi Tang and Ke Wang. 2018. Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. (2018).
    Google ScholarFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 6000–6010.
    Google ScholarLocate open access versionFindings
  • Pengfei Wang, Jiafeng Guo, Yanyan Lan, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2015. Learning hierarchical representation model for nextbasket recommendation. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. ACM, 403–412.
    Google ScholarLocate open access versionFindings
  • Shoujin Wang, Liang Hu, Longbing Cao, Xiaoshui Huang, Defu Lian, and Wei Liu. 2018. Attention-based Transactional Context Embedding for Next-Item Recommendation. (2018).
    Google ScholarFindings
  • Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems 2, 1-3 (1987), 37–52.
    Google ScholarLocate open access versionFindings
  • Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.
    Google ScholarFindings
  • Shuangfei Zhai, Keng-hao Chang, Ruofei Zhang, and Zhongfei Mark Zhang. 2016. Deepintent: Learning attentions for online advertising with recurrent neural networks. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1295–1304.
    Google ScholarLocate open access versionFindings
  • Yuyu Zhang, Hanjun Dai, Chang Xu, Jun Feng, Taifeng Wang, Jiang Bian, Bin Wang, and Tie-Yan Liu. 2014. Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks.. In AAAI, Vol. 14. 1369–1375.
    Google ScholarLocate open access versionFindings
  • Lei Zheng, Vahid Noroozi, and Philip S Yu. 2017. Joint deep modeling of users and items using reviews for recommendation. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, 425–434.
    Google ScholarLocate open access versionFindings
  • Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. 2018. ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16216
    Locate open access versionFindings
  • Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018. 1059–1068. https://doi.org/10.1145/3219819.3219823
    Locate open access versionFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科