AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
While our experiments are focused on the knowledge bases of locations and attributes that supports Google Maps, our proposed framework is useful for constructing KBs with tunable precision from unlabeled side information and noisy categorical observations collected via crowdsourc...

Constructing High Precision Knowledge Bases with Subjective and Factual Attributes.

KDD, (2019): 2050.0-2058.0

Cited by: 3|Views48
EI

Abstract

Knowledge bases (KBs) are the backbone of many ubiquitous applications and are thus required to exhibit high precision. However, for KBs that store subjective attributes of entities, e.g., whether a movie is kid friendly, simply estimating precision is complicated by the inherent ambiguity in measuring subjective phenomena. In this work, ...More

Code:

Data:

0
Introduction
  • Structured knowledge repositories–known as knowledge bases (KBs)–are the backbone of many high-impact applications and services.
  • Consider organizing a lunch meeting and issuing a KB query for cafes that are good for groups.
  • Since most KBs are built using noisy automated methods, special consideration must be paid.
  • Previous work echos this concern: in addition to employing trained automated components for data collection and prediction of missing values, systems that build KBs often turn to humans–largely considered to be more precise than the automated methods–for
Highlights
  • Structured knowledge repositories–known as knowledge bases (KBs)–are the backbone of many high-impact applications and services
  • For example: the Netflix1 movie recommendation engine relies on a KB of user-movie-rating triples, Google Maps2 is built atop a KB of geographic points of interest and PubMed3 offers a handful of tools that operate on its citation KB of biomedical research
  • While we study the location-attribute setting, our yes rate modeling framework can be applied in many instances of hybrid KB construction that rely on collecting categorical observations via crowdsourcing
  • We note that for the subjective attributes, our evaluation scheme produces a conservative estimate of model quality, which, we argue, is better than a non-conservative estimate given the importance of mitigating false positives
  • We study constructing a high precision KB of locations and their subjective and factual attributes
  • We evaluate the trained models via two other methods. We compute their F-scores in attribute prediction with respect to a set of gold labels
  • While our experiments are focused on the KB of locations and attributes that supports Google Maps, our proposed framework is useful for constructing KBs with tunable precision from unlabeled side information and noisy categorical observations collected via crowdsourcing
Methods
  • The authors use the Adagrad [10] optimizer with learning rate of 0.1 and a batch size of 256.
  • The authors compare the 3 architectures (Section 5) with two additional empirical baselines.
  • The authors compare the models learned via the 3 architectures.
  • The authors compare the models to two empirical baselines.
  • One baseline (Empirical) predicts a “1” when the observed yes rate for a pair is greater than μmin.
  • The second, more precise baseline (Empirical-P) only makes predictions for pairs with at least 3 observed votes.
  • The authors show the performance of the IAV raw model operating in “high-recall mode” (IAV-HR), meaning that a “1” is predicted when μla > 0.66 and no additional filtering is performed
Results
  • The authors evaluate the trained models via two other methods.
  • The authors compute their F-scores in attribute prediction with respect to a set of gold labels.
  • The authors measure the F-score of the attribute predictor s(·, ·) with respect to G.
  • The authors report the F-score of the predictor with respect to both the prior and posterior parameters (Section 3).
  • All neural models achieve between 6%-9% better posterior precision (Section 7.2) than the Empirical baseline.
  • Under the 5% false positive rate, the IAV model achieves the highest F-score of the neural models
Conclusion
  • The authors study constructing a high precision KB of locations and their subjective and factual attributes.
  • The authors probabilistically model the latent yes rate of each location-attribute pair, rather than modeling each pair as either True or False.
  • Model confidence is explicitly represented and used to control the KB’s false positive rate.
  • While the experiments are focused on the KB of locations and attributes that supports Google Maps, the proposed framework is useful for constructing KBs with tunable precision from unlabeled side information and noisy categorical observations collected via crowdsourcing
Tables
  • Table1: A sample of factual and subjective attributes
Download tables as Excel
Related work
  • The literature on crowdsourcing for data collection and subsequent model training is vast. Most approaches collect multiple redundant labelings for a set of tasks from a handful of crowd workers and then infer the true task labels. Even in cases where the tasks are subjective, the true labels are considered to correspond to the majority opinion [21]. Many of these methods learn latent variable models of user expertise and task difficulty; the learned models can be used for inferring the task labels [26, 33]. Some work models both worker reputation and each item’s label as a real-valued random variable (in [0, 1]) with a beta prior [9]. Like we do, other work develops beta-binomial models of the observed labels [6]. Unlike the prior art, we do not explicitly model the crowd workers. This is beneficial because it does not require collecting a minimum number of labels per worker and also protects worker anonymity. Whereas some previous work employs expectation-maximization [34], variational inference [19], Markov Chain Monte Carlo, or variants of belief propagation [16], we estimate parameters via back-propagation in neural networks. Some studies develop intelligent routing of tasks to workers based on task difficulty and user ability [15, 16]. In our work, questions are routed to geographically relevant users.
Reference
  • Martín Abadi et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.
    Google ScholarFindings
  • A. Almahairi, K. Kastner, K. Cho, and A. Courville. 2015. Learning distributed representations from reviews for collaborative filtering. In Conference on Recommender Systems.
    Google ScholarLocate open access versionFindings
  • T. Bansal, D. Belanger, and A. McCallum. 2016. Ask the GRU: Multi-task Learning for Deep Text Recommendations. In Conference on Recommender Systems.
    Google ScholarFindings
  • J. Bennett, S. Lanning, et al. 2007. The netflix prize. In Knowledge Discovery in Databases Cup and Workshop. New York, NY, USA.
    Google ScholarFindings
  • A. Carlson et al. 2010. Toward an Architecture for Never-Ending Language Learning.. In Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • B. Carpenter. 2008. Multilevel bayesian models of categorical data annotation. Unpublished manuscript (2008).
    Google ScholarFindings
  • R. Caruana. 1998. Multitask learning. In Learning to learn.
    Google ScholarFindings
  • H. Cheng et al. 2016.
    Google ScholarLocate open access versionFindings
  • L. de Alfaro, V. Polychronopoulos, and M. Shavlovsky. 2015. Reliable aggregation of boolean crowdsourced tasks. In Human Computation and Crowdsourcing.
    Google ScholarFindings
  • J. Duchi, E. Hazan, and Y. Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (2011).
    Google ScholarLocate open access versionFindings
  • A. M. Elkahky, Y. Song, and X. He. 2015. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In International Conference on World Wide Web.
    Google ScholarFindings
  • M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. 2011. CrowdDB: answering queries with crowdsourcing. In International Conference on Management of data.
    Google ScholarLocate open access versionFindings
  • P. Gopalan, J. M. Hofman, and D. M. Blei. 20Scalable recommendation with poisson factorization. arXiv:1311.1704 (2013).
    Findings
  • P. K. Gopalan, L. Charlin, and D. Blei. 20Content-based recommendations with poisson factorization. In Neural Information Processing Systems.
    Google ScholarLocate open access versionFindings
  • C.J. Ho, S. Jabbari, and J. W. Vaughan. 2013. Adaptive task assignment for crowdsourced classification. In International Conference on Machine Learning.
    Google ScholarLocate open access versionFindings
  • D. R. Karger, S. Oh, and D. Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Neural information processing systems.
    Google ScholarFindings
  • J. Lehmann et al. 2014. DBpedia - A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia. Semantic Web Journal (2014).
    Google ScholarLocate open access versionFindings
  • D. Liang, J. Altosaar, L. Charlin, and D. M. Blei. 2016. Factorization meets the item embedding: Regularizing matrix factorization with item co-occurrence. In Conference on Recommender Systems.
    Google ScholarLocate open access versionFindings
  • Q. Liu, J. Peng, and A. T. Ihler. 2012. Variational inference for crowdsourcing. In Neural information processing systems.
    Google ScholarFindings
  • A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. 2011. Crowdsourced databases: Query processing with people. In Conference on Innovative Data Systems Research.
    Google ScholarLocate open access versionFindings
  • R. Meng, H. Xin, L. Chen, and Y. Song. 2017. Subjective Knowledge Acquisition and Enrichment Powered By Crowdsourcing. arXiv:1705.05720 (2017).
    Findings
  • F. Niu, C. Zhang, C. Ré, and J. W. Shavlik. 2012. DeepDive: Web-scale Knowledgebase Construction using Statistical Learning and Inference. International Conference on Very Large Data search (2012).
    Google ScholarFindings
  • A. G. Parameswaran, H. Park, H. Garcia-Molina, N. Polyzotis, and J. Widom. 2012. Deco: declarative crowdsourcing. In International conference on Information and knowledge management.
    Google ScholarLocate open access versionFindings
  • S. Park, Y. Kim, and S. Choi. 2013. Hierarchical Bayesian Matrix Factorization with Side Information.. In International Joint Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • I. Porteous, A. Asuncion, and M. Welling. 2010. Bayesian matrix factorization with side information and dirichlet process mixtures. In Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. 2010. Learning from crowds. Journal of Machine Learning Research (2010).
    Google ScholarLocate open access versionFindings
  • S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Conference on uncertainty in artificial intelligence.
    Google ScholarLocate open access versionFindings
  • S. Riedel, L. Yao, A. McCallum, and B. M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. (2013).
    Google ScholarFindings
  • R. Salakhutdinov and A Mnih. 2007. Probabilistic Matrix Factorization.. In Neural
    Google ScholarLocate open access versionFindings
  • V. S. Sheng, F. Provost, and P. G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In International conference on Knowledge discovery and data mining.
    Google ScholarLocate open access versionFindings
  • F. M. Suchanek, G. Kasneci, and G. Weikum. 2007. Yago: a core of semantic knowledge. In International conference on World Wide Web.
    Google ScholarFindings
  • H. Wang, N. Wang, and D. Yeung. 2015. Collaborative deep learning for recommender systems. In International Conference on Knowledge Discovery and Data Mining.
    Google ScholarLocate open access versionFindings
  • P. Welinder, S. Branson, P. Perona, and S. J. Belongie. 2010. The multidimensional wisdom of crowds. In Neural information processing systems.
    Google ScholarFindings
  • J. Whitehill, T. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Neural information processing systems.
    Google ScholarFindings
  • L. Zheng, V. Noroozi, and P. S. Yu. 2017. Joint deep modeling of users and items using reviews for recommendation. In International Conference on Web Search and Data Mining.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科