AI helps you reading Science
AI generates interpretation videos
AI extracts and analyses the key points of the paper to generate videos automatically
AI parses the academic lineage of this thesis
AI extracts a summary of this paper
While our experiments are focused on the knowledge bases of locations and attributes that supports Google Maps, our proposed framework is useful for constructing KBs with tunable precision from unlabeled side information and noisy categorical observations collected via crowdsourc...
Constructing High Precision Knowledge Bases with Subjective and Factual Attributes.
KDD, (2019): 2050.0-2058.0
Knowledge bases (KBs) are the backbone of many ubiquitous applications and are thus required to exhibit high precision. However, for KBs that store subjective attributes of entities, e.g., whether a movie is kid friendly, simply estimating precision is complicated by the inherent ambiguity in measuring subjective phenomena. In this work, ...More
PPT (Upload PPT)
- Structured knowledge repositories–known as knowledge bases (KBs)–are the backbone of many high-impact applications and services.
- Consider organizing a lunch meeting and issuing a KB query for cafes that are good for groups.
- Since most KBs are built using noisy automated methods, special consideration must be paid.
- Previous work echos this concern: in addition to employing trained automated components for data collection and prediction of missing values, systems that build KBs often turn to humans–largely considered to be more precise than the automated methods–for
- Structured knowledge repositories–known as knowledge bases (KBs)–are the backbone of many high-impact applications and services
- For example: the Netflix1 movie recommendation engine relies on a KB of user-movie-rating triples, Google Maps2 is built atop a KB of geographic points of interest and PubMed3 offers a handful of tools that operate on its citation KB of biomedical research
- While we study the location-attribute setting, our yes rate modeling framework can be applied in many instances of hybrid KB construction that rely on collecting categorical observations via crowdsourcing
- We note that for the subjective attributes, our evaluation scheme produces a conservative estimate of model quality, which, we argue, is better than a non-conservative estimate given the importance of mitigating false positives
- We study constructing a high precision KB of locations and their subjective and factual attributes
- We evaluate the trained models via two other methods. We compute their F-scores in attribute prediction with respect to a set of gold labels
- While our experiments are focused on the KB of locations and attributes that supports Google Maps, our proposed framework is useful for constructing KBs with tunable precision from unlabeled side information and noisy categorical observations collected via crowdsourcing
- The authors use the Adagrad  optimizer with learning rate of 0.1 and a batch size of 256.
- The authors compare the 3 architectures (Section 5) with two additional empirical baselines.
- The authors compare the models learned via the 3 architectures.
- The authors compare the models to two empirical baselines.
- One baseline (Empirical) predicts a “1” when the observed yes rate for a pair is greater than μmin.
- The second, more precise baseline (Empirical-P) only makes predictions for pairs with at least 3 observed votes.
- The authors show the performance of the IAV raw model operating in “high-recall mode” (IAV-HR), meaning that a “1” is predicted when μla > 0.66 and no additional filtering is performed
- The authors evaluate the trained models via two other methods.
- The authors compute their F-scores in attribute prediction with respect to a set of gold labels.
- The authors measure the F-score of the attribute predictor s(·, ·) with respect to G.
- The authors report the F-score of the predictor with respect to both the prior and posterior parameters (Section 3).
- All neural models achieve between 6%-9% better posterior precision (Section 7.2) than the Empirical baseline.
- Under the 5% false positive rate, the IAV model achieves the highest F-score of the neural models
- The authors study constructing a high precision KB of locations and their subjective and factual attributes.
- The authors probabilistically model the latent yes rate of each location-attribute pair, rather than modeling each pair as either True or False.
- Model confidence is explicitly represented and used to control the KB’s false positive rate.
- While the experiments are focused on the KB of locations and attributes that supports Google Maps, the proposed framework is useful for constructing KBs with tunable precision from unlabeled side information and noisy categorical observations collected via crowdsourcing
- Table1: A sample of factual and subjective attributes
- The literature on crowdsourcing for data collection and subsequent model training is vast. Most approaches collect multiple redundant labelings for a set of tasks from a handful of crowd workers and then infer the true task labels. Even in cases where the tasks are subjective, the true labels are considered to correspond to the majority opinion . Many of these methods learn latent variable models of user expertise and task difficulty; the learned models can be used for inferring the task labels [26, 33]. Some work models both worker reputation and each item’s label as a real-valued random variable (in [0, 1]) with a beta prior . Like we do, other work develops beta-binomial models of the observed labels . Unlike the prior art, we do not explicitly model the crowd workers. This is beneficial because it does not require collecting a minimum number of labels per worker and also protects worker anonymity. Whereas some previous work employs expectation-maximization , variational inference , Markov Chain Monte Carlo, or variants of belief propagation , we estimate parameters via back-propagation in neural networks. Some studies develop intelligent routing of tasks to workers based on task difficulty and user ability [15, 16]. In our work, questions are routed to geographically relevant users.
- Martín Abadi et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.
- A. Almahairi, K. Kastner, K. Cho, and A. Courville. 2015. Learning distributed representations from reviews for collaborative filtering. In Conference on Recommender Systems.
- T. Bansal, D. Belanger, and A. McCallum. 2016. Ask the GRU: Multi-task Learning for Deep Text Recommendations. In Conference on Recommender Systems.
- J. Bennett, S. Lanning, et al. 2007. The netflix prize. In Knowledge Discovery in Databases Cup and Workshop. New York, NY, USA.
- A. Carlson et al. 2010. Toward an Architecture for Never-Ending Language Learning.. In Conference on Artificial Intelligence.
- B. Carpenter. 2008. Multilevel bayesian models of categorical data annotation. Unpublished manuscript (2008).
- R. Caruana. 1998. Multitask learning. In Learning to learn.
- H. Cheng et al. 2016.
- L. de Alfaro, V. Polychronopoulos, and M. Shavlovsky. 2015. Reliable aggregation of boolean crowdsourced tasks. In Human Computation and Crowdsourcing.
- J. Duchi, E. Hazan, and Y. Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (2011).
- A. M. Elkahky, Y. Song, and X. He. 2015. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In International Conference on World Wide Web.
- M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. 2011. CrowdDB: answering queries with crowdsourcing. In International Conference on Management of data.
- P. Gopalan, J. M. Hofman, and D. M. Blei. 20Scalable recommendation with poisson factorization. arXiv:1311.1704 (2013).
- P. K. Gopalan, L. Charlin, and D. Blei. 20Content-based recommendations with poisson factorization. In Neural Information Processing Systems.
- C.J. Ho, S. Jabbari, and J. W. Vaughan. 2013. Adaptive task assignment for crowdsourced classification. In International Conference on Machine Learning.
- D. R. Karger, S. Oh, and D. Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Neural information processing systems.
- J. Lehmann et al. 2014. DBpedia - A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia. Semantic Web Journal (2014).
- D. Liang, J. Altosaar, L. Charlin, and D. M. Blei. 2016. Factorization meets the item embedding: Regularizing matrix factorization with item co-occurrence. In Conference on Recommender Systems.
- Q. Liu, J. Peng, and A. T. Ihler. 2012. Variational inference for crowdsourcing. In Neural information processing systems.
- A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. 2011. Crowdsourced databases: Query processing with people. In Conference on Innovative Data Systems Research.
- R. Meng, H. Xin, L. Chen, and Y. Song. 2017. Subjective Knowledge Acquisition and Enrichment Powered By Crowdsourcing. arXiv:1705.05720 (2017).
- F. Niu, C. Zhang, C. Ré, and J. W. Shavlik. 2012. DeepDive: Web-scale Knowledgebase Construction using Statistical Learning and Inference. International Conference on Very Large Data search (2012).
- A. G. Parameswaran, H. Park, H. Garcia-Molina, N. Polyzotis, and J. Widom. 2012. Deco: declarative crowdsourcing. In International conference on Information and knowledge management.
- S. Park, Y. Kim, and S. Choi. 2013. Hierarchical Bayesian Matrix Factorization with Side Information.. In International Joint Conference on Artificial Intelligence.
- I. Porteous, A. Asuncion, and M. Welling. 2010. Bayesian matrix factorization with side information and dirichlet process mixtures. In Conference on Artificial Intelligence.
- V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. 2010. Learning from crowds. Journal of Machine Learning Research (2010).
- S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Conference on uncertainty in artificial intelligence.
- S. Riedel, L. Yao, A. McCallum, and B. M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. (2013).
- R. Salakhutdinov and A Mnih. 2007. Probabilistic Matrix Factorization.. In Neural
- V. S. Sheng, F. Provost, and P. G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In International conference on Knowledge discovery and data mining.
- F. M. Suchanek, G. Kasneci, and G. Weikum. 2007. Yago: a core of semantic knowledge. In International conference on World Wide Web.
- H. Wang, N. Wang, and D. Yeung. 2015. Collaborative deep learning for recommender systems. In International Conference on Knowledge Discovery and Data Mining.
- P. Welinder, S. Branson, P. Perona, and S. J. Belongie. 2010. The multidimensional wisdom of crowds. In Neural information processing systems.
- J. Whitehill, T. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Neural information processing systems.
- L. Zheng, V. Noroozi, and P. S. Yu. 2017. Joint deep modeling of users and items using reviews for recommendation. In International Conference on Web Search and Data Mining.