The Information Age has made publishing, distributing and collecting information easier, resulting in the exponential growth of information available to us. Databases were once ledgers written by hand by a single person; today they can be vast stores of data agglomerated from a myriad of disparate sources. The mass media, formerly limited to newspapers and television programs held to strict journalistic standards, has expanded to include collaborative content such as blogs, wikis, and message boards. Documents covering nearly every topic abound on the Internet, but the authors are often anonymous and the accuracy uncertain. To cope with this new abundance, we employ information retrieval to suggest documents, and information extraction to tell us what they say, but how can we determine what we should actually believe? Not all information sources are equally trustworthy, and simply accepting the majority view often leads to errors: a Google search for “water runs downhill” returns 17.5K documents, while “water runs uphill” yields 116K. When we consider a collection of data with various authorship, we may view it as a set of information sources each making one or more claims. Sources often make claims that are contradictory (“Shakespeare was born on April 26th, 1564” and “Shakespeare was born on April 23rd, 1564”) and, even in the absence of contradiction, we have no guarantee that the sole presented claim is true. How, then, can we know which claims to believe, and which sources to trust? The typical approach is simple: take a vote and choose the claim made by the largest number of sources. However, this implicitly (and implausibly) assumes that all sources are equally trustworthy and, moreover, ignores the wealth of other claims being made by both these and other sources that could inform our belief in the particular claim at hand. For example, if we can ascertain that John’s other claims of birthdays for historic figures were correct, his claim about Shakespeare should (ceteris paribus) carry more weight. A diverse class of algorithms collectively known as fact-finders does just this, using the full network of sources and claims to jointly estimate both the trustworthiness of the sources and the believability of the claims. This is useful not just in judging the assertions made by authors in articles, but also in areas such as sensor networks and crowdsourcing. Crowdsourcing of information–where information is polled from a wider population–can be done via direct voting, the most famous being reCaptcha (Von Ahn, Maurer, McMillen, Abraham, & Blum, 2008), which uses humans to solve difficult OCR problems as a Turing test and accepts the text for a candidate word image once it has accrued enough votes. Similarly, the ESP Game (Von Ahn & Dabbish, 2004) obtains image labelings, presenting the task as a game. In both cases, the annotators are presented with examples

Decision makers and citizens are heavily influenced these days by information they obtain from online resources – from news portals, online encyclopedias, blogs and forums, product websites and reviews, etc. However, the lack of control over what gets published online can often lead to dissemination of unreliable and misleading information. In such a scenario, how can one verify if a claim is true or determine which sources are trustworthy? In this tutorial, the instructors systematically consider various approaches that address this problem. This tutorial is aimed at AI researchers who are interested in future research on trustworthiness of information. The goal of the tutorial is to present the audience with an exhaustive survey of relevant research on this topic and to outline future directions of research that may interest the AI community. No specific background knowledge is assumed of the audience.

A frequent problem when dealing with data gathered from multiple sources on the web (ranging from booksellers to Wikipedia pages to stock analyst predictions) is that these sources disagree, and we must decide which of their (often mutually exclusive) claims we should accept. Current state-of-the-art information credibility algorithms known as "fact-finders" are transitive voting systems with rules specifying how votes iteratively flow from sources to claims and then back to sources. While this is quite tractable and often effective, fact-finders also suffer from substantial limitations; in particular, a lack of transparency obfuscates their credibility decisions and makes them difficult to adapt and analyze: knowing the mechanics of how votes are calculated does not readily tell us what those votes mean, and finding, for example, that a source has a score of 6 is not informative. We introduce a new approach to information credibility, Latent Credibility Analysis (LCA), constructing strongly principled, probabilistic models where the truth of each claim is a latent variable and the credibility of a source is captured by a set of model parameters. This gives LCA models clear semantics and modularity that make extending them to capture additional observed and latent credibility factors straightforward. Experiments over four real-world datasets demonstrate that LCA models can outperform the best fact-finders in both unsupervised and semi-supervised settings.

When information sources are unreliable, information networks have been used in data mining literature to uncover facts from large numbers of complex relations between noisy variables. The approach relies on topology analysis of graphs, where nodes represent pieces of (unreliable) information and links represent abstract relations. Such topology analysis was often empirically shown to be quite powerful in extracting useful conclusions from large amounts of poor-quality information. However, no systematic analysis was proposed for quantifying the accuracy of such conclusions. In this paper, we present, for the first time, a Bayesian interpretation of the basic mechanism used in fact-finding from information networks. This interpretation leads to a direct quantification of the accuracy of conclusions obtained from information network analysis. Hence, we provide a general foundation for using information network analysis not only to heuristically extract likely facts, but also to quantify, in an analytically-founded manner, the probability that each fact or source is correct. Such probability constitutes a measure of quality of information (QoI). Hence, the paper presents a new foundation for QoI analysis in information networks, that is of great value in deriving information from unreliable sources. The framework is applied to a representative fact-finding problem, and is validated by extensive simulation where analysis shows significant improvement over past work and great correspondence with ground truth.

The Information Age has created an increasing abundance of data and has, thanks to the rise of the Internet, made that knowledge instantly available to humans and computers alike. This is not without caveats, however, as though we may read a document, ask an expert, or locate a fact nearly effortlessly, we lack a ready means to determine whether we should actually believe them. We seek to address this problem with a computational trust system capable of substituting for the user's informed, subjective judgement, with the understanding that truth is not objective and instead depends upon one's prior knowledge and beliefs, a philosophical point with deep practical implications. First, however, we must consider the even more basic question of how the trustworthiness of an information source can be expressed: measuring the trustworthiness of a person, document, or publisher as the mere percentage of true claims it makes can be extraordinarily misleading at worst, and uninformative at best. Instead of providing simple accuracy, we instead provide a comprehensive set of trust metrics, calculating the source's truthfulness, completeness, and bias, providing the user with our trust judgement in a way that is both understandable and actionable. We then consider the trust algorithm itself, starting with the baseline of determining the truth by taking a simple vote that assumes all information sources are equally trustworthy, and quickly move on to fact-finders , iterative algorithms capable of estimating the trustworthiness of the source in addition to the believability of the claims, and proceed to incorporate increasing amounts of information and declarative prior knowledge into the fact-finder's trust decision via the Generalized and Constrained Fact-Finding frameworks while still maintaining the relative simplicity and tractability of standard fact-finders.

This demonstration presents Apollo, a new sensor information processing tool for uncovering likely facts in noisy participatory sensing data1. Participatory sensing, where users proactively document and share their observations, has received significant attention in recent years as a paradigm for crowd-sourcing observation tasks. However, it poses interesting challenges in assessing confidence in the information received. By borrowing clustering and ranking tools from data mining literature, we show how to group data into sets (or claims), corroborating specific events or observations, then iteratively assess both claim and source credibility, ultimately leading to a ranking of described claims by their like-lihoold of occurrence. Apollo belongs to a category of tools called fact-finders. It is the first fact-finder designed and implemented specifically for participatory sensing. Apollo uses Twitter as the underlying engine for sharing participatory sensing data. Twitter is widely popular, can be interfaced to cell-phones that share sensor data, and comes with a powerful search API, as well as a publish-subscribe mechanism. We evaluate it using a participatory sensing application that collects and posts noisy vehicular traffic data on Twitter, as well as a set of 60,000 (human) tweets collected during the Haiti tsunami and a set of 500,000 tweets collected about Cairo during its recent unrest. Viewers of the demonstration will interact with Apollo for various fact-finding tasks.

Once information retrieval has located a document, and information extraction has provided its contents, how do we know whether we should actually believe it? Fact-finders are a state-of-the-art class of algorithms that operate in a manner analogous to Kleinberg's Hubs and Authorities, iteratively computing the trustworthiness of an information source as a function of the believability of the claims it makes, and the believability of a claim as a function of the trustworthiness of those sources asserting it. However, as fact-finders consider only "who claims what", they ignore a great deal of relevant background and contextual information. We present a framework for "lifting" (generalizing) the fact-finding process, allowing us to elegantly incorporate knowledge such as the confidence of the information extractor and the attributes of the information sources. Experiments demonstrate that leveraging this information significantly improves performance over existing, "unlifted" fact-finding algorithms.

Information retrieval may suggest a document, and information extraction may tell us what it says, but which information sources do we trust and which assertions do we believe when different authors make conflicting claims? Trust algorithms known as fact-finders attempt to answer these questions, but consider only which source makes which claim, ignoring a wealth of background knowledge and contextual detail such as the uncertainty in the information extraction of claims from documents, attributes of the sources, the degree of similarity among claims, and the degree of certainty expressed by the sources. We introduce a new, generalized fact-finding framework able to incorporate this additional information into the fact-finding process. Experiments using several state-of-theart fact-finding algorithms demonstrate that generalized fact-finders achieve significantly better performance than their original variants on both semi-synthetic and real-world problems.

Existing computational trust systems analyze information networks to determine the “trustworthiness” of the nodes, but the scalar values they produce are both opaque and semantically variable, and knowing only that the trustworthiness of a website is “27” is not helpful to the user. Moreover, the simplistic means by which they are typically calculated can yield misleading results, sometimes dramatically so. We present a new, standardized set of trust metrics that instead compute the trustworthiness of an information source as a triple of truthfulness, completeness, and bias scores, and argue that these must be calculated relative to the user to be meaningful. We then explore these new metrics with a user study.

Although much work in NLP has focused on simply determining what a document means, we also must know whether or not to believe it. Fact-finding algorithms attempt to identify the "truth" among competing claims in a corpus, but fail to take advantage of the user's prior knowledge and presume that truth itself is universal and objective rather than subjective. We introduce a framework for incorporating prior knowledge into any fact-finding algorithm, expressing both general "common-sense" reasoning and specific facts already known to the user as first-order logic and translating this into a tractable linear program. As our results show, this approach scales well to even large problems, both reducing error and allowing the system to determine truth respective to the user rather than the majority. Additionally, we introduce three new fact-finding algorithms capable of outperforming existing fact-finders in many of our experiments.

We introduce a new probabilistic model for transliteration that performs significantly better than previous approaches, is language-agnostic, requiring no knowledge of the source or target languages, and is capable of both generation (creating the most likely transliteration of a source word) and discovery (selecting the most likely transliteration from a list of candidate words). Our experimental results demonstrate improved accuracy over the existing state-of-the-art by more than 10% in Chinese, Hebrew and Russian. While past work has commonly made use of fixed-size n-gram features along with more traditional models such as HMM or Perceptron, we utilize an intuitive notion of "productions", where each source word can be segmented into a series of contiguous, non-overlapping substrings of any size, each of which independently transliterates to a substring in the target language with a given probability. To learn these parameters, we employ Expectation-Maximization (EM), with the alignment between substrings in the source and target word training pairs as our latent data. Despite the size of the parameter space and the 2 (|w|-1) possible segmentations to consider for each word, by using dynamic programming each iteration of EM takes O(m^6 * n) time, where m is the length of the longest word in the data and n is the number of word pairs, and is very fast in practice. Furthermore, discovering transliterations takes only O(m^4 * w) time, where w is the number of candidate words to choose from, and generating a transliteration takes O(m 2 * k 2 ) time, where k is a pruning constant (we used a value of 100). Additionally, we are able to obtain training examples in an unsupervised fashion from Wikipedia by using a relatively simple algorithm to filter potential word pairs.

Much of the information on the Web is found in articles from online news outlets, magazines, encyclopedias, review collections, and other sources. However, extracting this content from the original HTML document is complicated by the large amount of less informative and typically unrelated material such as navigation menus, forms, user comments, and ads. Existing approaches tend to be either brittle and demand significant expert knowledge and time (manual or tool-assisted generation of rules or code), necessitate labeled examples for every different page structure to be processed (wrapper induction), require relatively uniform layout (template detection), or, as with Visual Page Segmentation (VIPS), are computationally expensive. We introduce maximum subsequence segmentation, a method of global optimization over token-level local classifiers, and apply it to the domain of news websites. Training examples are easy to obtain, both learning and prediction are linear time, and results are excellent (our semi-supervised algorithm yields an overall F1-score of 97.947%), surpassing even those produced by VIPS with a hypothetical perfect block-selection heuristic. We also evaluate against the recent CleanEval shared task with surprisingly good cross-task performance cleaning general web pages, exceeding the top "text-only" score (based on Levenshtein distance), 87.8% versus 84.1%.

Wikipedia, the popular online encyclopedia, has in just six years grown from an adjunct to the now-defunct Nupedia to over 31 million pages and 429 million revisions in 256 languages and spawned sister projects such as Wiktionary and Wikisource. Available under the GNU Free Documentation License, it is an extraordinarily large corpus with broad scope and constant updates. Its articles are largely consistent in structure and organized into category hierarchies. However, the wiki method of collaborative editing creates challenges that must be addressed. Wikipedia’s accuracy is frequently questioned, and systemic bias means that quality and coverage are uneven, while even the variety of English dialects juxtaposed can sabotage the unwary with differences in semantics, diction and spelling. This paper examines Wikipedia from a research perspective, providing basic background knowledge and an understanding of its strengths and weaknesses. We also solve a technical challenge posed by the enormity of text (1.04TB for the English version) made available with a simple, easily-implemented dictionary compression algorithm that permits time-efficient random access to the data with a twenty-eight-fold reduction in size.