We present a novel method for detecting text lines in historical handwritten and printed document images. Our hybrid technique begins by leveraging deep neural networks to perform multi-class pixel-wise prediction. The predictor not only discovers text and graphics pixels in the document, but it is also designed to automatically adhere contiguous regions from the same text line while also predicting buffers that prevent disassociated text lines from merging. The system breaks neural "ties" through dynamic programming. To the best of our knowledge, our system is the first neural system to predict the entire perimeters of full text lines. Also, to aid in scaling and full-scope awareness, the network during training is initially given small regions of the image to study and then expands its scope to full images as training continues. Our goal for line segmentation is to enable automatic transcription on huge heterogenous collections of historical images, so we use transcription accuracy as our metric. We document and leverage our state-of-the-art transcription system as an evaluation harness for scoring our segmenter along with various other competitor segmenters. We then show the effectiveness of our system as it relates to other systems by comparing it to both known data sets (IAM) and to three 50K-word "in the wild" test sets consisting of (a) US handwritten wills and deeds, (b) US historical newsprint images, and (c) Spanish Church and Government records.

Before one can begin applying automatic transcription processes to a document image, a line segmentation algorithm usually must be applied first in order to identify the individual text lines upon which recognition will be performed. Most line segmentation algorithms use standard image processing techniques and/or statistics to identify inking activity in the image. Unfortunately, these algorithms have no awareness of inking that is intentional (such as that written by an author) versus that which is merely background, darkness of copy, noise, etc. Moreover, the algorithms themselves cannot always tell when lines have been overor under-segmented. Neural networks can be taught to learn such distinctions. We have trained a neural process which can detect image line phenomena and can be used to improve automatic line segmentation. To the best of our knowledge, no other researchers have demonstrated similar processes. We observe definite accuracy gains using our neurally-enhanced line segmenter over a previous high quality line segmenter, and we believe such improvements will apply when using other line segmentation algorithms. 1. Background on Line Segmentation When transcription of document images is a user’s goal, it is almost always a requirement to first transform the document into the sequence of its textual lines so that subsequent processing can then proceed on a line-by-line basis. The process of transforming a document into its constituent lines is referred to as line segmentation. The vast majority of line segmentation techniques leverage either statistics or common image processing techniques which include projection-based methods, the finding of connected components, smearing approaches, clustering, and Hough transforms (see, for examples [1], [2]). Other techniques also exist which try to use more sophisticated approaches such as peak and trough detections and carving seams between them, hidden Markov model-like analyses, and graphical approaches (see [1], [3], [4]). These methods provide good or even excellent results depending on the image collection, especially when collections are small or homogeneous. Yet when the images in a collection number in the millions or billions and are quite diverse, as in a genealogical data set, typical line segmentation algorithms fail to account for many of the phenomena that will occur. This is largely because these systems are designed to identify contrastive areas of pixel darkness and/or pixel connectivity as opposed to trying to capture what we will refer to here as deliberate marks: markings on the page which were intentionally placed there by the document’s author. Let us consider some situations where line segmentation can be problematic for activity-based or connectivity-based algorithms: a) In the presence of faint copy where inking is weak, systems that rely on connectivity get confused because single lines of text can appear as disconnected. This can result in falsely detecting multiple lines. Activity detectors, on the other hand, may treat these areas as having insufficient information to even predict the appearance of some of the lines. b) Speckle noise can cause challenges for a connectivity-based algorithm since it may be treated as myriad regions of disconnected components and thus over-generate lines. c) If textual lines have significant overlap (where, say, the descenders from one line intersect frequently with ascenders from below), connectivity-based algorithm may falsely merge these lines together. Activity-based algorithms, on the other hand, can treat such regions of overlap as if they were additional lines since there may be sufficient pixel darkness to warrant it. d) When the image is copied from a book or on a black background resulting in the image having side regions with dark vertical streaks, the activity algorithms can end up over-generating lines if the streaks are of variable thicknesses. e) If the slope of the text lines drifts from one side of the page to the other, both kinds of algorithms can either over-generate lines or can produce hypothesized lines where the left half and right half of the lines actually are drawn from different textual lines. This list can be extended. Yet it suffices as an illustration that the base algorithms, absent a search for deliberate marks, are likely to have challenges with certain kinds of documents. In this paper, we demonstrate that deep neural networks can be built which can identify deliberate marks and which can be used to improve a system’s ability to properly find textual lines. We demonstrate through this effort that not only is the error rate of line segmentation reduced through our neural processes, but we actually train a handwriting recognition system on a very large corpus of US legal documents and show that the resultant line segmentation improves performance by 1.2% absolute (7.5% relative reduction in error). 2. DNNs to Count Lines Prior to the work described in this paper, we had created a line segmentation algorithm (which we will refer to here as PreDNN) which leverages many of the leading techniques mentioned in Section 1. The base components of our algorithm are described elsewhere (see [5]), but suffice it to say that it uses multi-swath projections to identify activity peaks, dynamic programming to carve seams between those activity peaks, and overlap of connected components to detect falsely merged lines. These processes were then extended using statistical analyses to find and remove the overgeneration of detected peaks and to trim out falsely carved seams. The resultant algorithm seems to have quite high on a huge collections of generic printed documents and has accuracy (perhaps about 85%-90%) on a collection of tens of thousands of handwritten document spanning four centuries. We expect that the results of the PreDNN system rival those of state-of-the-art and we have observed that they are quite usable. That said, leaving out up to 15% of a document’s content is undesirable. 2.1. Line Count Cells Deep neural networks provide a potential means to overcome some of these gaps. We reasoned that if we could build a neural network which, given a snippet of an image, could predict if the snippet contains no text lines, fractions of a line, multiple lines, or exactly one line, then such a network could be used to either completely perform line segmentation or to improve upon an existing line segmentation algorithm. To test this hypothesis, we assembled a mixed corpus consisting of printed Latin-script texts, Chinese print, and Latinscript handwritten documents. From these, we automatically generated a huge collection of image snippets of variable sizes whose edges were not necessarily straight. These snippets were then condensed down into small cells of size 30 pixels by 30 pixels. We tagged each of these 30x30 cells with one of seven different tags based on the number of lines observed in each cell. As shown in Figures 1a-g, these snippets fall into categories of: Fig1.a Fig1.b Fig1.c Fig1.d Fig1.e Fig1.f Fig1.g (a) no-text-lines, (b) single-text-line, (c) vertical-bar-only, (d) lessthan-one-text-line, (e) two fragment lines, (f) more-than-one-butfewer-than-two, and (g) two-plus-lines. The colors that are selected here are deliberate in that they will be used throughout this paper in colorized images representing the various classifications. Our collection of tagged cells currently consists of 70.5K different images of which 65,259 are used for training and 5,277 are used for testing. No specific effort was made to have a comparable number of each of the seven classes of cells, so the final distribution is reflective of what is observed in practice. Table 1 shows the actual distribution by category in both the training and testing sets: Table 1: Distribution of Counts of Tagged Cells CATEGORY # in TRAIN # in TEST no-text-lines 7330 625 single-text-line 12651 848 vertical-bar-only 703 89 less-than-one-text-line 16813 1098 two fragment lines 5027 407 more-than-1/fewer-than-2 8573 801 two-plus-lines 14162 1409 An interesting thing about tagging cells in the way specified is that the tags remain the same even if the cells are flipped with respect to the y-axis, or if they are rotated 180 degrees. That means that if we consider these four permutations, we can likewise multiply the size of our collection by four resulting in a training set with 280K elements and a test set with 21K. 2.2. Training a DNN for Line Counts With training cells available, we can now train a deep neural network (DNN) to try to predict the line count of future image cells. To train our DNN, we make use of Google’s open source Tensorflow [6] engine. Additionally, the Tensorflow developers created a “recipe” using convolutional neural networks (CNN) as applicable to the MNIST digit-recognition task which we have modified for our task. We use the same number of layers in our network as they have, but we use a kernel size of 3, and our first through third hidden layers have, respectively, 16, 32, and 216 nodes. Each of our layers are smaller than those of the recipe, but this is beneficial for recognition speed (which seems essential for our task). Our particular DNN topology yields an average tagging accuracy of 91%. We can get up to 2% better prediction accuracy by doing one of the following: (a) on a per cell basis, apply the DNN predictor to each of the four legal permutations of that cell and vote, or (b) create cells from overlapping rectangular regions and keep the result if the prediction for both regions agree or otherwise use the prediction of the overlap region as a tie breaker. Since computational cost is a factor, method (a) is somewhat less desirable because one must perform four times more computation for a 2% gain. On the other hand, method (b) can be performed for only about 10% more cost than just doing cell-by-cell evaluation because the rectangles take the place of the cells and the tie b

Due to the ability of modern obituaries to provide rich genealogical information for family members who have died within the bounds of “living memory,” family history organizations have recently begun to acquire and index obituaries in vast quantities. The indexing process for these documents is typically done using human labor. Yet we describe an effort by FamilySearch which leverages various kinds of machine learning, statistical analyses, and rule-based processing to automatically index such documents without human intervention at rates thousands of times faster than humans while still achieving high levels of accuracy.

We present a method for transcribing historical newsprint images. To begin, for training and evaluation, we created a corpus of human-generated transcripts for almost 38,000 image snippets which contains nearly five million words. This may be one of the largest corpora of transcribed historical newsprint ever created. Then, we developed our automatic transcription process by leveraging the pattern recognition and statistical components of the state-of-the-art speech recognition toolkit, Kaldi. Specifically, we modified its language model behavior and we replaced Kaldi's speech-to-features transformation components with our own image-to-features process. Our replacement components include the use of word partials; image rotation; line segmentation which extends state-of-the-art methods; and customized feature generation. We conduct two evaluations of our technology: (a) an evaluation based on random selections of newspaper snippets; and (b) a diachronic evaluation of newspaper snippets by time frame. We compare our results of these evaluations to those of the commercial engines ABBYY Fine Reader Version 12 and OmniPage 18, as well as to the freely available system, Tesseract. We demonstrate that our process typically yields accuracies which are comparable to or exceed the accuracies of these other engines.

Personal names are often key elements desired from the processing of historical documents. An understanding of name syntax can be very valuable in aiding automation and analysis. Yet current grammatical parsers classify personal names as merely noun phrases with proper nouns as constituents. To that end, we have created a Personal Name Treebank (PNTB) and associated Statistical Personal Name Parser (SPNP) which are designed to carefully analyze syntactic structure of personal names. The PNTB consists of almost 10 million instances of constituency-parsed personal names attached to genealogically-related contexts. These instances are drawn from almost 200 different countries across millenia. The SPNP leverages the PNTB to achieve 94.4% parse constituency accuracy on a huge held-out set of names. The PNTB and SPNP represent significant new resources which we intend to make available to the research community. To our knowledge, comparable resources have never previously been created.

Over the last few decades, significant strides have been made in handwriting recognition (HR), which is the automatic transcription of handwritten documents. HR often focuses on modern handwritten material, but in the electronic age, the volume of handwritten material is rapidly declining. However, we believe HR is on the verge of having major application to historical record collections. In recent years, archives and genealogical organizations have conducted huge campaigns to transcribe valuable historical record content with such transcription being largely done through human-intensive labor. HR has the potential of revolutionizing these transcription endeavors. To test the hypothesis that this technology is close to applicability, and to provide a testbed for reducing any accuracy gaps, we have developed an evaluation paradigm for historical record handwriting recognition. We created a huge test corpus consisting of four historical data collections of four differing genres and three languages. In this paper, we provide the details of these extensive resources which we intend to release to the research community for further study. Since several research organizations have already participated in this evaluation, we also show initial results and comparisons to human levels of performance.

The FamilySearch Indexing project has enabled hundreds of thousands of volunteers to transcribe billions of records, making it one of the largest crowdsourcing initiatives in the world. Assuring high quality transcriptions (i.e., indexes) with a reasonable amount of volunteer effort is essential to keep pace with the mounds of newly digitized documents. Using historical data, we show the relationship between prior experience and native language on transcriber agreement. We then present a field experiment comparing the effectiveness (accuracy) and efficiency (time) of two quality control mechanisms: (1) Arbitration -- the existing mechanism wherein two volunteers independently transcribe records and disagreements go to an arbitrator, and (2) Peer Review -- a mechanism wherein one volunteer's work is reviewed by another volunteer. Peer Review is significantly more efficient, though not as effective for certain fields as Arbitration. Design suggestions for FamilySearch Indexing and related crowdsourcing initiatives are provided.

In the last decade, significant, largely-governmental funding has been applied to the automatic transcription of handwritten documents. Uses for this kind of technology are somewhat limited given that the numbers of handwritten documents are on the decline. However, certain types of handwritten historical records can be crucial for genealogical research in that they identify key vital facts. In recent years, organizations like FamilySearch have exhausted huge efforts to identify, digitize, and transcribe these kinds of genealogically-rich records. Until now, such transcription has largely been done through massive crowd-sourced labor. We believe handwriting recognition technology is only a few years away from profitable application to genealogical documents. To test this hypothesis, we developed an evaluation paradigm for measuring handwriting recognition performance on four data collections of differing genres and languages. We invited research organizations to participate in the evaluation and compared performance to the outcome of human annotation. In this paper, we provide the details of this paradigm, including the guidelines, corpora and evaluation tools. Then we illustrate the exciting system results which suggest that the state-of-the-art is very close to providing real-world benefit to the automatic transcription of genealogically-rich documents.

In this paper, we illustrate the creation of a completely new Treebank which has significant application to the genealogical space and, to the best of our knowledge, has never been described before. We document the creation of a Personal Name Treebank (PNTB) which, though still a work in progress, already contains over 150,000 name structure classifications for people names derived from all the cultures, time frames, and writing scripts that are observed in our 800-million-name Common Pedigree at new.familysearch.org. The Common Pedigree includes names from various millennia, name from all countries of the world, and names rendered not only in Latin, but also in scripts such as Cyrillic and CJK. We describe the PNTB and its components, and we give a number of examples where this is particularly beneficial to genealogical search.

Personal names are the most critical elements for discovery and compilation of one’s heritage. However, historical and multilingual names are subject to many alterations and variations which make searching and matching of such names a great challenge. Consequently, we have created a name matching corpus and evaluation which capture most of these variations and provide a means whereby different name matching systems can be compared as to their effectiveness on matching these historical and cross-lingual names. It is our plan to make this corpus and evaluation available to the public in order to provide a means for wide-scale improvements in historical name matching. We here describe the formation of the corpus, the evaluation methodology and metrics. Lastly we show the performance of a number of a name matching systems and identify potential directions for future enhancements.

There are considerable differences in remuneration and environment between crowd-sourced workers and the traditional laboratory study participant. If crowd-sourced participants are to be used for information retrieval user studies, we need to know if and to what extent their behavior on information retrieval tasks differs from the accepted standard of laboratory participants. With both crowd-sourced and laboratory participants, we conducted an experiment to measure their relevance judging behavior. We found that while only 30% of the crowd-sourced workers qualified for inclusion in the final group of participants, 100% of the laboratory participants qualified. Both groups have similar true positive rates, but the crowd-sourced participants had a significantly higher false positive rate and judged documents nearly twice as fast as the laboratory participants.

Relation extraction has been a subject of significant study over the past decade. Most relation extractors have been developed by combining the training of complex computational systems on large volumes of annotations with extensive rule writing by language experts. Moreover, many relation extractors are reliant on other non-trivial NLP technologies which themselves are developed through significant human efforts, such as entity tagging, parsing, etc. Due to the high cost of creating and assembling the required resources, relation extractors have typically been developed for only high-resourced languages. In this paper, we describe a near-zero-cost methodology to build relation extractors for significantly distinct non-English languages using only freely available Wikipedia and other web documents, and some knowledge of English. We apply our methodology and build alma-mater, birthplace, father, occupation, and spouse relation extractors in Greek, Spanish, Russian, and Chinese. We conduct evaluations of induced relations at the file level which are the most refined we have seen in the literature.

For years, the LDS Church has allowed genealogical patrons to search for their ancestors amongst large collections of indexed records where each record has been treated as independent of all other records. Recently, the Church has begun to consider the potential for providing patrons with clustered results of records. These clusters would attempt to identify the same entity across multiple record collections. This clustering process, which we will refer to as “RecordMatch” is, in essence, a very large entity disambiguation task across partially-populated records. We describe and implement a rigorous evaluation for RecordMatch by taking advantage of the groundwork lain by comparable evaluations in the research community. We then apply this evaluation framework to the results of an existing genealogical record-merging system, GenMergeDB, and demonstrate the high-level of success of existing technologies for the purpose of producing RecordMatch-type clusters.

Previous content extraction evaluations have neglected to address problems which complicate the incorporation of extracted information into an existing knowledge base. Previous question answering evaluations have likewise avoided tasks such as explicit disambiguation of target entities and handling a fixed set of questions about entities without previous determination of possible answers. In 2009 NIST conducted a Knowledge Base Population track at its Text Analysis Conference to unite the content extraction and question answering communities and jointly explore some of these issues. This exciting new evaluation attracted 13 teams from 6 countries that submitted results in two tasks, Entity Linking and Slot Filling. This paper explains the motivation and design of the tasks, describes the language resources that were developed for this evaluation, offers comparisons to previous community evaluations, and briefly summarizes the performance obtained by systems. We also identify relevant issues pertaining to target selection, challenging queries, and performance measures.

The TCAR team developed multiple systems in just a matter of weeks for both participating in the TAC-KBP evaluation under the entity linking and the slot filling paradigms. TCAR's entity linking processes consisted of one that leveraged information retrieval and a separate that sought to appeal to extraction-based techniques. With regard to slot filling, since the TAC evaluation was touted as a task that would appeal to both the question answering and content extraction communities, we opted to build two systems where the first made use of our question answering system and the latter exploited relation-finding from a number of our content extraction systems. We here provide detailed descriptions of our systems and their performance at TAC-KBP.

Lexical mismatch is a problem that confounds automatic question answering systems. While existing lexical ontologies such as WordNet have been successfully used to match verbal synonyms (e.g., beat and defeat) and common nouns (tennis is-a sport), their coverage of proper nouns is less extensive. Question answering depends substantially on processing named entities, and thus it would be of significant benefit if lexical ontologies could be enhanced with additional hypernymic (i.e., is-a) relations that include proper nouns, such as Edward Teach is-a pirate. We demonstrate how a recently developed statistical approach to mining such relations can be tailored to identify named entity hyponyms, and how as a result, superior question answering performance can be obtained. We ranked candidate hyponyms on 75 categories of named entities and attained 53% mean average precision. On TREC QA data our method produces a 9% improvement in performance.

In this paper, we describe a system by which the multilingual characteristics of Wikipedia can be utilized to annotate a large corpus of text with Named Entity Recognition (NER) tags requiring minimal human intervention and no linguistic expertise. This process, though of value in languages for which resources exist, is particularly useful for less commonly taught languages. We show how the Wikipedia format can be used to identify possible named entities and discuss in detail the process by which we use the Category structure inherent to Wikipedia to determine the named entity type of a proposed entity. We further describe the methods by which English language data can be used to bootstrap the NER process in other languages. We demonstrate the system by using the generated corpus as training sets for a variant of BBN's Identifinder in French, Ukrainian, Spanish, Polish, Russian, and Portuguese, achieving overall F-scores as high as 84.7% on independent, human-annotated corpora, comparable to a system trained on up to 40,000 words of human-annotated newswire.

We approached our first participation in TREC with an interest in performing retrieval on the output of automatic speech-to-text (speech recognition) systems and a background in performing topic-labeling on such output. Our primary thrust, therefore, was to participate in the SDR track. In conformance with the rules, we also participated in the Ad Hoc text-retrieval task, to create a baseline for comparing our converted topic-labeling system with other approaches to IR and to assess the effect of speech-transcription errors. A second thrust was to explore rapid prototyping of an IR system, given the existing topic-labeling software.

The QACTIS system has been tested in previous years at the TREC Question Answering Evaluations. This paper describes new enhancements to the system specific to TREC-2006, including basic improvements and thresholding experiments, filtered and Internet-supported pseudo-relevance feedback for information retrieval, and emerging statistics-driven question-answering. For contrast, we also compare our TREC-2006 system performance to that of our top systems from TREC-2004 and TREC-2005 applied to this year’s data. Lastly, we analyze evaluator-declared unsupportedness of factoids and nugget decisions of “other”questions to understand major negative changes in performance for these categories over last year.