Although keyword-based queries are now a familiar part of any user’s experience with the World Wide Web, they are of limited direct applicability to the vast and growing quantity of multimedia information becoming available in materials such as broadcast news, video teleconferences, reconnaissance data, and audio-visual recordings of corporate meetings and classroom lectures. Content-based indexing, archiving and retrieval would facilitate access to large databases of such materials. For example, in the broadcast news domain, content-based archiving is particularly useful. Archiving can be done by exploiting the speech contained in the audio track, the images contained in the video track, and the text in video overlays. One application is to filter down huge volumes of raw news footage to create the nicely packaged news broadcasts that we watch on television. Another use is to create a news-on-demand system for viewing news more efficiently. We can create a database of news broadcasts annotated for later retrieval of news clips of interest. The query “Tell me about the recent elections in Bosnia” would bring up news clips related to the elections. MAESTRO (Multimedia Annotation and Enhancement via a Synergy of Technologies and Reviewing Operators) is a research and demonstration system developed at SRI International for exploring the contribution of a variety of analysis technologies — for example, speech recognition, image understanding, and optical character recognition — to the indexing and retrieval of multimedia. Informedia [1] and Broadcast News Navigator [2] are similar projects that use these technologies for archiving and retrieval. The main goal of the MAESTRO project is to discover, implement, and evaluate various combinations of these technologies to achieve analysis performance that surpasses the sum of the parts. For example, British Prime Minister Tony Blair can be identified in the news by his voice, his appearance, captions, and other cues. A combination of these cues should provide more reliable identification of the Prime Minister than using any of the cues on their own. MAESTRO is a highly multidisciplinary effort, involving contributions from three laboratories across two divisions at SRI. Each of these SRI technologies is described in more detail below. The integrating architecture makes it easy to combine these in different ways, and to incorporate new analysis technologies developed by our team or by others.

Analysts face a daunting task: they must accurately analyze, categorize, and assimilate a large body of information from a variety of sources and for a variety of domains of interest. The complexity of the task necessitates a variety of information access and extraction tools which technology up to this point has not been able to provide. SRI's TIPSTER Phase III project has focused on two major obstacles to the development of such tools: inadequate degrees of accuracy and portability. We begin by providing an overview of SRI's information extraction (IE) system, FASTUS, and then describe our efforts in these two areas in turn. We then conclude with some thoughts concerning future directions.

FASTUS is a system for extracting information from natural language text for entry into a database and for other applications. It works essentially as a cascaded, nondeterministic finite-state automaton. There are five stages in the operation of FASTUS. In Stage 1, names and other fixed form expressions are recognized. In Stage 2, basic noun groups, verb groups, and prepositions and some other particles are recognized. In Stage 3, certain complex noun groups and verb groups are constructed. Patterns for events of interest are identified in Stage 4 and corresponding ``event structures'' are built. In Stage 5, distinct event structures that describe the same event are identified and merged, and these are used in generating database entries. This decomposition of language processing enables the system to do exactly the right amount of domain-independent syntax, so that domain-dependent semantic and pragmatic processing can be applied to the right larger-scale structures. FASTUS is very efficient and effective, and has been used successfully in a number of applications.

During the past year, signiicant improvements have been made in the natural-language processing technology used in the SRI ATIS spoken-language understanding system. The principal developments have been (1) the incorporation of information from the natural-language grammar and lexicon into a statistical language model that is used in both recognition and understanding, (2) implementation of a robust interpretation component that constructs queries out of grammatical fragments when an utterance cannot be analyzed as a single phrase or utterance, and (3) a new context mechanism for air travel planning that constructs an explicit model of the user's intended itinerary.

FASTUS is a (slightly permuted) acronym for Finite State Automaton Text Understanding System. It is a system for extracting information from free text in English (Japanese is under development), for entry into a database, and potentially for other applications. It works essentially as a set of cascaded, nondeterministic finite state automata.

: FASTUS is a system for extracting information from free text in English, and potentially other languages as well, for entry into a database, and potentially for other applications. It works essentially as a cascaded, nondeterministic finite state automaton. There are four steps in the operation of FASTUS. In Step (1) sentences are scanned for certain trigger words to determine whether further processing should be done. In Step (2) noun groups, verb groups, and prepositions and some other particles are recognized. The input to Step (3) is the sequence of phrases recognized in Step (2); patterns of interest are identified in Step (3) and corresponding incident structures are built up. In Step (4) incident structures that derive from the same incident are identified and merged, and these are used in generating database entries. FASTUS is an order of magnitude faster than any comparable system; it can process a news report in an average of less than eleven seconds. This translates directly into fast development time. In the three and a half weeks between its first use and the MUC-4 evaluation in May 1992, we were able to build up its domain knowledge to a point where it was among the leaders in the evaluation.

Tables are a pervasive problem in the real-world corpora that are the focus of information extraction applications. Moreover, they constitute a problem of significant linguistic interest. In this paper we present a general method for recognizing and interpreting tables in text, and describe its implementation in a particular application.