The last decade has witnessed the growth of information retrieval from a boutique discipline in information and library science to an everyday experience for billions of people around the world. This revolution has been driven in large measure by the Internet, with vendors focused on search and navigation of Web resources and Web content management. Simultaneously, enterprises have invested in networking all of their information together to the point where it is increasingly possible for employees to have a single window into the enterprise. Although these employees seek Web-like experiences in the enterprise, the Internet and enterprise domains differ fundamentally in the nature of the content, user behavior, and economic motivations.

In the course of researching a subject, it is often necessary to submit the same search request to multiple heterogeneous information sources in order to (a) aggregate as much information as possible, and (b) integrate different aspects of the subject into a coherent report. While it is clear that there is value in providing a federated search solution to make dealing with multiple sources less time-consuming, not all organizations aggregate from the same sources, and once the information has been retrieved, not all organizations want them to be integrated in the same way.The Verity Federated Infrastructure addresses this problem by providing a flexible framework for adding new sources and customizing the way in which results are integrated, post-processed and presented. A new source is made available by writing a Java module called a worker that abides by the search interface of the source. Sources can range from simple information feeds to more complex applications, e.g., CRM systems, relational databases, etc. Workers also perform post-processing on the results returned by other workers, e.g., to provide uniform scores for results from different sources, filtering, etc. This post-processing enables different results to be integrated into a coherent report. Post-processing is triggered by events that propagate between workers and is done asynchronously in the background while results are being viewed. This ability to do background post-processing allows execution of time-consuming operations that provide substantial value without adversely affecting user experience. Finally, search results are returned and viewed incrementally, which enables searching of peer-to-peer networks via peer workers that we have developed.

This paper presents several paradigms by which users of Verity business portals (from within as well as from outside an enterprise) discover and navigate relevant semistructured data in corpora with millions of documents. These paradigms include (i) combining free-text and structured queries, (ii) automatic classification, and (iii) personalization.