In this poster, we propose the framework of BizCQ, a system to apply Continual Queries [7][8] on Web-based content to manage information exchanges between two business partners. In this poster, we describe ways to leverage previous research in Web monitoring techniques applied to the everyday problem of managing change within a business environment, and focus on the difficulties of managing changes that are caused by external parties in business-to-business (B2B) information exchanges.

Information change monitoring services are becoming increasingly useful as more and more informationis published on the Web. A major research challengeis how to make the service scalable to serve millionsof monitoring requests. Such services usually use softtriggers to model users' monitoring requests. We havedeveloped an effective trigger grouping scheme to optimize the trigger processing. The main idea behind thisscheme is to reduce repeated computation by groupingmonitoring requests of similar structures together. Inthis paper, we evaluate our approach using both measurements on real systems and simulations. The studyshows significant performance gains using the triggergrouping approach. Moreover, the gains are criticallydependent on group size and group size distribution(e.g., Zipf). We also discuss the benefit, trade-off, andruntime characteristics of the proposed approach.

Information monitoring systems are publish-subscribe systems that continuously track information changes and notify users (or programs acting on behalf of humans) of relevant updates according to specified thresholds. Internet-scale information monitoring presents a number of new challenges. First, automated change detection is harder when sources are autonomous and updates are performed asynchronously. Second, information source heterogeneity makes the problem of modelling and representing changes harder than ever. Third, efficient and scalable mechanisms are needed to handle a large and growing number of users and thousands or even millions of monitoring triggers fired at multiple sources. In this dissertation, we model users' monitoring requests using continual queries (CQs) and present a suite of efficient and scalable solutions to large scale information monitoring over structured or semistructured data sources. A CQ is a standing query that monitors information sources for interesting events (triggers) and notifies users when new information changes meet specified thresholds. In this dissertation, we first present the system level facilities for building an Internet-scale continual query system, including the design and development of two operational CQ monitoring systems OpenCQ and WebCQ, the engineering issues involved, and our solutions. We then describe a number of research challenges that are specific to large-scale information monitoring and the techniques developed in the context of OpenCQ and WebCQ to address these challenges. Example issues include how to efficiently process large number of continual queries, what mechanisms are effective for building a scalable distributed trigger system that is capable of handling tens of thousands of triggers firing at hundreds of data sources, how to effectively disseminate fresh information to the right users at the right time. We have developed a suite of techniques to optimize the processing of continual queries, including an effective CQ grouping scheme, an auxiliary data structure to support group-based indexing of CQs, and a differential CQ evaluation algorithm (DRA). The third contribution is the design of an experimental evaluation model and testbed to validate the solutions. We have engaged our evaluation using both measurements on real systems (OpenCQ/WebCQ) and simulation-based approach. To our knowledge, the research documented in this dissertation is to date the first one to present a focused study of research and engineering issues in building large-scale information monitoring systems using continual queries.

This paper presents WebCQ, a continual query system for large-scale Web information monitoring. WebCQ is designed to discover and detect changes to Web pages efficiently, and to notify users of interesting changes with personalized messages. Users' Web page monitoring requests are modeled as continual queries on the Web and referred to as Web page sentinels. The system consists of five main components: a change detection robot that discovers and detects changes, a proxy cache service that reduces the communication traffics to the original information provider on the remote server, a trigger evaluation tool that can filter only the changes that match certain thresholds, a personalized change presentation tool that highlights Web page changes, and a change notification service that displays and delivers interesting changes and fresh information to the right users at the right time. This paper describes the WebCQ system with an emphasis on the general issues in designing and engineering a large-scale information change monitoring system on the Web. There are two main contributions. First, we present the mechanisms that WebCQ provides to support various types of Web page sentinels for finding and displaying interesting changes to Web pages. The large collection of sentinel types allows WebCQ to efficiently locate and monitor a wide range of changes in Web pages. The second contribution is the development of sentinel grouping techniques for efficient and scalable processing of large number of concurrently running triggers and Web page sentinels. We report our initial experimental results showing the effectiveness of the proposed solutions.

Query routing is an intelligent service that can direct query requests to appropriate servers that are capable of answering the queries. The goal of a query routing system is to provide efficient associative access to a large, heterogeneous, distributed collection of information providers by routing a user query to the most relevant information sources that can provide the best answer. Effective query routing not only minimizes the query response time and the overall processing cost, but also eliminates a lot of unnecessary communication overhead over the global networks and over the individual information sources. The AQR-Toolkit divides the query routing task into two cooperating processes: query refinement and source selection. It is well known that a broadly defined query inevitably produces many false positives. Query refinement provides mechanisms to help the user formulate queries that will return more useful results and that can be processed efficiently. As a complimentary process, source selection reduces false negatives by identifying and locating a set of relevant information providers from a large collection of available sources. By pruning irrelevant information sources, source selection also reduces the overhead of contacting the information servers that do not contribute to the answer of the query. The system architecture of AQR-Toolkit consists of a hierarchical network (a directed acyclic graph) with external information providers at the leaves and query routers as mediating nodes. The end-point information providers support query-based access to their documents. At a query router node, a user may browse and query the meta information about information providers registered at that query router or make use of the router's facilitates for query refinement and source selection.

WebCQ is a prototype system for large-scale Web information monitoring and delivery. It makes heavy use of the structure present in hypertext and the concept of continual queries. In this paper we discuss both mechanisms that WebCQ uses to discover and detect changes to the World Wide Web (the Web) pages e ciently, and the methods to notify users of interesting changes with a personalized customization. The WebCQ system consists of four main components: a change detection robot that discovers and detects changes, a proxy cache service that reduces communication tra cs to the original information servers, a personalized presentation tool that highlights changes detected by WebCQ sentinels, and a change noti cation service that delivers fresh information to the right users at the right time. A salient feature of our change detection robot is its ability to support various types of web page sentinels for detecting, presenting, and delivering interesting changes to web pages. This paper describes the WebCQ system with an emphasis on general issues in designing and engineering a large-scale information change monitoring system on the Web.

The World Wide Web (the Web) has made an enormous amount of data freely accessible over the Internet. However, nding the right information in the midst of this mountain of data has been likened to nding the proverbial needle in a haystack. Commonly used search engines (e.g., AltaVista) and directory services (e.g., Yahoo) have practical but limited success. The exponential growth of the Web is increasing the haystack rapidly. Instead of pull-based browsing, update monitoring is a promising area of research where the system brings the right information to the right user at the right time. In this paper we present the design and implementation of the Conquer continual query system, designed for update monitoring over the Web information sources. A Continual Query (CQ) is a standing query that monitors update of interest using distributed triggers and noti es the user of changes whenever an update of interest reaches speci ed thresholds or some time limit is reached. In contrast to normal queries whose scope is limited to past and present data, the scope of a continual query also includes future data. The rst contribution of the Conquer system is the speci cation language and the inherent semantics of continual queries. The second contribution consist of the mechanisms for e cient and scalable processing of large numbers of continual queries. The third contribution is the three-tier architecture that provides active capabilities at both mediator tier and wrapper tier and interoperability among multiple web information sources.

There has been tremendous interest in information integration systems that auto:matically gather, manipulate, and integrate data from multiple information sources on a user’s behalf. Unfortunately, web sites are primarily designed for human browsing rather than for use by a computer program. Mechanically extracting their content is in general a rather difficult job if not impossible [4]. Software systems using such web informa.tion sources typically use hand-coded wrappers to extract information content of interest from web sources and translate query responses to a more structured format (e.g., relational form) before unifying them into an integrated answer to a. user’s query. The most recent generation of information mediator systems (e.g., Ariadne [3], CQ [5, 71, Internet Softbots [4], TSIMMIS [2]) addresses this problem by enabling a pre-wrapped set of web sources to be accessed via database-like queries. However, hand-coding a wrapper is time consuming and error-prone. We have also observed that, by using a good design methodology, only a relatively small part of the code deals with t,he source-specific access details, the rest of the code is either common among wrappers or can be expressed in a high level, more structured fashion. As the Web grows, maintaining a reasonable number of wrappers becomes impractical. First, the number of information sources of interest to a user query can be quite large, even within a particular domain. Second, new information sources are constantly added on the Web. Thirdly, the content and presentation format of the existing information sources may change frequently and autonomously. With these observations in mind, we have developed a wrapper generation system, called XWrap, for semi-automatic construction of wrappers for Web information sources. The system contains a library of commonly used functions, such as receiving queries from applications, handling of filter queries, and packaging results. It also contains some source-specific facilities that are in charge of mapping a mediator query to a remote cormection call to fetch the relevant pages and translating the retrieve’d page(s) into a more structured format (such as XML documents or relational tables). A distinct feature of our wrapper generator is its ability to pro-

We introduce the concept of continual queries, describe the design of a distributed event-driven continual query system-OpenCQ, and outline the initial implementation of OpenCQ on top of the distributed interoperable information mediation system DIOM. Continual queries are standing queries that monitor update of interest and return results whenever the update reaches specified thresholds. In OpenCQ, users may specify to the system the information they would like to monitor (such as the events or the update thresholds they are interested in). Whenever the information of interest becomes available, the system immediately delivers it to the relevant users; otherwise, the system continually monitors the arrival of the desired information and pushes it to the relevant users as it meets the specified update thresholds. In contrast to conventional pull-based data management systems such as DBMSs and Web search engines, OpenCQ exhibits two important features: it provides push-enabled, event-driven, content-sensitive information delivery capabilities; and it combines pull and push services in a unified framework. By event-driven we mean that the update events of interest to be monitored are specified by users or applications. By content-sensitive, we mean the evaluation of the trigger condition happens only when a potentially interesting change occurs. By push-enabled, we mean the active delivery of query results or triggering of actions without user intervention.

This paper presents the design and implementation methodology of the JCQ system, a Java-based Continual Query system for update monitoring over Web information sources. A continual query is a standing query that monitors updates of interest using distributed triggers and notifies users whenever the updates reach specified thresholds. In this paper we focus on the strategies and techniques developed in JCQ for scalable and efficient trigger firing and the execution model for flexible and robust change notification. We evaluate our approach through a performance study of the most recent release of the JCQ system and a comparison with related work.

Conquer is an extensible architecture designed for transforming a pull-based data management system (e.g., conventional DBMSs, web search engines and query systems) into a push-enabled data management systems. The key ideas of Conquer are the concept of continual queries; the mechanisms for eecient and scalable implementation of continual queries 20], including synchronous or asynchronous event monitoring, immediate or deferred execution coupling; and the three-tier architecture that provides data independence in both mediator level and wrapper level. The distinct features of the Conquer architecture include: reusing and extending conventional DBMS components, providing push-enabled services by incorporating distributed event-driven triggers, and combining pull and push services in a uniied framework. This paper presents the design of Conquer and its initial implementation on top of the distributed interoperable information mediation system DIOM 17, 15].

The CQ project at OGI, funded by DARPA, aims at developing a scalable toolkit and techniques for update monitoring and event-driven information delivery on the net. The main feature of the CQ project is a “personalized update monitoring” toolkit based on continual queries [3]. Comparing with the pure pull (such as DBMSs, various web search engines) and pure push (such as Pointcast, Marimba, Broadcast disks) technology, the CQ project can be seen as a hybrid approach that combines the pull and push technology by supporting personalized update monitoring through a combined client-pull and server-push paradigm.