Integrating Distributed Information Sources with CARROT II

We describe CARROT II (C2), an agent-based architecture for distributed information retrieval and document collection management. C2 can consist of an arbitrary number of agents, distributed across a variety of platforms and locations. C2 agents provide search services over local document collections or information sources. They advertise content-derived metadata that describes their local document store. This metadata is sent to other C2 agents which agree to act as brokers for that collection, and every agent in the system has the ability to serve as such a broker. A query can be sent to any C2 agent, which can decide to answer the query itself from its local collection, or to send the query on to other agents whose metadata indicate that they would be able to answer the query, or send the query on further. Search results from multiple agents are merged and returned to the user. C2 differs from similar systems in that metadata takes the form of an automatically generated, unstructured feature vector, and that any agent in the system can act as a broker, so there is no centralized control. We present experimental results of retrieval performance and effectiveness in a distributed environment.