A major challenge when dealing with large collections of digital images is to find relevant objects, especially when no metadata on the objects is available. Content-based image retrieval (CBIR) addresses this problem but usually lacks query images that are good enough to express the user’s information need. Therefore, in Query-by-Sketch, CBIR has been considered with user provided sketches as query objects – but so far, this has suffered from the limitations of existing user interfaces. In this paper, we present a novel user interface for query by sketch that exploits emergent interactive paper and digital pen technology. Users can draw sketches on paper in a user-friendly way. Search can be started interactively from the paper front-end, due to a streaming interface from the digital pen to the underlying CBIR system. We present the implementation of the interactive paper/digital pen interface on top of QbS, our system for CBIR using sketches, and we present in detail the evaluation of the system on the basis of the MIRFLICKR-25000 image collection.

This demo will interactively show a system that exploits a novel user interface, running on Tablet PCs or graphic tablets, that provides query-by-sketch based image retrieval using color sketches. The system uses Angular Radial Partitioning (ARP) for the edge information in the sketches and color moments in the CIELAB space, combined with a distance metric that is robust to deviations in color as they usually need to be taken into account with user-generated color sketches.

With the increasingly growing size of digital image collections, known image search is gaining more and more importance. Especially in collections where individual objects are not tagged with metadata describing their content, content-based image retrieval (CBIR) is a promising approach. However, the application of CBIR to known item search usually suffers from the unavailability of query images that are good enough to express the user’s information need. In this technical report, we present the QbS system that provides content-based search in large image collections based on user-drawn sketches. The QbS system combines angular radial partitioning for the extraction of features in the user-provided sketch, taking into account the spatial distribution of edges, and the image distortion model. This combination offers several highly relevant invariances that allow the query sketch to slightly deviate from the searched image in terms of rotation, translation, relative size, and/or unknown objects in the background. To illustrate the benefits of the QbS approach, we present search results from the evaluation of our system on the basis of the MIRFLICKR collection with 25,000 objects and compare the retrieval results of pure metadata-driven approaches, pure content-based retrieval using different sketches, and combinations thereof.

With the increasingly growing size of digital image collections, known image search is gaining more and more importance. Especially in collections where individual objects are not tagged with metadata describing their content, content-based image retrieval (CBIR) is a promising approach, but usually suffers from the unavailability of query images that are good enough to express the user's information need. In this paper, we present the QbS system that provides CBIR based on user-drawn sketches. The QbS system combines angular radial partitioning for the extraction of features in the user-provided sketch, taking into account the spatial distribution of edges, and the image distortion model. This combination offers several highly relevant invariances that allow the query sketch to slightly deviate from the searched image in terms of rotation, translation, relative size, and/or unknown objects in the background. To illustrate the benefits of the approach, we present search results from the evaluation of the QbS system on the basis of the MIRFLICKR collection with 25,000 objects and compare the retrieval results of pure metadata-driven approaches, pure content-based retrieval using different sketches, and combinations thereof.

With the increasingly growing size of digital image collections, known image search is gaining more and more importance. Especially when the objects in such collections do not possess appropriate metadata (e.g., tags, annotations), content-based image retrieval (CBIR) is a promising approach. However, the application of CBIR to known item search usually suffers from the unavailability of query images that are good enough to express the user's information need. In order to improve this situation, we propose the QbS system which provides an approach to content-based search in large image collections based on user-drawn sketches. By exploiting novel devices for human-computer interaction like interactive paper, tablet PCs, or graphic tablets, users are able to draw a sketch that reflects their information need and start a content-based search using this sketch. The QbS system provides query support and offers several invariances that allow the user-generated sketch to slightly deviate from the searched image in terms of rotation, translation, relative size, and/or unknown objects in the background. To illustrate the benefits of the approach, we show search results from the evaluation of QbS on the basis of the MIRFLICKR collection with 25'000 objects.

The image distortion model (IDM) is a deformation model that is used for computing the (dis-)similarity between images. Therefore it evaluates displacements of individual pixels between two images within a so-called warp range and also takes into account the surrounding pixels (local context). It can be used with a kNN classifier and has shown good retrieval quality in handwritten character recognition as well as in past runs of the medical automatic annotation task of ImageCLEF workshops. However, one of its limitations is computational complexity and the resulting long query response times, that may limit its use for a wider range of applications and for modifications to further improve retrieval quality. In particular an enlarged local context and warp range are candidates for such improvements, but would even further increase computational complexity.In our approach, we therefore apply several optimizations that reduce the retrieval time without degrading the result quality. First, we use an early termination strategy for the individual distance computations which contribute a speedup of a factor of 4.3-4.9. Second, we make efficient use of multithreading. With these extensions, we are able to perform the IDM in less than 1.5 s per query on an 8-way server and 16 s on a standard Pentium 4 PC without any degradation of retrieval quality compared to the non-optimized version. We extend the possible displacements to an area of 7 x 7 pixels, using a local context of either 5 x 5 or 7 x 7 pixels. The results of the extended IDM have been submitted to the medical automatic annotation task of ImageCLEF 2007 and were ranked in the upper third. More importantly, the used techniques for reducing the execution time are not limited strictly to IDM but are also applicable to other expensive distance measures. (C) 2008 Elsevier B.V. All rights reserved.

Content-based image retrieval mainly follows a Query-byExample approach and therefore requires well selected examples to start an initial search. This position paper describes how Regions of Interest (ROI) can be used to better adapt the system to the user’s information needs. In particular, it highlights how novel input devices such as Interactive Paper or TabletPCs can be used to capture much more details about what the user is precisely looking for already at the time the query is defined or when relevance feedback is specified.

The Image Distortion Model (IDM) has previously shown good retrieval quality. However, one of the limitations that may limit its use in a wider range of applications is computational complexity. In this paper, we present an approach that applies several optimizations to decrease the retrieval time of IDM without degrading the quality of query results. We were able to perform the IDM in less than 1.5 seconds per query on an 8-way server and 16 seconds on a standard Pentium 4. In particular, the early termination strategy we applied contributed a speedup of up to 4.9. We also extended the possible displacements to an area of 7×7 pixels with a local context of up to the same size. The results submitted to the medical automatic annotation task of ImageCLEF’2007 were ranked in the upper third. Most importantly, the proposed techniques are not limited to IDM but can also be applied to other expensive distance measures.

Digital Libraries (DLs) in eHealth are composed of electronic artefacts that are generated and owned by different healthcare providers. A major characteristic of eHealth DLs is that information is under the control of the organisation where data has been produced. The electronic health record (EHR) of patients therefore consists of a set of distributed artefacts and cannot be materialised for organisational reasons. Rather, the EHR is a virtual entity. The virtual integration of an EHR is done by encompassing services provided by specialised application systems into processes. This paper reports, from an application point of view, on national and European attempts to standardise electronic health EHR. From a technical perspective, the paper addresses how services can be made available in a distributed way, how distributed P2P infrastructures for the management of EHRs can be evaluated, and how novel contentbased access can be provided for multimedia EHRs.

This paper introduces D ILIGENT , a digital library infrastructure built by integrating digital library and Grid technologies and resources. This infrastructure allows different communities to dynamically build specialised digital libraries capable to support the entire e-Science knowledge production and consumption life-cycle by using shared computing, storage, content, and application resources. The paper presents some of the main software services that implement the D ILIGENT system. Moreover, it exemplifies the provided features by presenting how the D ILIGENT infrastructure is being exploited in supporting the activity of user communities working in the Earth Science Environmental sector.

Content-based retrieval has become a very popular and also powerful paradigm for searching in multimedia collections, especially in large collections of images. However, such queries require that one or even several reference images are available prior to the start of the search process. These reference images must be close to the final result so that the user can take them to express her information need. If such reference images are not available or if the information need is covered only by parts of the query object, the result usually does not meet the user’s expectation. Therefore, more flexible user interfaces are needed that allow users to sketch a query image by hand drawings and to dynamically select regions of interest from a given query image. In this paper, we present a novel approach to query by sketch where interactive paper and image similarity search are seamlessly combined. It is based on the iPaper/iServer system of ETH Zurich and the ISIS/OSIRIS content-based image retrieval system of the University of Basel. The paper presents the integrated system which has already very successfully been applied to the development of an interactive museum catalogue. Moreover, it reports on ongoing activities that aim at extending the system to support handwritten sketches, gestures and/or dynamic region selection to make the retrieval process more flexible and less dependent from existing query objects.

Content-based retrieval has become a very popular and also powerful paradigm for searching in multimedia collections, especially in large collections of images. However, such queries require that one or even several reference images are available prior to the start of the search process. These reference images must be close to the final result so that the user can take them to express her information need. If such reference images are not available or if the information need is covered only by parts of the query object, the result usually does not meet the user’s expectation. Therefore, more flexible user interfaces are needed that allow users to sketch a query image by hand drawings and to dynamically select regions of interest from a given query image. In this paper, we present a novel approach to query by sketch where interactive paper and image similarity search are seamlessly combined. It is based on the iPaper/iServer system of ETH Zurich and the ISIS/OSIRIS content-based image retrieval system of the University of Basel. The paper presents the integrated system which has already very successfully been applied to the development of an interactive museum catalogue. Moreover, it reports on ongoing activities that aim at extending the system to support handwritten sketches, gestures and/or dynamic region selection to make the retrieval process more flexible and less dependent from existing query objects.

Future information spaces such as Digital Libraries require new infrastructures that allow to use and to combine various kinds of functions in a unified and reliable way. The paradigm of service-oriented architectures (SoA) allows providing application functionality in a modular, self-contained way and to individually combine this functionality. The paper presents the ISIS/OSIRIS system which consists of a generic infrastructure for the reliable execution of distributed service-based applications (OSIRIS) and a set of dedicated Digital Library application services (ISIS) that provide, among others, content-based search in multimedia collections.

DelosDLMS is a prototype of a next-generation Digital Library (DL) management system. It is realized by combining various specialized DL functionalities provided by partners of the DELOS network of excellence. Currently, DelosDLMS combines text and audio-visual searching, offers new information visualization and relevance feedback tools, provides novel interfaces, allows retrieved information to be annotated and processed, integrates and processes sensor data streams, and finally, from a systems engineering point of view, is easily configured and adapted while being reliable and scalable. The prototype is based on the OSIRIS/ISIS platform, a middleware environment developed by ETH Zürich and now being extended at the University of Basel.

Future digital libraries will not only contain pure text documents, but increasingly hold massive amounts of compound documents which comprise many multimedia objects, e.g. texts, images, audio, and video. Already existing collections of documents, e.g. all electronic health records of one clinic can form a digital library with millions of multimedia objects and a total storage of several terabytes. It is therefore important to provide ways for effective and efficient retrieval for those collections. This paper proposes a novel approach for compound document matching using a filter-and-refinement algorithm for similarity-based retrieval. At the same time, this approach increases the effectiveness by establishing only semantically meaningful matches and providing greater expressiveness in queries by restricting the number of allowed matches to a single query object.

This paper presents the OSIRIS middleware (Open Service Infrastructure for Reliable and Integrated process Support) which allows combining different distributed services into processes, thereby implementing flexible digital library applications. In OSIRIS, we distinguish between system services and application services. System services are used internally for coordinating the execution of processes in a distributed way, without relying on a central execution engine/scheduler. In addition, the distributed architecture of OSIRIS allows for reliable peer-to-peer process management and sophisticated failure handling according to the transactional process model and is able to balance the load among several providers of the same service. For application services, we distinguish between loosely coupled and tightly coupled services. Usually, a part of the distributed OSIRIS middleware (called OSIRIS layer) runs on each host providing application services. These application services are considered as tightly coupled since, in case information on their transactional properties like compensation or re-invocation (retriability) is available, dedicated transactional guarantees for processes can be provided. Loosely-coupled application services are those that have to be called remotely, without a local ORISIS layer available. The integration/invocation is done via WSDL for service description and SOAP for invocation. The core of OSIRIS has been developed at ETH Zürich. Currently, it is being extended (e.g., for processing continuous streams of data) jointly at UMIT and at the University of Basel. There are two versions of OSIRIS: one is implemented in C++ and runs on Microsoft platforms, the other version is implemented in Java. Second, this paper also introduces the ISIS application (Interactive SImilarity Search) that has been built on top of the OSIRIS platform. ISIS consists of a set of pre-defined processes and several application services (like feature extraction, index management, index access, relevance feedback, etc.) which are used within these processes. The paper presents how the ISIS application has been built and the application services it relies on. Finally, it also gives detailed information on how new digital library applications can be built on the basis of existing services.