The LivingKnowledge project aimed to enhance the current state of the art in search, retrieval and knowledge management on the web by advancing the use of sentiment and opinion analysis within multimedia applications. To achieve this aim, a diverse set of novel and complementary analysis techniques have been integrated into a single, but extensible software platform on which such applications can be built. The platform combines state-of-the-art techniques for extracting facts, opinions and sentiment from multimedia documents, and unlike earlier platforms, it exploits both visual and textual techniques to support multimedia information retrieval. Foreseeing the usefulness of this software in the wider community, the platform has been made generally available as an open-source project. This paper describes the platform design, gives an overview of the analysis algorithms integrated into the system and describes two applications that utilise the system for multimedia information retrieval.

The data contained on the web and the social web are inherently multimedia and consist of a mixture of textual, visual and audio modalities. Community memories embodied on the web and social web contain a rich mixture of data from these modalities. In many ways, the web is the greatest resource ever created by human-kind. However, due to the dynamic and distributed nature of the web, its content changes, appears and disappears on a daily basis. Web archiving provides a way of capturing snapshots of (parts of) the web for preservation and future analysis. This paper provides an overview of techniques we have developed within the context of the EU funded ARCOMEM (ARchiving COmmunity MEMories) project to allow multimedia web content to be leveraged during the archival process and for post-archival analysis. Through a set of use cases, we explore several practical applications of multimedia analytics within the realm of web archiving, web archive analysis and multimedia data on the web in general.

Millions of images are tweeted every day, yet very little research has looked at the non-textual aspect of social media communication. In this work we have developed a system to analyse streams of image data. In particular we explore trends in similar, related, evolving or even duplicated visual artefacts in the mass of tweeted image data - in short, we explore the visual pulse of Twitter.

The 2013 MediaEval Crowdsourcing task looked at the problem of working with noisy crowdsourced annotations of image data. The aim of the task was to investigate possible techniques for estimating the true labels of an image by using the set of noisy crowdsourced labels, and possibly any content and metadata from the image itself. For the runs in this paper, we’ve applied a shotgun approach and tried a number of existing techniques, which include generative probabilistic models and further crowdsourcing.

The 2013 MediaEval Retrieving Diverse Social Images Task looked to tackling the problem of search result diversication of Flickr results sets formed from queries about geographic places and landmarks. In this paper we describe our approach of using a min-max similarity diversier coupled with pre-lters and a reranker. We also demonstrate a number of novel features for measuring similarity to use in the diversication step.

The data contained within the web is inherently multimedia; consisting of a rich mix of textual, visual and audio modalities. Prospective Web Observatories need to take this into account from the ground up. This paper explores some uses for the automatic analysis of multimedia data within a Web Observatory, and describes a potential platform for an extensible and scalable multimedia Web Observatory.

This paper describes a modular architecture for searching and hyperlinking clips of TV programmes. The architecture aimed to unify the combination of features from different modalities through a common representation based on a set of probability density functions over the timeline of a programme. The core component of the system consisted of analysis of sections of transcripts based on a textual query. Results show that search is made worse by the addition of other components, whereas in hyperlinking precision is increased by the addition of visual features.

The data contained on the web and social web is inherently multimedia; consisting of a mix of textual, visual and audio modalities. Community memories embodied on the web and social web contain a rich mixture of data from these modalities. This paper explores some uses for the automatic analysis of multimedia data within the context of the archival and post-archival analysis of community memories on the web and social web.

This paper describes the approach we take to the analysis of social media, combining opinion mining from text and multimedia (images, videos, etc), and centred on entity and event recognition. We examine a particular use case, which is to help archivists select material for inclusion in an archive of social media for preserving community memories, moving towards structured preservation around semantic categories. The textual approach we take is rule-based and builds on a number of sub-components, taking into account issues inherent in social media such as noisy ungrammatical text, use of swear words, sarcasm etc. The analysis of multimedia content complements this work in order to help resolve ambiguity and to provide further contextual information. We provide two main innovations in this work: first, the novel combination of text and multimedia opinion mining tools; and second, the adaptation of NLP tools for opinion mining specific to the problems of social media.

Combining items from social media streams, such as Flickr photos and Twitter tweets, into meaningful groups can help users contextualise and effectively consume the torrents of information now made available on the social web. This task is made challenging due to the scale of the streams and the inherently multimodal nature of the information to be contextualised. We present a methodology which approaches social event detection as a multi-modal clustering task. We address the various challenges of this task: the selection of the features used to compare items to one another; the construction of a single sparse affinity matrix; combining the features; relative importance of features; and clustering techniques which produce meaningful item groups whilst scaling to cluster large numbers of items. In our best tested configuration we achieve an F1 score of 0.94, showing that a good compromise between precision and recall of clusters can be achieved using our technique.

There is a wide array of online photographic content that is not geotagged. Algorithms for efficient and accurate geographical estimation of an image are needed to geolocate these photos. This paper presents a general model for using both textual metadata and visual features of photos to automatically place them on a world map.

ImageTerrier is a novel easily extensible open-source, scalable, high-performance search engine platform for content-based image retrieval applications. The platform provides a comprehensive test-bed for experimenting with bag-of-visual-words image retrieval techniques. It incorporates a state-of-the-art implementation of the single-pass indexing technique for constructing inverted indexes and is capable of producing highly compressed index data structures. ImageTerrier is written as an extension to the open-source Terrier, "Terabyte Retriever", test-bed platform for textual information retrieval research. The ImageTerrier platform is demonstrated to successfully index and search a corpus of over 10 million images containing just under 10,000,000,000 quantised SIFT visual terms.

ImageTerrier is a novel easily extensible open-source, scalable, high-performance search engine platform for content-based image retrieval applications. The platform provides a comprehensive test-bed for experimenting with bag-of-visual-words image retrieval techniques. It incorporates a state-of-the-art implementation of the single-pass indexing technique for constructing inverted indexes and is capable of producing highly compressed index data structures. ImageTerrier is written as an extension to the open-source Terrier, "Terabyte Retriever", test-bed platform for textual information retrieval research. The ImageTerrier platform is demonstrated to successfully index and search a corpus of over 10 million images containing just under 10,000,000,000 quantised SIFT visual terms.

In this paper, we describe the work we are undertaking in producing a truly multimedia platform for the analysis of facts and opinions on the web. The system integrates the analysis of multimodal data (images, text and page layout) into a distributable platform that can be built upon for various applications. We give an overview of the natural language processing tools that have been developed for extracting facts and opinions from the textual content of articles, the image analysis techniques used to extract facts and to help support the opinions found in the contextually related written information, as well as other multimodal tools developed for the analysis of online articles. We describe two applications that have been developed as part of ongoing work of the Living Knowledge project: the News Media Analysis application for the semi-automation of the work of a media analysis company and the Future Predictor application which allows exploration of claims that are made through time.

OpenIMAJ and ImageTerrier are recently released open-source libraries and tools for experimentation and development of multimedia applications using Java-compatible programming languages. OpenIMAJ (the Open toolkit for Intelligent Multimedia Analysis in Java) is a collection of libraries for multimedia analysis. The image libraries contain methods for processing images and extracting state-of-the-art features, including SIFT. The video and audio libraries support both cross-platform capture and processing. The clustering and nearest-neighbour libraries contain efficient, multi-threaded implementations of clustering algorithms. The clustering library makes it possible to easily create BoVW representations for images and videos. OpenIMAJ also incorporates a number of tools to enable extremely-large-scale multimedia analysis using distributed computing with Apache Hadoop. ImageTerrier is a scalable, high-performance search engine platform for content-based image retrieval applications using features extracted with the OpenIMAJ library and tools. The ImageTerrier platform provides a comprehensive test-bed for experimenting with image retrieval techniques. The platform incorporates a state-of-the-art implementation of the single-pass indexing technique for constructing inverted indexes and is capable of producing highly compressed index data structures.

In this paper we present our experience of representing the knowledge behind HealthAgents (HA), a distributed decision support system for brain tumour diagnosis. Our initial motivation came from the distributed nature of the information involved in the system and has been enriched by clinicians' requirements and data access restrictions. We present in detail the steps we have taken towards building our ontology starting from knowledge acquisition to data access and reasoning. We motivate our representational choices and show our results using domain examples used by clinical partners in HA.

AbstractNew biomedical technologies enable the diagnosis of brain tumours by using non-invasive methods. HealthAgents is a European Union-funded research project that aims to build an agent-based distributed decision support system (dDSS) for the diagnosis of brain tumours. This is achieved using the latest biomedical knowledge, information and communication technologies and pattern recognition (PR) techniques. As part of the PR development of HealthAgents, an independent and automatic classification framework (CF) has been developed. This framework has been integrated with the HealthAgents dDSS using the HealthAgents agent platform. The system offers (1) the functionality to search for distributed classifiers to solve specific questions; (2) automatic classification of new cases; (3) instant deployment of new validated classifiers; and (4) the ability to rank a set of classifiers according to their performance and suitability for the case in hand. The CF enables both the deployment of new classifiers using the provided Extensible Markup Language1 classifier specification, and the inclusion of new PR techniques that make the system extensible. These features may enable the rapid integration of PR laboratory results into industrial or research applications, such as the HealthAgents dDSS. Two classification nodes have been deployed and they currently offer classification services by means of dedicated servers connected to the HealthAgents agent platform: one node being located at the Katholieke Universiteit Leuven, Belgium and the other at the Universidad Politécnica de Valencia, Spain. These classification nodes share the current set of brain tumour classifiers that have been trained from in vivo magnetic resonance spectroscopy data. The combination of the CF with a distributed agent system constitutes the basis of the brain tumour dDSS developed in HealthAgents.

AbstractThe HealthAgents project aims to provide a decision support system for brain tumour diagnosis using a collaborative network of distributed agents. The goal is that through the aggregation of the small data sets available at individual hospitals, much better decision support classifiers can be created and made available to the hospitals taking part. In this paper, we describe the technicalities of the HealthAgents framework, in particular how the interoperability of the various agents is managed using semantic web technologies. On the broad scale the architecture is based around distributed data-mart agents that provide ontological access to hospitals’ underlying data that has been anonymized and processed from proprietary formats into a canonical format. Classifier producers have agents that gather the global data from participating hospitals such that classifiers can be created and deployed as agents. The design on a microscale has each agent built upon a generic-layered framework that provides the common agent program code, allowing rapid development of agents for the system. We believe that our framework provides a well-engineered, agent-based approach to data sharing in a medical context. It can provide a better basis on which to investigate the effectiveness of new classification techniques for brain tumour diagnosis.