We will show how to build innovative multimodal dialog user interfaces that integrate multiple heterogeneous web services as data sources on the basis of the Ontology-based Dialog Platform (ODP). More specifically, we will describe how to exploit ODP's well-defined extension points and how generic ODP processing modules can be adopted, in order to support a rapid dialog system engineering process. By means of the latest ODP-based educational information system CIRIUS and the ODP workbench, a set of Eclipse-based editors and tools, we demonstrate step-by-step along the generic multimodal dialog processing chain what has to be done for developing a new multimodal dialog user interface for a specific application domain.

The design and implementation of combined mobile and touchscreen-based multimodal Web 3.0 interfaces should include new approaches of intelligent semantic mediation, knowledge acquisition and user interaction when dealing with a semantic-based digitalization of mostly unstructured textual or image-based source information. In this article, we propose a semantic-based model for those three tasks. The technical components rely on semantic web data structures in order to, first, transcend the traditional keyboard and mouse interaction metaphors, and second, provide the representation structures for more complex, collaborative interaction scenarios that may combine mobile with terminal-based interaction to accommodate the growing need to store, organize, and retrieve all these data. Interactive knowledge acquisition plays a major role in increasing the quality of automatic annotations as well as the usability of different intelligent user interfaces to control, correct, and add annotations to unstructured text and image sources. Examples are provided in the context of the Medico and Texo use cases.

Harvesting processes are in fact industrial manufacturing processes that follow a tight schedule. Unexpected incidents can disturb a harvest and require a replanning of the process in order to avoid severe financial losses. When a new plan has been found, it must be communicated to the affected process participants, i.e., drivers of agricultural machinery. This paper presents a cloud-based system for orchestrating and coordinating a fleet of agricultural machinery and their drivers during an ongoing harvest in case of an unexpected incident. A management dashboard allows the real-time replanning of a harvesting process and sends updated instructions to each affected driver’s mobile device. The paper focuses on the communication between the contractor and a driver in the field as well as the interaction of the driver with his mobile device. It is explained how the system accomplishes a fast, traceable, and safe communication with the drivers that may suffer from bad network conditions and a high cognitive load. In order to understand the details of his new tasks, a driver can examine them in a multimodal dialogue including speech with the system. This is beneficial in a driving situation. By interacting with the mobile client, the system also deduces if the driver correctly understood his new instructions and can intervene if not. Keywords–Multimodal Dialogue; Task Assignment; Task Introspection; Agriculture; Harvesting Process.

This paper presents a model-driven development approach to rapidly create multimodal dialogue applications for new domains. A reusable and consistent base model and generic processes inside a multimodal dialogue framework enable advanced dialogue phenomena and allow for a scenarioand domain-specific customization without the necessity to adapt the core framework. We introduce declarative adaptation and extension points within the discussed models for input interpretation, output presentation, and semantic content in order to easily integrate new modalities, domainspecific interactions, and service back-ends. Three multimodal dialogue applications for different use-cases prove the practicability of the presented approach.

We describe a model-driven approach for developing multimodal user interfaces for services that are especially appropriate for mobile consumption. Thereby, we refine an existing service engineering methodology with respective meta-models and provide initial tool support for user experience experts by means of an Eclipse-based, graphical editor implementing model-to-model transformations, consistency checks, and automatic code generation. The resulting runnable UI can either be consumed from a desktop pc as well as from modern smartphones via browser-based, multimodal client applications. This allows for a seamless user experience throughout different devices and situations. Such a client application is linked to a service-oriented, ontology-based dialogue platform that acts as a middleware between the client and the service backend and enables a user to naturally interact with services.

We present Calisto, a service-oriented information kiosk system for public places, like museums or hotel lobbies. Calisto supports collaboration between multiple users. They can connect their mobile devices to the large public terminal and share interesting facts and media contents via intuitive multimodal interaction. The novel contribution of our work is a seamless combination of a touch-based kiosk system and mobile devices for accessing heterogeneous information services.

In this paper, we describe a mobile Business-to-Business (B2B) interaction system. The mobile device supports users in accessing a service platform. A multimodal dialogue system allows a business expert to intuitively search and browse for services in a real-world production pipeline. We implemented a distributed client-server dialogue application for natural language speech input and speech output generation. On the mobile device, we implemented a multimodal client application which comprises of a GUI for touch gestures and a three-dimensional visualization. The client is linked to an ontology-based dialogue platform and fully leverages the device’s interaction capabilities in order to provide intuitive access to the service platform while on the go.

We present a new multimodal interaction sequence for a mobile multimodal Business-to-Business interaction system. A mobile client application on the iPhone supports users in accessing an online service marketplace and allows business experts to intuitively search and browse for services using natural language speech and gestures while on the go. For this purpose, we utilize an ontology-based multimodal dialogue platform as well as an integrated trainable gesture recognizer.

Hiking tourism is booming, innovative, and an important factor for the economic development of many countryside regions. Hikers have a strong need for information which they currently obtain through diverse, heterogeneous sources that are not ubiquitously accessible. They also have a strong community spirit which has not been addressed so far by intelligent systems. We worked on a novel mobile information system that allows ubiquitous access to the contained information, since hikers are seamlessly accompanied throughout all process stages before, during and after their actual trip. Furthermore, the system keeps them always up-to-date with relevant and geo-referenced information. For example, users are informed about a dangerous concentration of ozone in the atmosphere. The system also leverages the community spirit and enables communication and collaboration in case of an emergency, which significantly increase safety. We evaluated the system in a field trial in context of the German Hiking Summit which brought us valuable feedback and new ideas for further improvement.

Moving in virtual environments has become very common. On the one hand there are e.g. non-interactive route guidance systems which are not adequate for pedestrians. On the other hand interactive applications like Second Life fascinate a broad community. This work presents the information system UBIGIouS that combines reality with virtuality in an interactive mixed reality scenario. It is intended to link Virtual Reality Geographic Information Systems with purely virtual interactive applications. For ubiquitous access, users can participate from their homes by standard, stationary PCs or on site using mobile devices like smartphones. By means of visual building recognition capabilities UBIGIouS offers location-based services with or without positioning via the Global Positioning System GPS.