As we transition from the mobile internet era to the ‘Cognitive Internet,’ a transformative change is occurring in our interaction with technology, data, and intelligence. The Cognitive Internet transcends the Cognitive Internet of Things (Cognitive IoT) by integrating intelligence across objects, systems, and domains dynamically. This paper explores the foundational elements, characteristics, and benefits of the Cognitive Internet, emphasizing the crucial role of Hybrid Edge Cloud (HEC) platforms. We highlight how resilient AI infrastructures support the proliferation of cognitive services, a Knowledge as a Service (KaaS) economy, enhanced decision-making autonomy, and sustainable digital progress. This paper serves as a guide for understanding and leveraging the Cognitive Internet’s potential, supported by case studies and real-world applications.

Federated Learning (FL) is a novel paradigm for the shared training of models based on decentralized and private data. With respect to ethical guidelines, FL is promising regarding privacy, but needs to excel vis-\`a-vis transparency and trustworthiness. In particular, FL has to address the accountability of the parties involved and their adherence to rules, law and principles. We introduce AF^2 Framework, where we instrument FL with accountability by fusing verifiable claims with tamper-evident facts, into reproducible arguments. We build on AI FactSheets for instilling transparency and trustworthiness into the AI lifecycle and expand it to incorporate dynamic and nested facts, as well as complex model compositions in FL. Based on our approach, an auditor can validate, reproduce and certify a FL process. This can be directly applied in practice to address the challenges of AI engineering and ethics.

Artificial Intelligence and Digital Twins play an integral role in driving innovation in the domain of intelligent driving. Long short-term memory (LSTM) is a leading driver in the field of lane change prediction for manoeuvre anticipation. However, the decision-making process of such models is complex and non-transparent, hence reducing the trustworthiness of the smart solution. This work presents an innovative approach and a technical implementation for explaining lane change predictions of layer normalized LSTMs using Layer-wise Relevance Propagation (LRP). The core implementation includes consuming live data from a digital twin on a German highway, live predictions and explanations of lane changes by extending LRP to layer normalized LSTMs, and an interface for communicating and explaining the predictions to a human user. We aim to demonstrate faithful, understandable, and adaptable explanations of lane change prediction to increase the adoption and trustworthiness of AI systems that involve humans. Our research also emphases that explainability and state-of-the-art performance of ML models for manoeuvre anticipation go hand in hand without negatively affecting predictive effectiveness.

Machine learning offers promising capabilities to improve administrative procedures. At the same time, adequate training of models using traditional learning techniques requires the collection and storage of enough training data in a central place. Unfortunately, due to legislative and jurisdictional constraints, data in a central place is scarce and training a model becomes unfeasible. Against this backdrop, federated machine learning, a technique to collaboratively train models without transferring data to a centralized location, has been recently proposed. With each government entity keeping their data private, new applications that previously were impossible now can be a reality. In this paper, we demonstrate that accountability for the federated machine learning process becomes paramount to fully overcoming legislative and jurisdictional constraints. In particular, it ensures that all government entities' data are adequately included in the model and that evidence on fairness and reproducibility is curated towards trustworthiness. We also present an analysis framework suitable for governmental scenarios and illustrate its exemplary application for online citizen participation scenarios. We discuss our findings in terms of engineering and management implications: feasibility evaluation, general architecture, involved actors as well as verifiable claims for trustworthy machine learning.

Automated stress detection using physiological sensors is challenging due to inaccurate labeling and individual bias in the sensor data. Previous methods consider stress detection as a supervised classification task, where bad labeling leads to a large performance drop. Furthermore, the poor generalizability to unseen subjects reveals the importance of personalizing stress detection for both interand intra-individual sensor data variability. Towards this end we present a label-free feature extractor and an efficient personalization method with the ”human in the loop” approach. First, we capture the intra-individual variability and encode it in self-supervised learned features, which are usually well separable and independent of noisy stress labels. Next, personalization is achieved by assigning labels to critical reference points via very few interactions between subject and wearable device. The promising results of the conducted experiments show the effectiveness and efficiency of our proposed method.

Classifying stress in firefighters poses challenges, such as accurate personalized labeling, unobtrusive recording, and training of adequate models. Acquisition of labeled data and verification in cage mazes or during hot trainings is time consuming. Virtual Reality (VR) and Internet of Things (IoT) wearables provide new opportunities to create better stressors for firefighter missions through an immersive simulation. In this demo, we present a VR-based setup that enables to simulate firefighter missions to trigger and more easily record specific stress levels. The goal is to create labeled datasets for personalized multilevel stress detection models that include multiple biosignals, such as heart rate variability from electrocardiographic RR intervals. The multi-level stress setups can be configured, consisting of different levels of mental stressors. The demo shows how we established the recording of a baseline and virtual missions with varying challenge levels to create a personalized stress calibration.

Human Computer Interaction based on gestures offers enormous potential for designing ergonomic user interfaces in future smart environments. Although gestures can be perceived as very natural, the specific gesture set of a given dedicated interface might be complex and require some kind of self-description of expected body movements. We have therefore developed a machine-readable XML-based model of Labanotation, a camera-based movement analysis engine for automatic model creation, as well as a graphical editor for supporting manual design of gestures. In this paper, we present a tool for automatic generation of multimodal human-readable gesture documentation based on the XML-model. Currently, the tool supports text and 3D model animation and can be expanded to other modalities.

Elicitation studies are becoming recently popular methodology to investigate novel gestural interfaces. Yet, little is known about possible factors that may influence this type of studies. To our knowledge, this paper is the first to investigate the impact of physical impairments in full-body motion gesture elicitation studies. Our study was conducted with 20 healthy and 12 arm and/or hand impaired voluntary participants undergoing rehabilitation. In total, 1,707 gestures were logged, analyzed, and paired with think-aloud data for 27 referents performed with and without imposed physical constrains. Our findings supported by observational analyses aim to reveal the challenges to achieve a single canonical gesture set, the most popular strategies for defining gestures, the variation of physical body engagement with full-body motion gestures, and the tendency towards personalized and hybrid gestures. Our results add to few existing research papers aiming for better understanding of potential shortcomings of end-user interaction elicitation.

Intelligent Environments like Smart Homes consist of a variety of interconnected devices, which are increasingly controlled by NUI. However, provided interaction possibilities and functionality are often not obvious to users. Illustrating such scenario, we've developed an ambient light-control using two NUI devices for interaction. Building on this, four ambient manuals reflectively explaining available interaction possibilities and functionality on different output devices were developed in a participatory design process. With a total of 60 subjects all manuals were evaluated regarding workload.

Full body interactions are becoming increasingly important for Human-Computer Interaction (HCI) and very essential in thriving areas such as mobile applications, games and Ambient Assisted Living (AAL) solutions. While this enriches the design space of interactive applications in ubiquitous and pervasive environments, it dramatically increases the complexity of programming and customising such systems for end-users and non-professional interaction developers. This work addresses the growing need for simple ways to define, customise and handle user interactions by manageable means of demonstration and declaration. Our novel approach fosters the use of Labanotation (as one of the most popular movement description visual notations) and off-shelf motion capture technologies for interaction recoding, generation and analysis. This paper presents a novel reference implementation, called Ambient Movement Analysis Engine, to allow for recording movement scores and subscribing to events in Labanotation format from live motion data streams.

Dozens of novel natural interaction techniques are proposed every year to enrich interactive eco-systems with multitouch gestures, motion gestures, full body in motion, etc. We present a novel investigation of the community’s applied documentation practices for Natural User Interfaces (NUI). Our investigation includes analyzing a survey targeted at NUI designers and a large sample of recently published multitouch and motion-based interaction papers. To the best of our knowledge, this paper is the first to offer a close investigation of this kind. The results reveal that good NUI documentation practices are rare and largely compromised. Thus, we argue that engineering interactive systems for large-scale dynamic runtime deployment of existing interaction techniques is greatly challenged.

The increasing acceptance and innovation in Natural User Interfaces (NUIs) promise a widespread adoption of interactive systems following this paradigm. Although dozens of novel interaction techniques are being proposed every year, the currently applied approaches for designing and implementing NUI-based systems are greatly challenged. This workshop aims at outlining and discussing some of those emerging challenges based on four general research perspectives, namely large-scale and dynamic runtime deployment of interaction techniques; adequate long-term dissemination of interaction techniques; in-situ adaptation of interaction techniques; and dynamic interaction ensembles.

In the course of ubiquitous and pervasive computing a variety of smart devices are developed and entering our everyday life. These devices increasingly rely on novel interaction modalities from the field of Natural Interaction, such as gesture control. Common concepts to explain and illustrate devices’ interaction possibilities can’t be applied to these interaction techniques due to embedding of devices and as a consequence disappearing interfaces as well as distribution of functionalities among device ensembles in terms of IoT, AAL and Smart Home. These emerging and currently existing problems in accessing devices’ interaction possibilities present users with new challenges. In addition, current possibilities for device documentation provide only a limited viable option to learn devices. Hence, a general documentation for interconnected devices and thus functionality can not be created manually. In order to counteract these problems we present an approach for in-situ generation of an ambient manual for interconnected smart devices.

In this paper, we describe a distributed crowd-sensing infrastructure that integrates and bridges small scale personalized ad-hoc Internet of Things (IoT) spaces (consisting of personal interconnected smart devices, sensors and actuators dynamically deployed at runtime) to large scale IoT spaces. While a lot of innovation takes place on large scale IoT infrastructures, we focus on a personalized IoT infrastructure that allows user level control and management of personally owned IoT resources. Our approach uses a peer to peer (P2P) network together with distributed discovery- and directory-services, without the need for centralized infrastructure. The contribution of this paper is twofold: Firstly, we present an Android-based solution called Ambient Bridge that exposes a user-selected subset of the build-in sensors and actuators of a smart device as CoAP (Constrained Application Protocol) web services. Moreover, it is used to dynamically integrate external sensors and actuators at runtime that are normally only accessible via proprietary or non-networked interfaces. Secondly, we present a directory service and distributed semantic search engine called the Smart Service Proxy (SSP). The SSP allows application developers to search for sensors and actuators using SPARQL queries, which are automatically distributed between and processed by the cooperating SSPs.

Advances in Human Computer Interaction techniques continue to enrich Natural User Interface (NUI) research. While dozens of novel NUI interaction techniques are proposed every year, the potential of the human body’s sensory and motor systems is not yet fully utilized. Hence, new pressing calls have emerged for exploring the potential of the whole body in motion when interacting with real-world pervasive and ubiquitous computing ecosystems (ambient spaces). Given the adoption of NUI paradigm in ambient spaces, users will be increasingly expected to interact with multiple interactions techniques simultaneously. Whilst NUIs provide rich interaction possibilities and alternatives, they also introduce critical challenges for interactive ambient spaces. This dissertation aims to tackle three of these challenges; namely, large-scale dynamic runtime deployment of existing and future interaction techniques; long-term and adequate record-keeping and dissemination practices for interaction techniques; and in-situ adaptation of interaction possibilities. These challenges are often fueled by users’ increased mobility; the increasingly heterogeneity and availability of interaction resources; and the increasing diversity of the physical abilities of many user populations (e.g., elderly users). This dissertation presents a novel approach for adapting the interaction modalities available to a given application at runtime (as deployable interaction plugins). Accordingly, the capabilities and behaviour of an interactive system are optimized to fit the users’ physical abilities, needs, and context. The approach includes a theoretical concept (called Interaction Ensemble) that relies on decoupling the often tight binding between devices, interaction techniques, and applications. A reference implementation (called the STAGE framework) is presented as an evaluation of the concept.

Gestural interactions will continue to proliferate, enabling a wide range of possibilities to interact with mobile, pervasive, and ubiquitous environments. Particularly, motion gestures are getting an increasing attention amongst researchers. Likewise, a large adoption of motion gestures is noticeable on a commercial level. Motion gestures research strives to utilize the human body potential for interaction with interactive ecosystems. Despite the innovation and development in this field, we believe that describing motion gestures remains an unsolved challenge for the community to tackle and the effort in this direction is still limited. In our research, we focus on describing the human body movements for motion gestures based on movement description languages (particularly, Labanotation). In this paper, we argue that without adequate descriptions of gestural interactions, the engineering of interactive systems for large-scale dynamic runtime deployment of existing and future interaction techniques will be greatly challenged.

A large-scale dynamic runtime deployment of existing and future interaction techniques remains an enduring challenge for engineering real-world pervasive computing ecosystems (ambient spaces). The need for innovative engineering solutions to tackle this issue increases, due to the ever expanding landscape of novel natural interaction techniques proposed every year to enrich interactive eco-systems with multitouch gestures, motion gestures, full body in motion, etc. In this paper, we discuss the implementation of Interaction Plugins as a possible solution to address this challenge. The discussed approach enables interaction techniques to be constructed as standalone dynamically deployable objects in ambient spaces during runtime.

Light painting is a photographic technique, in which images are created using long exposures from a scene containing one or multiple light sources that move relative to the camera. We demonstrate an approach to encode and transmit information in such a painting via sequences of colored light blobs using smartphone devices.

The Web of Things (WoT) aims to extend the Web into the physical world by promoting the adoption of Web protocols by situated services and smart objects (ambient artifacts). However, real-world ambient artifacts often adopt proprietary and/or non-Web protocols, making them invisible to Web search engines and inaccessible to conventional Web agents. Smart Gateways have been proposed as a way to "Web-enable" proprietary ambient artifacts through intermediary proxy nodes, however, the requisite infrastructure is difficult to deploy at Web scale. To address such challenges, we are developing Ambient Dynamix (Dynamix): a plug-and-play context framework for mobile devices, which enables Web agents to interoperate with non-Web ambient artifacts - directly from the browser. In this paper, we describe how Dynamix can be used to transform the user's device into an ad-hoc Smart Gateway in-situ, enabling Web applications (in the device's browser) to seamlessly interact with non-Web ambient artifacts in the physical environment. We describe an operational prototype implementation, which enables Web apps to discover and control nearby UPnP and AirPlay media devices uniformly. We also present a performance evaluation that indicates the prototype imposes low processing and memory overhead, and is suitable for deployment on many commodity mobile devices.