When humans collaborate on a shared task, they use a myriad of verbal, para-verbal and non-verbal cues to achieve this end. Modern Artificial Intelligence sensing techniques such as Social Signals Processing (SSP) allow the characterization of users’ activities in Augmented or Virtual Environments through the analysis of heterogeneous multimodal data sources, such as interaction actions, gaze direction, avatars’ positions, and speech analysis. Therefore, this enables realtime assessment of team processes, including communication or team situation awareness. In return, it can provide context-specific feedback and information to augment team capabilities. However, implementing this vision remains a technical challenge, particularly in gathering real-time data sources in multi-user collaborative scenarios. This paper presents a framework for assessing and augmenting team collaboration with Extended Reality Environments (XRE). We propose a multimodal architecture to capture users’ activities and augment the team capabilities by displaying system-generated context-specific collaborative cues. Our proposed vision, System As A Collaborator (SAAC), frames the XRE as a direct actor embedded in the collaborative activity, improving team members’ capabilities with collaborative augmentations instead of being merely the space where collaboration occurs. We demonstrate the feasibility of our approach and system architecture through use cases of experimental XRE where our software architecture collects multimodal, heterogeneous, and multi-user interaction, behavioral and physiological data, and generates direct, reactive cues, and higher-level context-specific cues, augmenting team collaboration.

Immersive realities enable social interactions that are radically different from traditional communication technologies, but how we experience immersion together is not yet indistinguishable from face-to-face interactions. Some social signals are not stable across realities, may change in semantics, or are missing all together. Understanding how social signals impact behaviours and experiences of social connection in immersive environments is key to creating experiences that are meaningful, satisfying, and productive. We completed a lab study where 6 groups of 6 participants (N=36) completed co-located social tasks in an instrumented face-to-face environment and its digital twin, creating a rich open dataset of 1.8 million rows across 45 columns. Our quantitative results demonstrate the stability of position as a social signal, measure lower social synchronisation in XR compared to face-to-face, and propose a method for bench marking XR against face-to-face interactions. This enables direct quantitative comparisons between experiences of co-located physical and virtual interactions for the first time.

In high-stakes collaborative situations, a decline in collaboration quality can lead to adverse events with significant consequences. Analyses performed by Human factor (HF) specialists, while effective in identifying and addressing collaboration issues, are case-specific and most of the time performed a posteriori. To address these limitations, our research focuses on a real-time assessment of collaboration processes using multimodal signals collected and analyzed during the activity. Existing collaboration profiles taxonomies face limitations such as a posteriori profiles detection and the absence of quantitative behavioral indicators that can be measured during the activity. Leveraging Virtual Reality (VR), we have developed a framework for evaluating collaboration in controlled setting, testing the effectiveness of a subset of multimodal signals to detect collaboration profiles. We test our approach in a study including 11 stereotyped collaborative scenarios applied to a VR puzzle-solving task. This study reveals the effectiveness of our approach in distinguishing between non-collaborative and highly collaborative profiles. However, challenges arise in discriminating between closely related collaborative profiles. This paper also proposes some guidelines on how to improve the collaboration profile detection framework and address other collaborative situations.

Effective collaboration is essential in high-stakes environments where poor teamwork can lead to critical errors and adverse outcomes. This Work in Progress aims to contribute to research by providing real-time feedback to prevent critical situations arising from inadequate collaboration. We are developing an experiment to compare seven indicators of collaboration for their effectiveness in real-time context. Using a collaborative virtual environment, we can control the situation and environmental effects, allowing for precise and reliable assessment of each indicator. The goal is to identify the most efficient indicators for real-time assessment of collaboration, thereby enhancing team performance and preventing critical failures. This research will contribute to optimizing teamwork and operational success in critical fields, such as industrial applications, where collaboration is crucial.

In high-stakes collaborative situations, a decline in collaboration quality can lead to significant adverse events. Although human factors (HF) specialists are effective in identifying and resolving collaboration issues, their analyses are often case-specific and conducted retrospectively. To address these limitations, our research aims to assess collaboration processes in real-time using multimodal signals collected and analyzed during the activity. In this context, we propose a platform that integrates audio, visual, task logs, and spatial metrics computed along the activity, to provide an assessment of various dimensions of the collaboration process. This approach is applied to a triadic collaborative puzzle-solving task in virtual reality.

Collaboration during the completion of complex tasks requires synchronization and effective communication within teams. In this paper, we study multimodal collaboration metrics with the intention of designing collaboration support systems that enable the evaluation of collaboration quality. The goal is to provide real-time feedback and prevent the emergence of critical situations due to poor collaboration. We use a collaborative virtual environment to control the situation and the effects of the environment. As a first step to measure collaboration, we present a work in progress that aims to test the effectiveness of verbal and gaze measurements as an indicator of collaboration quality. To do so, an assembly task was performed by twelve dyads in our virtual environment. A tool was designed to process activity inputs in real time. Our findings show that verbal and gaze metrics can provide feedback on collaboration in virtual environment. Using these measures together with others would provide more accurate feedback on collaboration. Lastly, suggestions for improving the tool and an interest for other collaboration indicators are raised.

The use of robotic surgical systems creates new team dynamics in operating rooms and constitutes a major challenge for the development of crucial non-technical skills such as situation awareness (SA). Techniques for assessing SA mostly rely on subjective assessments, observation or interviews; few utilize multimodal measures that combine physiological, behavioural, and subjective indicators. We proposed a conceptual model relating situation awareness with mental workload (MW), stress and communication. To validate this model, we collected subjective feedback, measurable behaviours and physiological signals from surgeons performing a robot-assisted radical prostatectomy procedure. Preliminary results suggest that subjective MW is a better indicator of SA than subjective stress. Physiological measures did not correlate with subjective measures of stress and MW. Results also suggest that some indicators of communication quality associated with various levels of SA tend to be linked with surgical complexity.

The use of robotic surgical systems disrupts existing team dynamics inside operating rooms and constitutes a major challenge for the development of crucial non-technical skills such as situation awareness (SA). Techniques for assessing SA mostly rely on subjective assessments and questionnaires; few leverage multimodal measures combining physiological, behavioural, and subjective indicators. We propose a conceptual model relating SA with mental workload, stress and communication, supported by measurable behaviours and physiological signals. To validate this model, we collect subjective, behavioural, and physiological data from surgical teams performing radical prostatectomy using robotic surgical systems. Statistical analyses will be performed to establish relationships between SA, subjective assessment of stress and mental workload, communication processes, and the surgeons' physiological signals.