This paper explores the design of an expressive visual instrument that embraces the unique movement style of a dancer living with physical disability. Through a collaboration between the dancer and an interaction designer/visual artist, the creative qualities of wearable devices for motion tracking are investigated, with emphasis on integrating the dancer’s specific movement capabilities with their creative goals. The affordances of this technology for imagining new forms of creative expression play a critical role in the design process. These themes are drawn together through an experiential performance which augments an improvised dance with an ephemeral real-time visualisation of the performer’s movements. Through practice-based research, the design, development and presentation of this performance work is examined as a ‘testbed’ for new ideas, allowing for the exploration of HCI concepts within a creative context. This paper outlines the creative process behind the development of the work, the insights derived from the practice-based research enquiry, and the role of movement technology in encouraging new ways of moving through creative expression.
This paper discusses how improvisation theatre can be used as a tool for design and (design) education in the field of Human-Robot Interaction. This tool has been explored in settings focused on design education for students as well as care professionals. In this paper, a format and set of guiding principles that were found to work constructively will be presented, as well as a discussion on outcomes, results, and insights from a number of sessions.
In the physical world, we as humans are used to consciously utilizing our space to help navigate our balance, directions, and intention. This virtual reality installation enables the the mover/ player to explore their visible relationships to space by using a Mind’s Eye perspective to playfully interject themselves in a virtual space.
Following three years of working on the eTube with a collaborative interdisciplinary team, we describe the embodied and spatialized performance gestures developed in tandem with a microphone setup for interacting with musical improvising agents. Eric Lewis’ discussion of the intentional stance and make-believe are outlined as a way to conceptualize our working process and engagement with musical agents in improvisation. Within this context, we consider the various agencies at play, and how the musical agents challenge notions of the social and embodiment in improvisation. We will then describe certain artistic approaches and results that are afforded by merging the philosophical with practice. A collaboration with other artists will illustrate new eTube performance gestures. Finally, we outline spatialization models designed for the musical agents and how these were developed in the context of the eTube performance practice.
The workshop explores how interactive technologies based on biophysical sensing and analysis can help create deeper connections with the surrounding world and one another, by combining the latest affect-indicating biosensors with somatic practices and creative expression. The workshop incorporates cutting edge biophysical sensing measurement technologies to explore new modes of artistic expression that directly gauge human affect as an integral component in the creation of an art work or performance. The goal is to explore how real-time tracking of a participant’s affective state can be used as an interactive modality in performance and computational art. Human affect data is gathered from a custom biophysical sensing acquisition and analysis system called, The Source. Data is accessed in real-time using custom software tools built on popular platforms such as Arduino, Max/MSP, SuperCollider, Ableton Live, TouchDesigner, and Processing. Physiological measures, including Electrocardiography (ECG, heart rate), Electrodermal Activity (EDA), Electromyography (EMG, muscle), Electroencephalography (EEG, brain waves), Electrooculography (EOG, eye movement), and Respiratory effort (RSP, breathing) can be incorporated. During the workshop, we will explore the potential for creative uses, as well as more applied contexts to support physical therapy, group training and experiences.
Motion capture proves to be a powerful tool in the creation of interactive dance-related applications since it supports the recording and capturing of movement data, giving rise to multiple applications, including visualizing movement, analyzing movement, reflecting on movement, and providing feedback. Amongst its various applications, motion capture has emerged as a valuable means of promoting and preserving diverse forms of folk and traditional dances, contributing significantly to the preservation of cultural heritage. Pair dances, where two individuals engage in coordinated movements often in close physical proximity, are commonly met in the dance landscape. Such dances demand a high degree of cooperation, precise timing, and effective communication between the participating dancers. The conventional method for motion capturing pair dances involves simultaneously capturing the movements of both performers. However, this approach may pose challenges, including the need for two motion capture suits or sets of markers, which could be impractical for some studios, artists, or researchers due to cost constraints. To address the issue of motion capturing pair dancing with a single suit, one solution is to sequentially record each dancer and then combine the animations digitally. However, no matter how synchronized or well-rehearsed the dancers are, this method may not produce entirely realistic results, especially during moments of body contact or complex moves like spins. In response to this specific motion capture challenge, our work presents an Extended Reality (XR) experiment exploring alternative methods for capturing pair dances and examining how Extended Reality (XR) can enhance the practice, learning, or performance of pair dancing. The experimental concept involves two dancers performing a choreography, with the movements of one dancer (the leader) being motion captured. These captured movements are then assigned to a digital character, visualized in realistic size through a Virtual or Augmented reality headset. Subsequently, the second dancer (the follower) wears the motion capture suit and the headset, dancing alongside the digital character visible through the headset, while their movements are recorded. Afterwards, the two produced animations are combined, visualized and analyzed in a 3D environment. This approach aims to achieve greater synchronization between the two animations, resulting in more natural movements. For the experiment, we employed three visualization modalities for the digital partner: HoloLens (Augmented Reality), Oculus Quest2 (Virtual Reality), and Oculus Quest 2 Pass-through mode (Augmented Reality). We applied this methodology to two dance case studies: the Greek dance "Ballos" and the well-known Waltz dance. Through interviews, questionnaires with professional dancers, and the observation and analysis of animations in a 3D environment, we explored the strengths, constraints, and challenges of this proposed methodology, contributing valuable insights to the research areas of motion capture and digital dance partnering.
This paper introduces a project enabling non-coders to control a Poppy Ergo Jr. robotic arm with Dynamixel servomotors. Originally using a Raspberry Pi and Pixl board, various constraints related to importation led to adopting a ROBOTIS OpenCM9.04 board. A client-server architecture was implemented for remote control, with creative coding platforms (p5.js, Processing, Pure Data, Python) as clients. The server, utilizing a two-layer architecture, manages communication and interfaces with the ROBOTIS OpenCM9.04 board. The OSC and WebSocket protocols were chosen for communication due to their flexibility and their ease of use. Clients were developed for each platform, leveraging compatibility layers.
This work presents the design and development of a methodology investigating the ways that data deriving from human-embodied social experience can be synchronized, curated, and organized to be provided to artists for inspiration and integration in their creations. Given the new media artists’ increased interest in utilizing biosignals in their art-making, we experiment with data collected through wearable technologies during sociodrama sessions -public discussions between residents of selected communities facing social issues- from the participants’ embodied experience. While we consider biosignals to provide valuable insight into the unspoken communication that took place, they are framed by descriptions of the locals’ perspectives, interactions, and behaviors as observed by the sessions’ facilitators. The present case study revolved around the city of Eleusis, Greece. Utilizing data previously recorded in [7] during sociodrama sessions dedicated to its environmental, employment, and migration crises, we present our proposed methodology, which is designed to a) execute biosignal analysis to discover more about the embodied aspects of social interaction while simultaneously employing the outcomes of a psychology team’s qualitative sociodrama analysis (study [7]), b) provide this material through an online platform in a way that accommodates artistic needs when using biosignals, as detected in work [5], c) onboarding a community of well established global artists to create audiovisual works inspired by and using the material, and d) evaluating the utilizability of the platform and its material. Focusing on the heart rate, body temperature, and skin conductance of the participants, we start detailing the steps of data collection, extraction, and curation. These processes resulted in the detection of moments of intense biometric activity in the data, which we defined as episodes, and which served as the main structural unit of the data representation. Then, we describe the design of TransitionTo8 platform which organizes, presents, frames, and augments the collected data through descriptions of the social interactions and discussions that took place during the sociodrama sessions -the episodes’ transcripts, additional summaries, and commentary-, visualizations and sonifications. Next, we present the outcome of this methodology; artistic works created using the platform material and presented to the local community that inspired them in the form of a multimedia festival. Finally, we discuss whether our goal of distributing the curated data in a way that is meaningful and inspiring, enabling the connection between the sociodrama participant, the narratives that emerged during the sessions, and their embodied aspects, with artists from all over the world, was achieved.
Fitness video observation is a common approach in sports practice. However, in videos, important frames and others are presented at a constant speed, there is a substantial cognitive load in accurately capturing key movements and timing, especially in the quick-paced videos of complex activities such as dance. We hypothesize that extracting keyframes from dance videos and replaying them in sync with rhythm (frame-by-frame presentation) can reduce this cognitive load and improve the dance technique execution. Our first study using a 2D display suggests that frame-by-frame presentation is not only as preferred as conventional videos, but also enables more accurate learning of movements. Based on that, we developed DRF, a VR application that combines frame-by-frame presentation with motion trajectory visualization. User study results indicate that with DRF, users could significantly improve both choreographic dance technique and rhythm accuracy compared to video-based VR systems. Qualitative user evaluations from beginners, experienced dancers, and professionals expressed the benefits and potential use of the frame-by-frame presentation method.
Message Bank is a performance made for audiences observing and navigating a public square. The performance tells the story of Charlie who, tasked with the mission of solving a small-scale crime, encounters an unidentified agent with the alias of ‘Dancing Bear.’ The audience takes on the role of Charlie and is asked to choose between the promise of safety with a state-based surveillance agency or the potential for disruption with a radical libertarian collective. Message Bank premiered in Parramatta Square as part of Sydney Festival 2023. The Sydney Festival season is referred to here as the first iteration of the performance and can be framed in the context of digital theatre [1], digital performance [2], and locative media [3]. Early examples of creative practice in this area include Teri Rueb's Trace, (1999) [4], Blast Theory's, Riderspoke, (2007) [5] and RATS Theatre's Maryam (2013) by Rebecca Forsberg [6]. More recently and locally to Australia; Leah Barclay's WIRA (2015) [7] offered audiences a geolocated audio walk along the Noosa River, Claudia Chidiac's The Village by the Kids (2022) [7] provided a neighborhood storytelling tour co-created with young people living in Bondi and Malthouse Theatre's Hour of the Wolf (2023) [9] used FM transmission to facilitate an immersive multi-room murder mystery within the interior of a theatre. The form of these works’ ranges from installation to digital performance and digital theatre but each are similar in their attempts to make meaning between place, story and an audience in motion. Message Bank builds on the practice-based knowledge generated through these works and contributes further insights to how they might enliven public spaces. The use of the term enlivening here is inspired by Auslander's work on liveness [6] as it relates to spontaneity, community, feedback, and presence. The Message Bank team consists of a director / researcher, two additional writers, a digital artist, designer, composer, outside eye, production observer, creative producer, and actors. The team collaborated intermittently over a period of five months in which the project was conceived, workshopped, drafted through script and application development, recorded as audio and video extracts, and tested through prototypes. Discoveries from the first iteration in the areas of motion, interaction, co-present audiences, and the role of time have fed into further development of the performance leading to the second iteration of Message Bank presented at MOCO. This second iteration maintains the vision while further exploring the performances capacity for linking audiences together through handheld devices and a mix of real-time spatiotemporal data to enliven public squares. Working between digital theatre, digital performance, and locative media the creative team is engaged in an ongoing process of iteration that mixes dramaturgy and technical experimentation. This manifests through the ongoing development of Message Bank's narrative and the design of the accompanying software which facilitates the experience. The second iteration presented at MOCO has a specific focus on how the performance binds co-present audiences both intended and unintended as well as how it connects motion data to the revelation of story beats.
This research introduces an approach for translating traditional dance knowledge into interactive computational models extending beyond static dance performance recordings. Specifically, this paper presents the concept of "Human-AI co-dancing," which involves integrating human dancers with virtual dance partners powered by models derived from dance principles. To demonstrate this concept, the research focuses on the choreographic principles deconstructed from the knowledge of traditional Thai dance. The principles are analyzed and translated into computational procedures that dynamically manipulate the movements of a virtual character by altering animation keyframes and the motions of individual joints in real-time. We developed an interactive system that enables dancers to improvise alongside the virtual agent. The system incorporates voice control functionality, allowing the dancer, choreographer, and even the audience to participate in altering the choreography of the virtual agents by adjusting parameters that represent traditional Thai dance elements. Human-AI rehearsals yielded intriguing artistic results, with hybrid movement aesthetics emerging from the synergy and friction between humans and machines. The resulting dance production, "Cyber Subin," demonstrates the potential of combining intangible cultural heritage, intelligent technology, and posthuman choreography to expand artistic expression and preserve traditional wisdom in a contemporary context.
Artists have been progressively blurring the boundaries between audience and performer, particularly in technology-mediated performing arts. However, there is scarce literature systematising these approaches in the field of dance. This leads to our research question: What are the challenges with audience interaction in technology-mediated dance performances and what can be done to overcome them? To answer this question, we ran a focus group with 10 artists in the field of contemporary dance, with relevant and diverse experience. The analysis of the focus group allowed us to propose best practices for the design of audience interaction in technology-mediated dance performances. We also discuss these recommendations in light of existing literature.
Experiments with animating human bodies on stage are reported here. In our performances, animations are carried out by augmenting human motions, coercing actions, experiencing disruptions of realism, shifting kinesthetics, or altering anatomies. These activities are performed on the participants' bodies using mechatronic devices such as exoskeletons, supernumerary limbs, or symbiotic apparatuses. We adapt motion capture (mocap) techniques and data gathered from bodies, crowds, and devices to control, augment, and alter human movements as if we were animating 3D virtual characters, yet with the inevitable constraints of the physical world and human body integrity. In a constant shift of locus of perception, through 'ghost' control and gestural doubles, human volition is destabilized and turns into an uncanny paradox of pleasure and loss of self-control.
This paper presents a device that transforms a skate’s sound and motion into a real-time auditory tool for figure skaters. Inspired by the skating tradition of “figures,” the Motion Augmented Acoustic Skate system enhances a skate’s existing acoustic feedback for non-jumping implementations. Using a contact microphone, an Inertial Measurement Unit (IMU), a radio transmitter, and a Wi-Fi microcontroller, the system wirelessly transmits sound and motion to an off-ice computer for signal processing. The resulting sonification output, altered through physical modeling and motion mappings, transmits back to the skater in real-time. The design and implementation of the Motion Augmented Acoustic Skate (MAAS) prototype is discussed as a future artistic, athletic, and somatic tool.
With What? represents an unfolding collective discourse around technology as ancestry. It situates a socially engaged virtual reality (VR) experience within a lightweight visual art installation that introduces a writing prompt and crafting station. The project asks what we carry across borders and through time. Participants answer through written responses and by exploring ritualized paths across physical and digital spaces. The custom VR landscape, developed in Unity, introduces an experimental archive of familial objects, femme figures, and audio recordings gathered through creative workshops and residencies. Participants traverse the VR world by following predetermined pathways and discovering artifacts as they travel. This novel interaction design invites kinesthetically grounded inquiry, with viewers collecting visual, tactile, and contextual insights via improvisational methods. The project considers how socially engaged VR may highlight artifacts of experience, reflecting our uniquely human vernacular.
Somaesthetic experiential qualities can provide a window into process of meaning-making, both human and machinic. We draw such qualities from viola performance into the design-in-progress of a novel interactive performance system. In doing so, we introduce the concept of a Machine Somaesthete that senses and makes sense of these qualities from a second-person perspective. Our system comprises electromyographic (EMG) muscle sensing and a Variational Autoencoder. With a novel dataset, we aim to encode latent representations of performance movement that are meaningful from a somaesthetic perspective. We present our model and our design process, then analyse latent trajectories to interrogate how our system can be considered a Machine Somaesthete, and the nature of its sensitivity to bodily experiences of viola playing. At the intersection of artificial intelligence, music performance and intra-action design, we take a sympoietic (together-making) view of knowledge creation. We and our practices are transformed as we design - and design with - machine learning systems.
It takes less than half a second for a person to fall [8]. Capturing the essence of a fall from video or motion capture is difficult. More generally, generating realistic 3D human body motions from motion capture (MoCap) data is a significant challenge with potential applications in animation, gaming, and robotics. Current motion datasets contain single-labeled activities, which lack fine-grained control over the motion, particularly for actions as sparse, dynamic, and complex as falling. This work introduces a novel human falling dataset and a learned multi-branch, Attribute-Conditioned Variational Autoencoder model to generate novel falls. Our unique dataset introduces a new ontology of the motion into three phases: Impact, Glitch, and Fall. Each branch of the model learns each phase separately and the fusion layer learns to fuse the latent space together. Furthermore, we present encompassing data augmentation techniques and an inter-phase smoothness loss for natural plausible motion generation. We successfully generated high quality images, validating the efficacy of our model in producing high-fidelity, attribute-conditioned human movements.
In recent years, significant advances have been made in deep learning models for audio generation, offering promising tools for musical creation. In this work, we investigate the use of deep audio generative models in interactive dance/music performance. We adopted a performance-led research design approach, establishing an art-research collaboration between a researcher/musician and a dancer. First, we describe our motion-sound interactive system integrating deep audio generative model and propose three methods for embodied exploration of deep latent spaces. Then, we detail the creative process for building the performance centered on the co-design of the system. Finally, we report feedback from the dancer’s interviews and discuss the results and perspectives. The code implementation is publicly available on our github1.
As robot sensors, mechanisms, and artificial intelligence algorithms increase in capability and availability, the performing arts has incorporated robots into different creative works. These performances often focus on humans and robots as separately controlled entities that interact at specific points in time. In this paper, we propose, implement, and evaluate a human-robot teaming platform that enables the live choreography of human-robot teams that work together within an improvisational performance context. This platform unifies robots and humans using a novel integration of programming abstractions, wireless haptics, and autonomous behaviors that provide a foundation to construct and manage non-dyadic human-robot teams. Furthermore, we demonstrate this platform in a live human-robot choreographic experience with close audience interaction. We use this performance to gather qualitative data about audience sentiment and performer experience that informs how these human-robot teams might be perceived when operating in close proximity.
The challenge of simulating realistic Sign Language using avatars lies in achieving accurate human-like postures for effective communication. Unlike artistic or motion capture techniques, linguist-driven procedural generation methods are widely employed, relying on skeletal representations to synthesize a broad range of signs. However, determining appropriate joint limits for these avatars is intricate due to inter-joint and intra-joint dependencies, as well as variations in biomechanical properties. In this context, our work addresses this problem by introducing a pose corrector, enhancing an established Sign Language synthesis technique. Focused on rectifying extreme joint rotations, our approach incorporates a pre-trained poser based on existing work, integrated with a 21-joint character model. The correction process involves applying linguist-defined constraints using AZee language and subsequent pose corrections, showcasing promising advancements in obtaining more natural sign gestures.
