In recent years, there has been a growing interest in employing intelligent agents in writing. Previous work emphasizes the evaluation of the quality of end product—whether it was coherent and polished, overlooking the journey that led to the product, which is an invaluable dimension of the creative process. To understand how to recognize human efforts in co-writing with intelligent writing systems, we adapt Flower and Hayes’ cognitive process theory of writing and propose CoCo Matrix, a two-dimensional taxonomy of entropy and information gain, to depict the new human-agent co-writing model. We define four quadrants and situate thirty-four published systems within the taxonomy. Our research found that low entropy and high information gain systems are under-explored, yet offer promising future directions in writing tasks that benefit from the agent’s divergent planning and the human’s focused translation. CoCo Matrix, not only categorizes different writing systems but also deepens our understanding of the cognitive processes in human-agent co-writing. By analyzing minimal changes in the writing process, CoCo Matrix serves as a proxy for the writer’s mental model, allowing writers to reflect on their contributions. This reflection is facilitated through the measured metrics of information gain and entropy, which provide insights irrespective of the writing system used.
This paper discusses the digital animation artwork Motion and Trails, which builds on previous explorations of procedural techniques to generate unexpected visual results. Originating from animation processes that require quick workflows by embracing procedural methods, Motion and Trails continues this approach for the 2024 Creativity and Cognition Conference art exhibition by using simulated techniques in Maya and After Effects to transform 3D forms. Considering the theme of Creative Organic Spaces, the artwork aims to tap into creative possibilities through an informed but ‘unknown outcome’ approach to art making. As the latest iteration in an evolving organic process, Motion and Trails focuses animation as an innovative medium while demonstrating its immersive and cinematic capabilities. By utilizing procedural animation to embrace the unknown, this work emphasizes digital practice and abstraction to reveal imaginative new ways of creating virtual spaces.
This paper presents a novel 3D system for human motion analysis - Motion Data Visualization and Annotation (MoViAn). Designed to provide a comprehensive visual representation of 3D human motion data, MoViAn incorporates detailed visualization of gaze direction, hand movements, and object interactions, alongside an interactive interface for efficient data annotation. A user study involving eight participants indicates that MoViAn enables users to thoroughly explore and annotate human motion data, with System Usability Scale (SUS) results demonstrating a satisfactory usability level. The contribution of this paper lies in the development of an interactive and usable data analytics tool aimed at deepening the understanding of human behaviors and intentions in various creative, cognitive, and physical activities that ultimately can facilitate the design and creation of innovative tools that enhance human life in multiple domains.
How do we interpret a multi-participant choreographed performance in the public domain through digital technologies? In collaboration with data visualisation expert David Hunter from University of Colorado at Boulder, and visual artist Zach Duer from Virginia Tech, dAnCing LiNes explores how dance can generate a choreographic view of drawing through mediated representation. In this respect the artwork produced for dAnCing LiNes is not intended as a means of documentation of the live events but as a tool for new artistic production. The intention is to rethink performative drawing beyond the gestural trace of the body in movement through the use of data visualisations. Capturing chorographic scores and task-based instructions through digital technologies, the data visualisations explore how the agency of dance moves from the performative to the visual via technological means by using combinations of established computer vision techniques from OpenCV [1] like Optical Flow, Blob Detection. The visualisations not only reveal the rules of the underlying choreography in each location but also computationally play with and exemplify those rules on a per location basis (five in total).
Our most cutting-edge virtual reality remains focused on visual perception over embodiment. This article presents a dance performance project that stems from a foundation in somatics, creative dance, and HCI to embrace the challenge of centering bodies and embodiment within virtual reality. Our team of choreographers, dancers, and technologists recount the two-year development of the 360 video through ideation, movement tracking workshops, prototyping, process share-outs, to a final performance, to reflect on how our goal to centre bodies guided our artistic and technology choices throughout the entire process of making. Space became a key theme as we refined the choreography and design to use the full potentialities of the immersive environment. Place became a key theme as we prioritised configurations that enabled the performance to be showcased in the Pacific, Eastern Europe and Latin America. We discuss challenges and opportunities for supporting embodiment in virtual reality dance performance.
Existing literature on designing AI technologies focuses on personifying the AI system using metaphors like "collaborator" or "teammate." However, these personifying metaphors may have problematic consequences and do not reflect the full breadth of the design space for human-AI interaction. We report the design process of, and initial results from, a participatory speculative design workshop that supports participants in designing AI systems using novel, non-human metaphors. We articulate two current design recommendations for interaction or workshop designers interested in AI and metaphors: Defamiliarization of AI may support creative interpretation of ambiguous metaphors and Explicit metaphorical thinking may scaffold participants’ use of ambiguity as a resource. We also discuss two potential results suggested by our initial data collection: Ambiguous metaphorical thinking may support groups negotiating values and The term AI may limit attempts at defamiliarization.
A Space Resonating with Being is a visual-audio interactive artwork where the visual art on a 55-inch display changes in response to external intervention, focusing on experiencing the space as a living organism. The real-time visual effects that appear on the display reflect the local natural environment when there is no viewer interaction, symbolically integrating the natural environment into a space. When the viewer interacts with the artwork, it is transformed by an algorithm based on physiological signals from the viewer. The artwork changes based on their biorhythms, such as their breathing and heart rate, and they enjoy a unique experience where the art space responds to them, providing a dynamic and colorful visual-audio experience. The artwork draws attention to the impact of the viewer on space and how artwork affects the viewer’s internal world perception through the experience of the artwork gradually morphing to resonate with them.
The term live coding can refer either to a performative musical practice or to a lecture technique in the CS classroom. These two disparate practices are united in that they can be thought of as code performances, where a performer writes, edits, and runs code live in front of an audience. In my research, I aim to better understand code performances and how we can support them. In a recent project, I explored how a version control system could benefit live coding music. I found that backtracking affordances could facilitate incorporating musical form on the fly and that version trees could act as a visual aid for both the performer and the audience. In another ongoing project, I am exploring how to support live coding in the CS classroom, focusing on increasing audience learning and engagement. In the future, I plan to explore how to best represent code performances so performers can iterate on their own ideas more easily, as well as share and collaborate with others.
Welcome to a Skeuomorphic World critically explores the economic implications of substituting organic elements in architecture with synthetic materials, focusing on faux wood flooring for its cost-effective and low-maintenance qualities. The project presents a gradual skeuomorphic depiction of a wooden floor plate, translating a live organic wood log image into 880 abstract 12x12 units. These units are slowly showcased on a sizable LED panel in sequence. Using rephotographs and juxtaposition, the work satirically portrays an organic wood surface in an interactive format, prompting a reconsideration of perceptions surrounding synthetic materiality.
This doctoral research aims to to design interactive Explainable AI (XAI) tools in response to the challenge of fostering AI literacy among adults without technical expertise. The tools developed thus far focus on edge detection, confidence thresholds, and sensitivity, and were designed based on principles from learning sciences and user-centered design to make AI accessible and ethically reflective. My efforts have resulted in successful design, implementation, and preliminary evaluation. Conducted with 42 adult participants, the study reveals notable improvements in familiarity with and confidence in discussing AI concepts. Qualitative feedback highlights user engagement and enhanced understanding, demonstrating immediate impact and laying a foundation for ongoing work. Looking forward, my work will delve deeper into how non-experts can critically engage with AI’s decision-making processes, understand algorithmic trade-offs, and consider how AI can better serve society. This approach broadens AI literacy and leverages cognitive principles for creative and ethical technological interventions.
Through this poster submission I present my ongoing investigation into the dynamics of interpersonal relationships within online fandom communities, specifically focusing on the BTS fandom community known as ARMY. Amid the digital era’s paradox of increased content creation and consumption alongside declining meaningful social interactions, I hope to identify online fandom communities as unique spaces where interpersonal relationships not only thrive but contribute to the sustainability of these digital communities. Using a case study approach on the ARMY fandom, which expanded during the pandemic, my research aims to dissect the nuances of relationship formation and sustainability. Through a combination of literature review, online diary studies, interviews, and digital ethnography, I explore how fans engage, and support each other, nurturing a sense of community. Initial findings highlight the interplay between personal and collective identity formation, the supportive role of community spaces, the impact of social support on well-being, and a profound sense of belonging among members. This study offers valuable perspectives on designing digital spaces that foster strong social connections.
Deep learning generative AI models trained on huge datasets are capable of producing complex and high quality music. However, there are few studies of how AI Generated Content (AIGC) is actually used or appropriated in creative practice. We present two first-person accounts by musician-researchers of explorations of an interactive generative AI system trained on Irish Folk music. The AI is intentionally used by musicians from incongruous genres of Punk and Glitch to explore questions of how the model is appropriated into creative practice and how it changes creative practice when used outside of its intended genre. Reflections on the first-person accounts highlight issues of control, ambiguity, trust, and filtering AIGC. The accounts also highlight the role of AI as an audience and critic and how the musicians’ practice changed in response to the AIGC. We suggest that our incongruous approach may help to foreground the creative work and frictions in human-AI creative practice.
As young people increasingly use AI in their daily lives, it is imperative to foster these learners' AI literacy. We present Knowledge Net, a collaborative tangible tabletop museum exhibit aimed at teaching users about knowledge representations, which are central to understanding AI and understudied in AI education research. In this exhibit, we center creative making and embodied interaction by allowing learners to craft the appearance, behaviors, and traits of characters in a virtual world by manipulating semantic networks. Our poster features the exhibit design and corresponding rationale, and this paper contributes an exploration of how creative making and embodied interaction can be utilized to teach young learners about knowledge representations-and AI more broadly-in informal learning environments.
The creation of 3D content, crucial in various applications, is often challenging and time-intensive. While digital tools are prevalent for 3D content creation, traditional clay sculpting offers an embodied experience that fosters artists’ perceptual engagement with physical space, enhancing their interactive and cognitive connection with the creation process. We conducted an eight-day live sculpting session at an art academy, systematically comparing the creative workflows of eight professional artists in both physical and digital mediums. Our qualitative and quantitative analysis include artists’ differences in tool usage between physical and digital sculpting, variations in visual and tactile perceptual engagement, and the potential for future integration of the two modalities. Our study provides insights into the benefits of physical and digital sculpting and may inform future design of hybrid interfaces for 3D content creation.
Advances in AI have opened up the potential for creativity tools to computationally generate design feedback. In a future when designers can request feedback anytime on demand, how would the timing of these requests impact novices’ creative learning processes? What are the tradeoffs of providing access to feedback throughout a design task (in-action) versus only providing feedback after (on-action)? We explored these questions through a Wizard-of-Oz study (N=20) using an interactive design probe, where participants could request feedback either throughout the design process or only after they complete a full draft. We found that in-action participants frequently request feedback, resulting in better improvements as indicated by a greater decrease in issues in their final design. However, we saw that in-action feedback can also risk users overly relying on feedback instead of engaging in more holistic self-evaluation. We discuss the implications of our insights on designing tools for creative feedback.
An empirical record of glitch practice is portrayed in this pictorial. In wandering through the information environments of digital files, researchers can discover unexpected artifacts through experimental interactions. The production of these images span about seven years and range across various digital media, such as photography, video footage, 3D renders, and gaming. This work expands the concept of organic architecture to the realm of the digital medium. Inspired by nature, analogies from physical phenomena and scientific principles are used to inform the research-creation. In doing so, glitch experiments in digital files enable a cocreativity with technology which reveal their underlying structures and thereby produce a spectrum of expression. Often, uncanny resemblance to natural terrains manifest, indicating a connection through possibilities of configuration, or entropy.
The scientific process is inherently creative, requiring the generation and exploration of ideas for scientific inspiration, projects, study design, and communication. As large language models (LLMs) advance rapidly, scientists increasingly take advantage of their abilities. While LLMs show great promise in supporting many steps of the scientific process, researchers still face significant challenges in validating and steering their output. Interactions tailored to scientists and their specific tasks may empower them to harness the full creative potential of LLMs. I present a course of research that will lead to the development and evaluation of mixed-initiative methods for co-creation in scientific research. These methods aim to facilitate verification and control of AI output. I briefly describe my prior and proposed work on mixed-initiative methods for co-creating research inspiration, studies, and communication, and I detail my current project on an LLM-powered tool for co-creating research project ideas.
"Mosaic Immersion" is a mixed-reality installation that interweaves physical and virtual experiences in a shared space. Participants explore interconnected narratives through uniquely created rock objects with their hand images. Mosaic Immersion was inspired by cairn stacking, along hiking trails. Cairns, which are created by carefully balancing rocks on top of one another, serve as markers or symbols along the trail, guiding and providing a sense of connection for hikers. It is a practice that carries both cultural and personal significance. These rock formations create a shared experience, inviting others to contribute their own stones or simply admire the collective effort. Within this space, the boundaries between physical and virtual realms blur, inviting profound explorations of resilience and connection in the human experience.
Representational forms are central to how we explore, communicate, and learn. Yet, they can be challenging to engage with as designers because they vary across disciplines, cultures, and communities. In this paper, we describe our analysis of a dance and computing learning environment through the lens of distributed cognition to examine how representations and processing of information across people and systems impacted the learning process. We analyzed video and audio data from three workshops of dance and STEM instructors learning about, creating with, and co-designing computing activities with danceON—a creative computing platform that supports coding animations over dance videos. We identified the ways that the instructors used their bodies as a shared point of negotiation while co-creating dance artifacts, the participatory role of danceON within the sensemaking process, the ways that the instructors interpreted and translated representations across physical and digital spaces, and the impact of the instructors’ collaborative interactions to their sensemaking.
