Understanding everyday life behavior of young adults through personal devices, e.g., smartphones and smartwatches, is key for various applications, from enhancing the user experience in mobile apps to enabling appropriate interventions in digital health apps. Towards this goal, previous studies have relied on datasets combining passive sensor data with human-provided annotations or self-reports. However, many existing datasets are limited in scope, often focusing on specific countries primarily in the Global North, involving a small number of participants, or using a limited range of pre-processed sensors. These limitations restrict the ability to capture cross-country variations of human behavior, including the possibility of studying model generalization, and robustness. To address this gap, we introduce DiversityOne, a dataset which spans eight countries (China, Denmark, India, Italy, Mexico, Mongolia, Paraguay, and the United Kingdom) and includes data from 782 college students over four weeks. DiversityOne contains data from 26 smartphone sensor modalities and 350K+ self-reports. As of today, it is one of the largest and most diverse publicly available datasets, while featuring extensive demographic and psychosocial survey data. DiversityOne opens the possibility of studying important research problems in ubiquitous computing, particularly in domain adaptation and generalization across countries, all research areas so far largely underexplored because of the lack of adequate datasets.

In this third installment of our GenAI workshop series at DIS, we focus on 'stopsigns'-the blockages that impede progress in design research with GenAI. These stopsigns manifest as both semantic barriers (political, social, or mental frameworks) and pragmatic hurdles (technical limitations or implementation challenges) that persist despite the rapid advancements since the GenAI boom. Such stopsigns present a productive tension-they often contain partial truths worthy of consideration while simultaneously being shortsighted in ways that prevent progression. From blanket rejection to uncritical acceptance, these barriers affect how meaningfully we engage with GenAI's potential. Our workshop welcomes both returning and first-time participants to share their experiences with these persistent challenges and work together to develop practical solutions. Through analysis of real cases and hands-on activities,

Due to their architecture and how they are trained, artificial neural networks are typically not robust toward pruning or shuffling layers at test time. However, such properties would be desirable for different applications, such as distributed neural network architectures where the order of execution cannot be guaranteed or parts of the network can fail during inference. In this work, we address these issues through a number of training approaches for vision transformers whose most important component is randomizing the execution order of attention modules at training time. With our proposed approaches, vision transformers are capable to adapt to arbitrary layer execution orders at test time assuming one tolerates a reduction (about 20%) in accuracy at the same model size. We analyse the feature representations of our trained models as well as how each layer contributes to the models prediction based on its position during inference. Our analysis shows that layers learn to contribute differently based on their position in the network. Finally, we layer-prune our models at test time and find that their performance declines gracefully. Code available at https://github.com/matfrei/layershuffle.

Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being either unrecognizable or unrelated to the attacked prompts for humans. The existence of such images is problematic as it could be used by bad actors to maliciously interfere with CLIP-trained image retrieval models in production with comparably small effort as a single image can attack many different prompts. We demonstrate how fooling master images for CLIP (CLIPMasterPrints) can be mined using stochastic gradient descent, projected gradient descent, or blackbox optimization. Contrary to many common adversarial attacks, the blackbox optimization approach allows us to mine CLIPMasterPrints even when the weights of the model are not accessible. We investigate the properties of the mined images, and find that images trained on a small number of image captions generalize to a much larger number of semantically related captions. We evaluate possible mitigation strategies, where we increase the robustness of the model and introduce an approach to automatically detect CLIPMasterPrints to sanitize the input of vulnerable models. Finally, we find that vulnerability to CLIPMasterPrints is related to a modality gap in contrastive pre-trained multi-modal networks. Code available at https://github.com/matfrei/CLIPMasterPrints.

This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum's collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

Image-generation AI models have triggered a paradigm shift in how we can express ourselves in visual art. Despite their widespread use in a short amount of time, embedding these models into interactive artifacts is still largely unexplored. In this pictorial, we unpack the design and development process of GenFrame, an image generating picture frame that utilizes generative AI capabilities to mimic traditional paintings. Our work details the necessary steps to integrate generative AI into interactive artifacts and highlights important design considerations for controlling image-generation models in order to achieve specific design intents. Our insights provide interaction designers with a more comprehensive understanding and approach towards utilizing image-generation AI models for interactive artifacts. A demo can be viewed at https://youtu.be/1rhW4fazaBY

While diversity has become a debated issue in design, very little research exists on positive use-cases for diversity beyond scholarly criticism. The current work addresses this gap through the case of a diversity-aware chatbot, exploring what benefits a diversity-aware chatbot could bring to people and how do people interpret diversity when being presented with it. In this paper, we motivate a Q A chatbot as a technology probe and deploy it in two student communities within a study. During the study, we collected contextual data on people's expectations and perceptions when presented with diversity during the study. Our key findings show that people seek out others with shared niche interests, or their search is driven by exploration and inspiration when presented with diversity. Although interacting with chatbots is limited, participants found the engagement novel and interesting to motivate future research.

This one-day workshop seeks to bring together design researchers that have different orientations and practices for futures and futuring to unpack the role of futuring across design, design artefacts, and designers. Our goal in this workshop is to use examples of established futures to engage with and reflect on questions around the role of futures in HCI/design. We believe that there is a common thread between work from many different researchers, but that the people doing that work are spread across different disciplines and geographical locations. To that end, this workshop will assemble researchers working with different methods and approaches to futures in design and will aim to identify core challenges and opportunities for futures in HCI.

This workshop explores the transformative potential of generative artificial intelligence (GenAI) in design research. GenAI, capable of creating new content such as images, text, music, video, and code, raises important questions about authorship, agency, and design practice. Inspired by Roland Barthes’ "The Death of the Author," this workshop examines how GenAI reshapes design research roles and methods. Key topics include best practices, ethical considerations, knowledge generation, and collaboration patterns between human and AI creatives. Building on themes identified in the successful DIS 2023 workshop, this 2-day event invites designers and researchers to present completed projects, works-in-progress, and theoretical provocations. The structure allows time for both presentations and in-depth discussions, aiming to develop an online resource library and a collaborative publication. The workshop seeks to advance the discourse on GenAI, addressing its challenges and opportunities in design research.

GenFrame is an interactive picture frame that mimics traditional paintings while being equipped with generative AI capabilities. Painting “portraits of a girl” has been a trope in art history for centuries. However, in the era of generative AI models, the permanence of traditional art is challenged. When an AI model is able to mimic any painting style, it can also give agency to the viewer to repaint the painting per individual desires. In this installation, we modify the role of the museum placard to provide a minimal tangible interface to change the style of the image and the mood of the depicted girl. In this way, the generative AI model is interfaced in tangible ways, instead of the regular prompt paradigm. Demo video available here: https://youtu.be/ga7pFgAOPiY

Image-generation models are changing how we express ourselves in visual art. However, what people think of AI-generated art is still largely unexplored, especially compared to traditional art. In this paper, we present the design of an interactive research product, GenFrame - an image-generating picture frame that appears as a traditional painting but offers the viewer the agency to modify the depicted painting. In the current paper, we report on a study where we deployed the GenFrame in a traditional art museum and interviewed visitors about their views on AI art. When provoked by AI-generated art, people need more of the artist's backstory and emotional journey to make the artwork commensurate with traditional art. However, generative AI-enabled interactive experiences open new ways of engaging with art when a turn of a dial can modify art styles or motifs on a painting. A demo can be seen here: https://youtu.be/1rhW4fazaBY.

Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are increasingly gaining importance. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being unrecognizable for humans. We demonstrate how fooling master images can be mined by searching the latent space of generative models by means of an evolution strategy or stochastic gradient descent. We investigate the properties of the mined fooling master images, and find that images trained on a small number of image captions potentially generalize to a much larger number of semantically related captions. Further, we evaluate two possible mitigation strategies and find that vulnerability to fooling master examples is closely related to a modality gap in contrastive pre-trained multi-modal networks. From the perspective of vulnerability to off-manifold attacks, we therefore argue for the mitigation of modality gaps in CLIP and related multi-modal approaches. Source code and mined CLIPMasterPrints are available at https://github.com/matfrei/CLIPMasterPrints.

This one day workshop will explore the use of Generative Artificial Intelligence (GenAI) in design research and practice. Generative technologies are developing rapidly and many designers are using them. Yet, there remains little published work on the use of GenAI in design. Our goal is to not only showcase the potential of GenAI for design, but to engage in discussions of its shortcomings and opportunities as they have been already articulated by scholars. By synthesizing both published and unpublished works, we will develop best practices, ethical considerations, and future research directions for the use of GenAI in design. We will explore a range of topics and themes, including leveraging the characteristics of GenAI for design, mapping the diverse applications of GenAI in design, envisioning a framework for design, and guiding future work on GenAI in design research. Ultimately, we hope to provide a roadmap for the integration of GenAI into the design research process and to encourage designers and researchers to explore the potential of GenAI in a thoughtful and deliberate way.

Mood inference with mobile sensing data has been studied in ubicomp literature over the last decade. This inference enables context-aware and personalized user experiences in general mobile apps and valuable feedback and interventions in mobile health apps. However, even though model generalization issues have been highlighted in many studies, the focus has always been on improving the accuracies of models using different sensing modalities and machine learning techniques, with datasets collected in homogeneous populations. In contrast, less attention has been given to studying the performance of mood inference models to assess whether models generalize to new countries. In this study, we collected a mobile sensing dataset with 329K self-reports from 678 participants in eight countries (China, Denmark, India, Italy, Mexico, Mongolia, Paraguay, UK) to assess the effect of geographical diversity on mood inference models. We define and evaluate country-specific (trained and tested within a country), continent-specific (trained and tested within a continent), country-agnostic (tested on a country not seen on training data), and multi-country (trained and tested with multiple countries) approaches trained on sensor data for two mood inference tasks with population-level (non-personalized) and hybrid (partially personalized) models. We show that partially personalized country-specific models perform the best yielding area under the receiver operating characteristic curve (AUROC) scores of the range 0.78--0.98 for two-class (negative vs. positive valence) and 0.76--0.94 for three-class (negative vs. neutral vs. positive valence) inference. Further, with the country-agnostic approach, we show that models do not perform well compared to country-specific settings, even when models are partially personalized. We also show that continent-specific models outperform multi-country models in the case of Europe. Overall, we uncover generalization issues of mood inference models to new countries and how the geographical similarity of countries might impact mood inference.

Smartphones enable understanding human behavior with activity recognition to support people's daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study settings, making detecting simple activities less meaningful for context-aware applications. Hence, the understanding of (i) how multimodal smartphone sensors and machine learning models could be used to detect complex daily activities that can better inform about people's daily lives, and (ii) how models generalize to unseen countries, is limited. We analyzed in-the-wild smartphone data and ∼ 216K self-reports from 637 college students in five countries (Italy, Mongolia, UK, Denmark, Paraguay). Then, we defined a 12-class complex daily activity recognition task and evaluated the performance with different approaches. We found that even though the generic multi-country approach provided an AUROC of 0.70, the country-specific approach performed better with AUROC scores in [0.79-0.89]. We believe that research along the lines of diversity awareness is fundamental for advancing human behavior understanding through smartphones and machine learning, for more real-world utility across countries.

Emerging digital technology could enable communities to benefit from the diversity of their members mutually. This chapter explores the process of designing a chatbot application aiming to mediate people interaction through diversity-aware algorithms. In particular, we focus on designing a specific chatbot application that builds on the diversity of university students—the envisioned end-users—to accommodate their diverse needs and preferences while leveraging the diversity represented in their community.

The increasing availability of large-scale datasets such as sensor data or social media data and increasingly accessible data science tools create unique opportunities for design. However, the relationship between data science practices and design methods is still underdeveloped. In this paper, we propose that data exploration activities can be effectively embedded within a broader design inquiry framework and define a new design method, coined Data Exploration for Design, to support methodical designerly data exploration. The design method addresses the novice’s learning curve and supporting developing a data exploration inquiry mindset with procedures and curated tools. The empirical evaluation highlights support for producing exploration outcomes that are worth the additional technical effort. We close the paper by positioning the findings in design methodology literature and motivating data exploration principles for design inquiry.The principles urge to acknowledge biases in data collection, spending time with the data, using visualizations as a means-to-an-end, and designers being part of the data collection.

The current work investigates how creativity manifests when designers use data work in the early phase of design. Designers are increasingly interested in utilizing the massive amounts of data surrounding our everyday lives. However, data work is still challenging to incorporate into the design process. In this paper, we present a case study with three novice design teams who were tasked to integrate data work into their design process. During the study, we observed how creativity took place in framing a design problem. We present and discuss their actions from a creativity process perspective, highlighting how they used and rationalized data-inspired inquiries creatively in the early phase of design. The current results inform the development of a design framework to structure data work methodologically and coherently into design processes. We coin this design framework Exploratory Data Inquiry.

This workshop is concerned with the potentials and barriers of data in facilitating bottom-up innovation. We focus on two specific communities that could benefit from utilizing data, but may struggle currently. Firstly, civil society participating in grassroots actions, who use public data for innovation and to strengthen their advocacy. Secondly, organisations - especially SME's - who want to make use of data, especially large complex data sets, to design data-driven services.