As AI-assisted coding becomes standard in software development, computer science educators need a clearer understanding of how Large Language Models (LLMs) can support the learning process. Recent work has examined how students can benefit from using LLMs in their courses, but most studies rely on self-reported usage or controlled experiments with short, isolated programming tasks. To complement these approaches, this paper investigates how students organically leverage LLMs in an advanced computer science course where assignments reflect real-world complexity. We analyze 448 LLM chat logs from 147 students across two offerings of a senior-level web programming course at a large U.S. research university. Through open coding, we identified 14 distinct prompt–response pair types that cluster into three categories: to generate code, debug code, and explain programming concepts. Our analysis reveals that how students interact with LLMs correlates with academic performance. High-effort detailed specifications for code generation positively correlated with final grades (r = 0.25, p < 0.01), whereas low-effort behaviors such as pasting raw error messages showed negative correlations (r = −0.34, p < 0.01). We also observed a temporal shift toward explanation-oriented interactions, suggesting that students increasingly use LLMs as conceptual tutors and not just as code generators.

Interaction mining is a popular technique for capturing design and interaction data while a mobile app is being used. Over the years, researchers have leveraged interaction mining systems to build large repositories of interaction data, enabling novel, ML-based tools for UX practitioners, designers, and programmers. Existing interaction mining systems range from simple screen recorders — which are easy to use but capture sparse, unstructured data — to complex installations requiring dedicated hardware and custom OS forks — which yield rich, high-fidelity traces but are difficult to deploy outside of a lab environment. This paper presents ODIM, an on-device framework for mobile interaction mining that produces detailed trace metadata. The framework is reified in an Android implementation based on a simple APK that users can install on their personal devices. The paper outlines ODIM’s design principles, describes its implementation, evaluates the system on traces collected from 100 popular apps on the Google Play Store, and discusses future avenues for scaling the utility and impact of interaction mining systems. The ODIM software, source code, and online trace repository are all freely available at interactionmining.org, for anyone to use and contribute to.

User experience (UX) testing platforms capture many data types related to user feedback and behavior, including clickstream, survey responses, screen recordings of participants performing tasks, and participants’ think-aloud audio. Analyzing these multimodal data channels to extract insights remains a time-consuming, manual process for UX researchers. This paper presents a large language model (LLM) approach for generating insights from multimodal UX testing data. By unifying verbal, behavioral, and design data streams into a novel natural language representation, we construct LLM prompts that generate insights combining information across all data types. Each insight can be traced back to behavioral and verbal evidence, allowing users to quickly verify accuracy. We evaluate LLM-generated insight summaries by deploying them in a popular remote UX testing platform, and present evidence that they help UX researchers more efficiently identify key findings from UX tests.

Organizations increasingly rely on behavioral analytics tools like Google Analytics to monitor their digital experiences. Making sense of the data these tools capture, however, requires manual event tagging and filtering — often a tedious process. Prior approaches have trained machine learning models to automatically tag interaction data, but draw from fixed digital experience vocabularies which cannot be easily augmented or customized. This paper introduces a novel machine learning interaction pattern that generates customized tag predictions for organizations. The approach employs a general user experience word embedding to bootstrap an initial set of predictions, which can then be refined and customized by users to adapt the underlying vector space, iteratively improving the quality of future predictions. The paper presents a needfinding study that grounds the design choices of the system, and describes a real-world deployment as part of UserTesting.com that demonstrates the efficacy of the approach.

Digital user experiences are a mainstay of modern communication and commerce; multi-billion dollar industries have arisen around optimizing digital design. Usage analytics and A/B testing solutions allow growth hackers to quantitatively compute conversion over key user journeys, while user experience (UX) testing platforms enable UX researchers to qualitatively analyze usability and brand perception. Although these workflows are in pursuit of the same objective - producing better UX - the gulf between quantitative and qualitative testing is wide: they involve different stakeholders, and rely on disparate methodologies, budget, data streams, and software tools. This gap belies the opportunity to create a single platform that optimizes digital experiences holistically: using quantitative methods to uncover what and how much and qualitative analysis to understand why. Such a platform could monitor conversion funnels, identify anomalous behaviors, intercept live users exhibiting those behaviors, and solicit explicit feedback in situ. This feedback could take many forms: survey responses, screen recordings of participants performing tasks, think-aloud audio, and more. By combining data from multiple users and correlating across feedback types, the platform could surface not just insights that a particular conversion funnel had been affected, but hypotheses about what had caused the change in user behavior. The platform could then rank these insights by how often the observed behavior occurred in the wild, using large-scale analytics to contextualize the results from small-scale UX tests. To this end, a decade of research has focused on interaction mining: a set of techniques for capturing interaction and design data from digital artifacts, and aggregating these multimodal data streams into structured representations bridging quantitative and qualitative experience testing [1-4]. During user sessions, interaction mining systems record user interactions (e.g., clicks, scrolls, text input), screen captures, and render-time data structures (e.g., website DOMs, native app view hierarchies). Once captured, these data streams are aligned and combined into user traces, sequences of user interactions semanticized by the design data of their UI targets [5]. The structure of these traces affords new workflows for composing quantitative and qualitative methods, building toward a unified platform for optimizing digital experiences.

Vision-language navigation (VLN), in which an agent follows language instruction in a visual environment, has been studied under the premise that the input command is fully feasible in the environment. Yet in practice, a request may not be possible due to language ambiguity or environment changes. To study VLN with unknown command feasibility, we introduce a new dataset Mobile app Tasks with Iterative Feedback (MoTIF), where the goal is to complete a natural language command in a mobile app. Mobile apps provide a scalable domain to study real downstream uses of VLN methods. Moreover, mobile app commands provide instruction for interactive navigation, as they result in action sequences with state changes via clicking, typing, or swiping. MoTIF is the first to include feasibility annotations, containing both binary feasibility labels and fine-grained labels for why tasks are unsatisfiable. We further collect follow-up questions for ambiguous queries to enable research on task uncertainty resolution. Equipped with our dataset, we propose the new problem of feasibility prediction, in which a natural language instruction and multimodal app environment are used to predict command feasibility. MoTIF provides a more realistic app dataset as it contains many diverse environments, high-level goals, and longer action sequences than prior work. We evaluate interactive VLN methods using MoTIF, quantify the generalization ability of current approaches to new app environments, and measure the effect of task feasibility on navigation performance.

Computational approaches for user interfaces have been used in adapting interfaces for different modalities, usage scenarios and device form factors, understanding screen semantics for accessibility, task-automation, information extraction, and in assisting interface design. Recent advances in machine learning (ML) have drawn considerable attention across HCI and related fields such as computer vision and natural language processing, leading to new ML-based user interface approaches. Similarly, significant progress has been made with more traditional optimization- and planning-based approaches to accommodate the need for adapting UIs for screens with different sizes, orientations and aspect ratios, and in emerging domains such as VR/AR and 3D interfaces. The proposed workshop seeks to bring together researchers interested in all kinds of computational approaches for user interfaces across different sectors as a community, including those who develop algorithms and models and those who build applications, to discuss common issues including the need for resources, opportunities for new applications, design implications for human-AI interaction in this domain, and practical challenges such as user privacy.

We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a new dataset where the goal is to complete a natural language query in a mobile app. Current datasets for related tasks in interactive question answering, visual common sense reasoning, and question-answer plausibility prediction do not support research in resolving ambiguous natural language requests or operating in diverse digital domains. As a result, they fail to capture complexities of real question answering or interactive tasks. In contrast, MoTIF contains natural language requests that are not satisfiable, the first such work to investigate this issue for interactive vision-language tasks. MoTIF also contains follow up questions for ambiguous queries to enable research on task uncertainty resolution. We introduce task feasibility prediction and propose an initial model which obtains an F1 score of 61.1. We next benchmark task automation with our dataset and find adaptations of prior work perform poorly due to our realistic language requests, obtaining an accuracy of only 20.2% when mapping commands to grounded actions. We analyze performance and gain insight for future work that may bridge the gap between current model ability and what is needed for successful use in application.

Measuring similarity between two images often requires performing complex reasoning along different axes (e.g., color, texture, or shape). Insights into what might be important for measuring similarity can be provided by annotated attributes. Prior work tends to view these annotations as complete, resulting in them using a simplistic approach of predicting attributes on single images, which are, in turn, used to measure similarity. However, it is impractical for a dataset to fully annotate every attribute that may be important. Thus, only representing images based on these incomplete annotations may miss out on key information. To address this issue, we propose the Pairwise Attribute-informed similarity Network (PAN), which breaks similarity learning into capturing similarity conditions and relevance scores from a joint representation of two images. This enables our model to identify that two images contain the same attribute, but can have it deemed irrelevant (e.g., due to fine-grained differences between them) and ignored for measuring similarity between the two images. Notably, while prior methods of using attribute annotations are often unable to outperform prior art, PAN obtains a 4-9% improvement on compatibility prediction between clothing items on Polyvore Outfits, a 5% gain on few shot classification of images using Caltech-UCSD Birds (CUB), and over 1% boost to Recall@1 on In-Shop Clothes Retrieval. Implementation available at https://github.com/samarth4149/PAN

The rise of social media has changed the nature of the fashion industry. Influence is no longer concentrated in the hands of an elite few: social networks have distributed power across a broader set of tastemakers. To understand this new landscape of influence, we created FITNet --- a network of the top 10k influencers of the larger Twitter fashion graph. To construct FITNet, we trained a content-based classifier to identify fashion-relevant Twitter accounts. Leveraging this classifier, we estimated the size of Twitter's fashion subgraph, snowball sampled more than 300k fashion-related accounts based on following relationships, and identified the top 10k influencers in the resulting subgraph. We use FITNet to perform a large-scale analysis of fashion influencers, and demonstrate how the network facilitates discovery, surfacing influencers relevant to specific fashion topics that may be of interest to brands, retailers, and media companies.

The rise of social media has changed the nature of the fashion industry. Influence is no longer concentrated in the hands of an elite few: social networks have distributed power across a broader set of tastemakers. To understand this new landscape of influence, we created FITNet --- a network of the top 10k influencers of the larger Twitter fashion graph. To construct FITNet, we trained a content-based classifier to identify fashion-relevant Twitter accounts. Leveraging this classifier, we estimated the size of Twitter's fashion subgraph, snowball sampled more than 300k fashion-related accounts based on following relationships, and identified the top 10k influencers in the resulting subgraph. We use FITNet to perform a large-scale analysis of fashion influencers, and demonstrate how the network facilitates discovery, surfacing influencers relevant to specific fashion topics that may be of interest to brands, retailers, and media companies.

In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction. However, the datasets used for these problems fail to capture the complexity of real inputs and multimodal environments, such as ambiguous natural language requests and diverse digital domains. We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first to contain natural language requests for interactive environments that are not satisfiable, and we obtain follow-up questions on this subset to enable research on task uncertainty resolution. We perform initial feasibility classification experiments and only reach an F1 score of 37.3, verifying the need for richer vision-language representations and improved architectures to reason about task feasibility.

To accommodate training with the graph encoder (GE), we formulate learning similarity as an edge prediction task on graph with nodes as images as done in [3]. Each node in the graph represents an image and the edges represent ground truth similarity information. Edges are stored as an adjacency matrix A ∈ RN×N where Ai,j = 1 if there exists an edge between node i and node j, i.e., if the images corresponding to the nodes are labelled to be similar in the dataset. Ai,j = 0 otherwise. Note that this is a general formulation and encoders other than the GE can be trained this way. The graph defined just does not play any role in the model’s outputs in that case.

The Rico dataset, containing design data from more than 9.7 k Android apps spanning \(27\) categories, was released in 2017. It exposes visual, textual, structural, and interactive design properties of more than 72 k unique UI screens. Over the years since its release, the original paper has been cited nearly 100 times according to Google Scholar and the dataset has been used as the basis for numerous research projects. In this chapter, we describe the creation of Rico using a system that combined crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. We then describe two projects that we conducted using the dataset: the training of an autoencoder to identify similarity between UI designs, and an exploration of the use of Google’s Material Design within the dataset using machine learned models. We conclude with an overview of other work that has used Rico to understand our mobile UI world and build data-driven models that assist users, designers, and developers.

Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., "send money to Joe"), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., "Joe") to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.

When people talk about fashion, they care about the underlying meaning of fashion concepts,e.g., style.For example, people ask questions like what features make this dress smart.However, the product descriptions in today fashion websites are full of domain specific and low level words. It is not clear to people how exactly those low level descriptions can contribute to a style or any high level fashion concept. In this paper, we proposed a data driven solution to address this concept understanding issues by leveraging a large number of existing product data on fashion sites. We first collected and categorized 1546 fashion keywords into 5 different fashion categories. Then, we collected a new fashion product dataset with 853,056 products in total. Finally, we trained a deep learning model that can explicitly predict and explain high level fashion concepts in a product image with its low level and domain specific fashion features.

Recent advances in generative modeling will enable new kinds of user experiences around content creation, giving us "creative superpowers" and move us toward co-creation. This workshop brings together researchers and practitioners from both fields HCI and AI to explore and better understand both the opportunities and challenges of generative modelling from a Human-AI interaction perspective for the creation of both physical and digital artifacts.

Artificial intelligence (AI) and Human Computer Interaction (HCI) share common roots and early work on conversational agents has laid the foundation for both fields. However, in subsequent decades the initial tight connection between the fields has become less pronounced. The recent rise of deep learning has revolutionized AI and has led to a raft of practical methods and tools that significantly impact areas outside of core-AI. In particular, modern AI techniques now power new ways for machines and humans to interact. Thus it is timely to investigate how modern AI can propel HCI research in new ways and how HCI research can help direct AI developments. This workshop offers a forum for researchers to discuss new opportunities that lie in bringing modern AI methods into HCI research, identifying important problems to investigate, showcasing computational and scientific methods that can be applied, and sharing datasets and tools that are already available or proposing those that should be further developed. The topics we are interested in including deep learning methods for understanding and modeling human behaviors and enabling new interaction modalities, hybrid intelligence that combine human and machine intelligence to solve difficult tasks, and tools and methods for interaction data curation and large-scale data-driven design. At the core of these topics, we want to start the conversation on how data-driven and data-centric approaches of modern AI can impact HCI.