Large programming courses often use test-driven autograding systems (e.g., CodeRunner) for instant feedback, but these usually only show which tests failed without explaining why or how to fix the error. Novice students struggle with such minimal guidance and often resort to trial-and-error. Meanwhile, AI coding assistants (e.g., GitHub Copilot, ChatGPT) can provide hints and code suggestions, but novices may over-trust these outputs and lack the skills to verify them. To solve these issues, we present TraceMate, an IDE plugin that pairs the autograder’s tests with a conversational AI chatbot. TraceMate augments test feedback with context-aware explanations and inline suggestions for code modifications, and immediately validates each AI suggestion on the test suite. This workflow gives actionable hints while ensuring any AI-proposed changes are correct. In a user study with novice programmers, participants using TraceMate solved problems more effectively and reported higher confidence than those using only the autograder. These results suggest that pairing automated tests with an interactive AI assistant can enhance learning in introductory programming courses.
Continuous Integration and Deployment (CI/CD) pipelines are essential for modern software delivery, yet configuring them remains a challenge due to the complexity of textbased formats like YAML. These configurations are error-prone and require substantial expertise, posing a barrier especially to novices. In this paper, we introduce PipeBlocks, a block-based CI/CD tool featuring a graphical interface for visually constructing pipelines through modular, drag-and-drop blocks. PipeBlocks seamlessly integrates with GitHub Actions, allowing users to design, validate, and execute pipelines entirely within the tool while maintaining full compatibility with existing YAML workflows. A key innovation is the ability to trigger and monitor pipeline runs directly in PipeBlocks, eliminating context-switching.
Continuous Integration and Deployment (CI/CD) pipelines are essential for modern software delivery, yet configuring them remains a challenge due to the complexity of text-based formats like YAML. These configurations are error-prone and require substantial expertise, posing a barrier to novices. In this paper, we introduce PipeBlocks, a block-based CI/CD tool featuring a graphical interface for visually constructing pipelines through modular, drag-and-drop blocks. PipeBlocks seamlessly integrates with GitHub Actions, allowing users to design, validate, and execute pipelines entirely within the tool while maintaining full compatibility with existing YAML workflows. A key innovation is the ability to trigger and monitor pipeline runs directly in PipeBlocks, eliminating context-switching. We evaluated PipeBlocks through a controlled study with 10 participants configuring identical pipelines (build, test, deploy) using both PipeBlocks and YAML editing in GitHub Actions. Using the System Usability Scale (SUS) and NASA-TLX benchmarks, we found that PipeBlocks achieved a statistically significantly higher usability score and a lower cognitive load. The results suggest that block-based approaches can effectively lower CI/CD’s learning curve while maintaining functional completeness, making them particularly valuable for academic settings and early-career developers building CI/CD competencies.
Various programming environments feature live programming as a central part of their programming experience. Liveness in these environments is analyzed by several theories, which describe the different forms of liveness and how they influence the programming experience. Unfortunately, these are isolated theories only concerned with live programming features. They can not explain liveness in terms of general interactions with programming tools.One theory that has successfully explained how programmers interact with various kinds of tools is information foraging theory (IFT). In this paper, we propose an IFT interpretation of liveness. By interpreting liveness in terms of a more comprehensive theory of programmer behavior, we can explain the role of liveness in common software development activities such as debugging.We have explored our IFT interpretation of liveness in a first controlled experiment. We investigated the hypothesis that when we remove liveness from live dynamic introspection tools, programmers will use them less often, as using them has a higher cost. We have conducted a post hoc analysis of the results for which we used the edit-run-cycle model adapted to analyze the usage of live programming tools. The post hoc results hint that programmers did shift away from introspection tools. While our study only features post hoc results, it suggests that the IFT interpretation of liveness is a fruitful avenue.
User-centered design (UCD) is a foundational approach in software development that prioritizes users’ needs, behaviors, and expectations throughout the design process. For instance, Design Sprints effectively integrate UCD into fastpaced software development, enabling teams to fail fast, learn quickly, and refine solutions based on real user feedback. This paper explores the potential of Large Language Models (LLMs) as an integrated component of UCD methodologies, specifically within the Design Sprint framework. Our goal is to analyze how LLMs can assist designers throughout different phases of the Design Sprint, improving efficiency and aiding in developing user-centered solutions. To achieve this goal, we conducted a study with Computer Science, Software Engineering, and Design students during a course on innovative software product engineering. The study was structured into three phases: (i) theoretical instruction, (ii) practical application of the design sprint methodology with LLM integration, and (iii) presentation and discussion of the developed products. We conducted focus groups to gather insights on the participants’ experiences using an LLM as a support tool for Design Sprint activities. Our results provide insights into how LLM can be leveraged to assist in rapid product design while preserving UCD principles.
There is a growing trend of using general-purpose large language models (LLMs), such as ChatGPT, in software development workflows, often outside traditional IDEs or engineering practices. One emerging phenomenon is vibe coding, which refers to a fluid, intuitive, and emotionally attuned development style shaped by conversational AI tools, music, mood, and ambient factors. I propose research that investigates why software practitioners are drawn to general-purpose LLMs and how these tools fit into and enhance vibe coding experiences. In particular, I aim to understand how these environments affect productivity and well-being, especially for software practitioners with mental health conditions (e.g., ADHD, anxiety, depression). By examining both the technical and psychological aspects of general-purpose LLM use, including but not limited to vibe coding, I intend to facilitate the development of improved tools, settings, and practices that promote a more balanced and adaptable approach to LLM-driven software development.
Google’s Blockly library provides rudimentary support for types that constrain how blocks can be connected to form larger fragments. We argue that Blockly’s current type language does not reach beyond the level of context-freeness, and show how this can be changed — in a non-disruptive way — by adopting elements from attribute grammar and constraint-based typing. We demonstrate the generality of our approach by applying it to different languages, and describe our implementation and integration into Blockly.
—Following the rapid emergence of large language models, Multi-Agent Systems (MASs) became a promising approach for accomplishing complex tasks. In MASs, multiple autonomous agents with predetermined roles collaborate by dividing responsibilities. However, MAS developers often struggle to understand and diagnose agents’ behavior from thousands of inter-agent messages across multiple complex conversations. To identify key requirements and challenges related to evaluating, debugging, and managing MASs, we conducted a formative study with six MAS developers. We then introduce ConvoMap, a prototype that addresses a key challenge of MAS development-understanding agents’ behaviors across multiple conversations. ConvoMap integrates automated qualitative coding to enable multi-level inspection of agents’ behavior. ConvoMap can then visualize hundreds of MAS conversations by representing messages as points on a 2D map that encode their semantic meanings and interactions between agents. To better support navigation and deeper analysis, ConvoMap provides topic overviews and highlights relevant text segments. A comparison study showed that ConvoMap helped to understand agents’ behavior more accurately than the baseline.
As Artificial Intelligence (AI) systems become integral to modern software applications, the need to maintain them through testing, debugging, and repairing has become a fundamental task. However, it remains unclear how current computer science (CS) education prepares students for the maintenance of these systems. I propose research that investigates the extent to which model maintenance concepts are integrated into computer science curricula, how they are presented to students, and what gaps may exist in current educational practices. To achieve this, I am currently conducting a survey aimed at individuals involved in CS education and practice. I will use insights from these efforts to analyze the strengths and weaknesses of existing educational frameworks to prepare future software professionals. The goal is to generate recommendations and interventions that will facilitate the effective integration of AI model testing, debugging, and repair methodologies into CS education.
Exceptional behavior can be harmful to software quality and robustness. Like most modern programming languages, Java offers built-in mechanisms for expressing and handling exceptions. However, despite their languageintegrated design, exception handlers are easy to misuse. Researchers have identified exception-handling anti-patternsrecurring poor coding practices that threaten software quality. One of the most prevalent exception-handling anti-patterns-over-catching-occurs when a handler catches overly broad exception types, potentially obscuring the root cause of a failure. Integrated development environments (IDEs) play a critical role in helping developers identify and address software quality issues. Yet, existing tools can produce an overwhelming number of results, contributing to cognitive overload. To address this problem, we incorporate probabilistic reasoning into IDE-based exception-handling analysis. Exception Shipping & Handling (ESH), our prototype implementation, allows the IDE to present prioritized instances of the over-catch anti-patterns based on their likelihood of being problematic. We evaluate ESH through a user study focused on its usability for helping developers assess over-catching handlers. Our results show that participants found ESH intuitive and provide useful insights into how to integrate probabilistic reasoning into an IDE. Specifically, our study highlights the need for transparency and contextualized UI elements.
Programmer interruptions are common and disruptive. While research has focused on understanding their nature, impact, and associated recovery cost, others have looked at visual reminders of recent development actions as recovery support mechanisms. Until now, however, these have only extended as far as the timeline of source code change alone. Thus, there are opportunities to improve this information in a codebase’s history to better support programmers’ interruption recovery. We conducted a within-subjects exploratory study investigating how developers use code history during interruption recovery. We compared a new sub-goal history interface, which contextually organized code changes and web activity into sub-goals, with an existing chronological timeline tool support identified in the literature. Our findings reveal that developers significantly leveraged the sub-goal history interface more frequently during post-interruption recovery, suggesting its potential to improve how developers navigate past work. While not leading to a significant reduction in overall recovery time in this study, sub-goal history provided valuable support for integrating external resources, facilitated faster initial re-engagement, and showed a generally better recovery success rate. We observed developers employing diverse, multimodal strategies, and, importantly, our results highlight recovery as a multi-stage process rather than a single event. These insights emphasize the crucial role of history-aware tool support for recovery workflows and underscore the need for designs that balance contextual detail with usability and adaptability to effectively support both developers’ passive (reading through information) and active (searching for information) engagement with the recovery tool support.
Human subjects research is fundamental to advancing human-computer interaction (HCI), as it helps researchers understand user behaviors, needs, and experiences to enhance the design of software products [3]. However, conducting such human-centric studies is often hindered by challenges in participant recruitment, including inefficiencies in finding eligible participants and administrative burdens [1]. Participants are hindered from joining research studies due to logistical, eligibility, communication, or personal barriers that make participation impractical, inaccessible, or unappealing. These barriers not only slow down research progress, but also limit the diversity and representativeness of study participants. To address these issues, we propose ParticipantGuide-a structured labelbased approach to enhance participant recruitment through providing key and interpretable information to potential participants. We discuss existing work, present a preliminary design, and provide implications for future research.
Fairness in AI models has become essential as our society increasingly becoming more dependent on AI. A biased model can have a harmful impact on marginalized communities. To address this issue, practitioners have developed fairness tools over time. To understand their practical implication, we designed an interview to curate experiences with fairness tools from academic practioners’. In this paper, we discuss insights from our first round of interviews with practitioners from academia. Although numerous fairness tools have been developed, only a few industry-developed (e.g., AIF360 and Fairlearn) are practically usable and commonly employed by practitioners due to regular maintenance & visibility. The existing toolkit landscape is primarily equipped to solely handle textual data and lacks sufficient resources for language models. Our findings thus far provide insights into one perspective on fairness tool engagement; our future efforts will investigate experiences and perspectives on fairness tool support beyond traditional models.
With the continuous development of generative AI’s logical reasoning abilities, AI’s growing code-generation potential poses challenges for both technical and creative professionals. But how can these advances be directed toward empowering junior researchers and designers who often require additional help to build and express their professional and personal identities? We introduce Frontend Diffusion, a multiagent coding system transforming user-drawn layouts and textual prompts into refined website code, thereby supporting selfrepresentation goals. A user study with 13 junior researchers and designers shows AI as a human capability enhancer rather than a replacement, and highlights the importance of bidirectional human-AI alignment. We then discuss future work such as leveraging AI for career development and fostering bidirectional human-AI alignment of multi-agent systems.
Remote collaboration software (RCS) tools are widely adopted for global teamwork across professional, educational, and personal contexts. However, no method currently assesses RCS tools for coordination, communication, and collaboration (3C) dynamics regarding cognitive and social complexities that shape user interactions. To address this gap, we developed RemoteCollabEval, a novel inspection method refined through iterative design, combining personas with a specialized walkthrough to identify interface issues hindering effective collaboration, such as differing leadership or communication styles. Unlike user studies, which are costly and time-consuming, our method provides a practical alternative through feedback from interdisciplinary stakeholders during multiple action research phases, including case studies and workshops. Participants found the method thorough and thought-provoking, though some noted the repetitive nature as a potential limitation. This work underscores the need for RCS tools that better support nuanced collaboration dynamics, reinforcing the importance of user-centered design.
Technological advancements over the years have led to increasingly complex software systems. The recent surge in artificial intelligence (AI) has, while reducing human effort, further contributed to this complexity. Among the growing concerns in this domain are the ethical implications of developing and using such systems. Regardless of whether a system explicitly incorporates AI, underlying software infrastructures have demonstrated instances of discriminatory behavior, resulting in undesired outcomes. To address this challenge, my research focuses on discrimination testing as a means to ensure equitable behavior in software systems. As part of this effort, I plan to build on insights from my earlier work, which involved compiling a list of existing tools and user studies on their effectiveness. My goal is to design a more explainable and generalizable discrimination testing tool that can be applied across various systems.
This poster evaluated a four-month online professional development program helping seven Southeast Asian teachers from Lao PDR, Thailand, Cambodia and Indonesia embed Computational Thinking (CT) and Universal Design for Learning (UDL) into their lesson plans. Through structured sessions and iterative planning cycles, participants co-created, implemented and revised lessons. Qualitative analysis of their plans, reflections and community discussions showed marked growth: early drafts featured only basic CT and UDL elements, whereas later versions incorporated advanced CT skills (decomposition, data analysis, pseudocoding) and UDL strategies (multiple means of engagement, representation and expression). These results suggest that reflective, collaborative professional development (PD) can boost teachers’ competence and confidence in delivering inclusive, computation-rich instruction.
Over the past few years, millions of people have been using LLM-based AI tools to aid in programming, data analysis, and software engineering tasks. These AI coding assistants range from specialized tools like GitHub Copilot to general-purpose chatbots like Claude. In parallel, academics have published dozens of papers on forward-looking prototypes to expand our collective thinking beyond present-day industry trends. However, despite rapid advances in both sectors in recent years, we still lack an understanding of how their designs relate to one another and what tradeoffs are commonly made. At this key moment in 2025 when design patterns are starting to emerge, it is important to zoom out to see the forest instead of the trees. To do so, we performed the first comprehensive design analysis of 90 LLM-based AI coding assistants. We categorized the feature sets of 58 industry products and 32 academic projects, then formulated a design space that captures key variations in their user experiences. Our design space covers 10 dimensions related to UI modalities, system inputs, capabilities, and outputs. We use this design space to reveal trends in both industry and academic projects across three eras ranging from autocomplete to chat to agent-based interfaces. Lastly, to address the question of who the target users of these tools are, we present six user personas whose preferences lie in different regions of our design space: professional software engineers, HCI researchers and hobbyist programmers, UX designers, conversational programmers (e.g., product managers and marketers), data scientists, and students.
GitHub serves as the primary platform for software maintenance within the open-source community. Developers frequently report issues, ranging from bug reports to feature requests, on GitHub. The presence of well-maintained labels on these issues significantly enhances the problem-solving process by providing immediate context and facilitating the assignment of appropriate personnel, thereby expediting development. However, a substantial number of open-source repositories on GitHub lack consistent and comprehensive issue labeling. This paper addresses this challenge by proposing a classification framework for GitHub issues using ML, DNN & Transformer models. We conducted an extensive analysis of prominent GitHub opensource repositories and classified issues into common categories (e.g.,Documentation, Bug). Our empirical study demonstrates that DNN models consistently outperform ML & Transformer models in this classification task, achieving an accuracy of up to 83.75%. This work contributes to improving the efficiency of open-source software development by automating a crucial aspect of issue management.
Pair programming with AI often faces challenges in productive communication and engagement. Integrating embodiment offers a promising solution by making AI a more engaging and context-aware programming partner. To explore how embodied AI agent supports programming learning and affects user experiences, we designed a virtual reality (VR) programming environment with Wizard-of-Oz-controlled AI agents. Our study collected data from 18 participants through knowledge acquisition assessments and interviews. The results showed that embodiment improved engagement, enhanced communication efficiency, and offered emotional support. Specifically, the incorporation of embodied actions allows users to perceive the AI agent as a “programming partner” and introduces many interactions that resemble those shared with real-life partners. However, the effectiveness of embodied actions in supporting users with programming tasks depends on the timing and accuracy of those actions. This study reveals the potential of embodied AI agents in advancing programming education and provides valuable design insights for creating more intuitive and supportive AI programming partners.
