Collecting and documenting misconceptions in programming education has been shown to be of great value, yet existing collections often remain hard to browse and difficult to apply in teaching practice. To help educators find what matters, we developed a web-based misconception collection annotated with four teaching relevance criteria: discoverability, learning barrier, correction effort and frequency. These criteria help educators identify the most relevant misconceptions according to their instructional needs. Each entry also includes concrete teaching opportunities, such as peer instruction quizzes, to support active use in the classroom. Our collection is based on the analysis of 1,358 code snippets from 80 first-semester students learning Processing (a Java-based language) as well as 11 structured interviews. Focusing on arrays and functions, we identified 30 misconceptions, some new, some built on prior work.

Current research aims to utilize Large Language Models (LLMs) for tutoring beginning programming students efficiently and at scale. Students often struggle to interact effectively with LLMs to obtain meaningful feedback. We introduce an LLM-based Intelligent Tutoring System (ITS) with a structured interface and prompts aligned with Hattie's feedback model. To provide more focused feedback, we utilize the user interaction history for context. Additionally, we explore the question of proactivity. A user study with 9 participants compared history-based and current-state feedback methods using ChatGPT, showing a preference for history-based feedback in 69% of cases and with higher usefulness ratings (M = 7.57 vs. M = 4.1, p = 0.03 (statistically significant at p < 0.05)). This effect became more pronounced in later learning stages. For proactivity, we collected user data from the study, where participants explicitly requested feedback, and trained a neural network (NN) to predict optimal feedback timing. While the model achieved 97% accuracy on test data, the small sample size (N = 10) and the use of oversampling limit its generalizability. Future work will refine history-based feedback with eye-tracking data and integrate NN-driven proactive behavior to further enhance the effectiveness of LLM-based ITS in programming education.

Programming is an essential cross-disciplinary skill, yet teaching it effectively in large classes can be challenging due to the need for close feedback loops. Identifying and addressing common misconceptions is particularly important during the initial stages of learning to program. While automated interactive tutoring systems have the potential to offer personalized tutoring at scale, current systems tend to emphasize errors and predefined solutions rather than focusing on common misconceptions. In this study, we introduce a novel platform centered on addressing misconceptions in programming education. We describe methods for detecting misconceptions using Abstract Syntax Trees (AST) and providing tailored, level-specific feedback to emulate human-like tutoring. As an empirical basis for this project, we gathered data from various introductory programming courses. Additionally, we advocate for the establishment of a repository of common misconceptions, offering examples derived from both the literature and our own data. Investigating misconceptions can ultimately enhance the teaching strategies of both human educators and AI agents, such as GPT, in guiding learners effectively.

The tourr package in R has several algorithms and displays for showing multivariate data as a sequence of low-dimensional projections.It can display as a movie but has no capacity for interaction, such as stop/go, change tour type, drop/add variables.The tourrGui package provides these sorts of controls, but the interface is programmed with the dated RGtk2 package.This work explores using custom messages to pass data from R to D3 for viewing, using the Shiny framework.This is an approach that can be generally used for creating all sorts of interactive graphics.

Touch-sensitive surfaces are already a standard form of interaction. These surfaces come in many different sizes like tablets or touch walls. However, there is little research to characterize the impact of surface size on touch performance. We conducted a Fitts' Law study of three display sizes (13.5 '' tablet, 28 '' monitor, 69.5 '' large monitor), comparing various performance measures. We found that the smallest size (13.5 '') is problematic both objectively (high error rate) and subjectively (imprecise, difficult-to-use). In contrast, both the medium (28 '') and the large display (69.5 '') perform equally well. However, small displays allow for greater interaction speed compared to very large screens. Our results can help interaction designers and automatic algorithms to optimize current and future touch devices.

We argue that future mobile interfaces should differentiate between various contextual factors like grip and active fingers, adjusting screen elements and behaviors automatically, thus moving from merely responsive design to responsive interaction. Toward this end we conducted a systematic study of screen taps on a mobile device to find out how the way you hold your device impacts performance, precision, and error rate. In our study, we compared three commonly used grips and found that the popular one-handed grip, tapping with the thumb, yields the worst performance. The two-handed grip, tapping with the index finger, is the most precise and least error-prone method, especially in the upper and left halves of the screen. In landscape orientation (two-handed, tapping with both thumbs) we found the best overall performance with a drop in performance in the middle of the screen. Additionally, we found differentiated trade-off relationships and directional effects. From our findings we derive design recommendations for interface designers and give an example how to make interactions truly responsive to the context-of-use.

Predicting the efficiency of interaction techniques can be crucial for designing user interfaces. While models like Fitts' law make general predictions, there is little research on how efficiency varies under different conditions like in which screen region a movement starts and in which direction it is going, and whether the surface is horizontal or vertical. This study investigates these aspects with regard to translation movements on a touch screen, using an extended Fitts' law setup and considering arm kinematics. The results show that on horizontal displays translation is faster and causes less arm fatigue than on vertical ones. Also, on horizontal displays, we identified screen regions and movement directions that allow significantly faster movement compared to others. Finally, movements that employ shorter kinematic chains (e. g. just the wrist) are significantly faster than those that use longer ones (e. g. wrist, elbow, shoulder). We suggest adjustments to Fitts' original formulation. In the future, our findings can inform or partially automate positioning decisions in interaction design.

The translation and rotation of objects with two fingers is a well explored multitouch technique. However, there are some unsolved questions regarding the optimal conditions under which this technique functions best. Does it matter in which direction the movement is oriented? Does parallel or sequential performance of the two operations work best? This study attempts to answer this question using a typical Fitts' Law setup but with varying translation-rotation orientation combinations. The results show that right-oriented movements were faster and easier than left-oriented ones. Movement combinations which went in different directions (translation right, rotation left, and vice versa) were found more tiresome and resulted in more strategy switches compared to equi-directional combinations. Our findings can inform positioning decisions in interaction design and contribute to theoretical adjustments to Fitts' Law.

The qualitative analysis of nonverbal communication is more and more relying on 3D recording technology. However, the human analysis of 3D data on a regular 2D screen can be challenging as 3D scenes are difficult to visually parse. To optimally exploit the full depth of the 3D data, we propose to enhance the 3D view with a number of visualizations that clarify spatial and conceptual relationships and add derived data like speed and angles. In this paper, we present visualizations for directional body motion, hand movement direction, gesture space location, and proxemic dimensions like interpersonal distance, movement and orientation. The proposed visualizations are available in the open source tool JMocap and are planned to be fully integrated into the ANVIL video annotation tool. The described techniques are intended to make annotation more efficient and reliable and may allow the discovery of entirely new phenomena.

Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared environment to ground referring expressions, both for human and robot speakers. What is less clear is whether the benefit of speaker gaze is due to the inference of referential intentions (Staudte and Crocker, 2011) or simply the (reflexive) shifts in visual attention. That is, is gaze special in how it affects simultaneous utterance comprehension? In four eye-tracking studies we directly contrast speech-aligned speaker gaze of a virtual agent with a non-gaze visual cue (arrow). Our findings show that both cues similarly direct listeners' attention and that listeners can benefit in utterance comprehension from both cues. Only when they are similarly precise, however, does this equality extend to incongruent cueing sequences: that is, even when the cue sequence does not match the concurrent sequence of spoken referents can listeners benefit from gaze as well as arrows. The results suggest that listeners are able to learn a counter-predictive mapping of both cues to the sequence of referents. Thus, gaze and arrows can in principle be applied with equal flexibility and efficiency during language comprehension.

Precision tasks in 3D like object manipulation or character animation call for new gestural interfaces that utilize many input degrees of freedom. We present MotionBender, a sensor-based interaction technique for post-editing the motion of e. g. the hands in character animation data. For the visualization of motion we use motion paths, often used for showing e.g. the movement of the hand through space over time, and allow the user to directly "bend" the 3D motion path with his/her hands and twist it into the right shape. In a comparative evaluation with a mouse-based interface we found that subjects using our technique were significantly faster. Moreover, with our technique, subject movement was more coordinated, i.e. movement was done in all three dimensions in parallel, and the participants preferred our technique in a post-experiment questionnaire. We also found a gender effect: male users both like the gesture interaction better and achieve better performance.

Human motion is challenging to analyze due to the many degrees of freedom of the human body. While the qualitative analysis of human motion lies at the core of many research fields, including multimodal communication, it is still hard to achieve reliable results when human coders transcribe motion with abstract categories. In this paper we tackle this problem in two respects. First, we provide facilities for qualitative and quantitative comparison of annotations. Second, we provide facilities for exploring highly precise recordings of human motion (motion capture) using a low-cost consumer device (Kinect). We present visualization and analysis methods, integrated in the existing ANVIL video annotation tool (Kipp, 2001), and provide both a precision analysis and a "cookbook" for Kinect-based motion analysis.

Creating interactive applications with multiple virtual characters comes along with many challenges that are related to different areas of expertise. The definition of context-sensitive interactive behavior requires expert programmers and often results in hard-to-maintain code. To tackle these challenges, we suggest a visual authoring approach for virtual character applications and present a revised version of our SceneMaker tool. In SceneMaker a separation of content and logic is enforced. In the revised version, the Visual SceneMaker, we introduce concurrency and specific history structures as key concepts to facilitate (1) clearly structured interactive behavior definition, (2) multiple character modeling, and (3) extensions to existing applications. The new integrated developer environment allows sceneflow visualization and runtime modifications to support the development of interactive character applications in a rapid prototyping style. Finally, we present the result of a user study, which evaluates usability and the key concepts of the authoring tool.

While the availability of multimedia data, including human movement recording by motion capture, is steadily growing, the integrated viewing, annotation, and analysis of such complex data is still a challenge. The ANVIL tool, a widely used multi-track video and audio annotation tool, has now been extended to allow the synchronized handling of multiple media, especially the 3D viewing of motion capture data, to perform SQL queries and to conduct automated statistical analysis. The underlying database in conjunction with association detection allow analysis across tracks and modalities. This can be exploited in many contexts, from qualitative behavior analysis to the collection of training data for information extraction. Apart from describing the underlying methods and their realization in ANVIL, we discuss open issues like tool interoperabilty and scheme standardization.

This paper shows how interoperable dialogue act annotations, using the multidimensional annotation scheme and the markup language DiAML of ISO standard 24617-2, can conveniently be obtained using the newly implemented facility in the ANVIL annotation tool to produce XML-based output directly in the DiAML format. ANVIL offers the use of multiple user-defined `tiers' for annotating various kinds of information. This is shown to be convenient not only for multimodal information but also for dialogue act annotation according to ISO standard 24617-2 because of the latter's multidimensionality: functional dialogue segments are viewed as expressing one or more dialogue acts, and every dialogue act belongs to one of a number of dimensions of communication, defined in the standard, for each of which a different ANVIL tier can conveniently be used. Annotations made in the multi-tier interface can be exported in the ISO 24617-2 format, thus supporting the creation of interoperable annotated corpora of multimodal dialogue.

This paper shows how interoperable dialogue act annotations, using the multidimensional annotation scheme and the markup language DiAML of ISO standard 24617-2, can conveniently be obtained using the newly implemented facility in the ANVIL annotation tool to produce XML-based output directly in the DiAML format.

For many application areas, where a task is most naturally represented by talking or where standard input devices are difficult to use or not available at all, virtual characters can be well suited as an intuitive man-machineinterface due to their inherent ability to simulate verbal as well as nonverbal communicative behavior. This type of interface is made possible with the help of multimodal dialog systems, which extend common speech dialog systems with additional modalities just like in human-human interaction. Multimodal dialog systems consist at least of an auditive and graphical component, and communication is based on speech and nonverbal communication alike. However, employing virtual characters as personal and believable dialog partners in multimodal dialogs entails several challenges, because this requires not only a reliable and consistent motion and dialog behavior but also regarding nonverbal communication and affective components. Besides modeling the “mind” and creating intelligent communication behavior on the encoding side, which is an active field of research in artificial intelligence, the visual representation of a character including its perceivable behavior, from a decoding perspective, such as facial expressions and gestures, belongs to the domain of computer graphics and likewise implicates many open issues concerning natural communication. Therefore, in this report we give a comprehensive overview how to go from communication models to actual animation and rendering.

Speaker gaze affects utterance comprehension beyond visual attention shifts Maria Staudte & Matthew W. Crocker Department of Computational Linguistics, Saarland University Saarbr¨ucken, Germany Alexis Heloir & Michael Kipp DFKI, Embodied Agents Research Group Saarbr¨ucken, Germany Abstract Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared visual environment to ground referring expressions, both for human and robot speakers. What is less clear is whether listeners exploit speaker gaze to infer referential intentions (Staudte & Crocker, 2010), or whether the benefits of gaze can be more simply explained by (reflexive) gaze following (Friesen & Kingstone, 1998). To investigate this issue, we conducted two eye-tracking studies which directly contrast speech-aligned speaker gaze of a vir- tual agent with a non-gaze visual cue (arrow). Our findings show that speaker gaze is beneficial to listeners only when the order of gaze cues matched the order of mentioned objects in the utterance. Similarly timed arrow cues, however, benefit listeners regardless of the order in which they occur. These findings are consistent with the view that gaze is interpreted as reflecting the speaker’s referential intentions, while other vi- sual cues regarding mentioned objects can be exploited more flexibly and strategically. Keywords: gaze; comprehension; visual attention shift; ar- rows; referring expressions Introduction In face-to-face communication, the speaker’s gaze to objects in a shared scene provides the listener with a visual cue to the speaker’s focus of (visual) attention (Emery, 2000; Flom, Lee, & Muir, 2007). This potentially offers the listener valu- able information to ground and disambiguate referring ex- pressions, to hypothesize about the speaker’s communicative intentions and goals and, thus, to facilitate comprehension (Hanna & Brennan, 2007). It is an open question, however, whether this functionality of speaker gaze results simply from its established ability to drive listeners’ visual attention, as do other cues as well, or whether gaze uniquely expresses (ref- erential) intentions. More precisely, there are two levels on which a visual at- tention shift in response to a speaker’s gaze may affect utter- ance processing (Staudte & Crocker, 2010). On a perceptual level, gaze-following may be considered as (reflexive) visuo- spatial orienting which increases the visual saliency of the particular target object and/or location in focus (Driver et al., 1999; Friesen & Kingstone, 1998; Langton & Bruce, 1999). On a cognitive level, gaze may additionally be understood as a cue to the speaker’s referential intentions which elicits expectations about which referent would be mentioned next (Hanna & Brennan, 2007). Previously, these two levels have been identified as the Visual and the Intentional Account, re- spectively (Staudte & Crocker, 2010). Crucially, the issue whether gaze is processed on both levels – that is, whether the Intentional Account but not the Visual Account alone – of- fers a satisfying explanation of gaze effects on utterance com- prehension, is still under debate. However, recent evidence seems to converge in support of such an assumption (Bec- chio, Bertone, & Castiello, 2008; Meltzoff, Brooks, Shon, & Rao, 2010; Staudte & Crocker, 2010). Staudte and Crocker (2010), for instance, synchronized gaze movements of a robot (as one instance of an artificial agent) with its speech in a human-like manner. This was shown to be similarly useful for grounding and resolving spo- ken references as human gaze (Hanna & Brennan, 2007). Fur- ther, Staudte and Crocker (2010) have shown that the order of respective gaze and speech cues is important for efficient comprehension whereas the temporal alignment of those cues is not. That is, when referential gaze cues and the correspond- ing referring expressions occurred in a coherent linear order, utterance comprehension was facilitated. When this order was reversed, however, gaze did not only not help but instead even slowed comprehension. In contrast, whether the respec- tive gaze cues occurred one second or five seconds prior to the corresponding referring expression onsets, did not affect the facilitative or disruptive influence of gaze order on com- prehension time. Previous studies have typically manipulated only the va- lidity or credibility of such gaze cues and neglected a direct assessment of the question whether effects on utterance pro- cessing are due to shifts in visual attention per se, or whether speaker gaze specifically (as opposed to other exogenous or even endogenous visual cues, Posner, 1980) elicited those at- tention shifts. To further explore the hypothesis that gaze is indeed interpreted with respect to referential intentions, we adopt an improved experimental design from (Staudte & Crocker, 2010) with a virtual character replacing the robot. We then contrast the influence of gaze and arrows by replac- ing the gaze cue with an arrow cue, directly comparing the effects of two, possibly different types of visual cues. Specif- ically, we report supporting evidence from two studies that, firstly, replicate the results on the relevance of gaze cue or- der for comprehension (Experiment 1) and, secondly, show that other, purely visual cues such as arrows (Experiment 2), induce similar attention shifts as gaze but crucially lack an ef- fect of (inconsistent) order. This supports the hypothesis that gaze does, but arrows do not, elicit inferences about referen- tial intentions.

Gaze is known to be an important social cue in face-to-face communication indicating focus of attention. Speaker gaze can influence object perception and situated utterance comprehension by driving both interlocutors’ visual attention towards the same object; hence facilitating grounding and disambiguation [1]. The precise temporal and causal processes involved in on-line gaze-following during concurrent utterance comprehension are, however, still largely unknown. Specifically, the alignment of referential gaze and speech cues may be essential to such benefit. In this paper, we report findings from an eye-tracking study exploiting a virtual character [2] to systematically assess how speaker gaze influences listeners’ on-line comprehension.