Systems’ latency – the time between user input and system response – slows down the human-computer interaction loop. Several studies revealed negative objective and subjective effects of high latency, typically treating latency as a constant delay. Because latency varies significantly in practice, recent work also assessed the effects of large and sudden latency changes. In practice, however, latency variations are small but frequent. As the effects of such variations are unclear, we investigate how small latency variations (± 50 ms) affect users’ performance and perceived task load for 2D target selection tasks with static and moving targets. For static targets, we found that latency variation causes significantly higher completion times and less efficient trajectories, however with small effect sizes. In contrast, we found no significant effects on any performance measure for moving targets. Our findings indicate that the effect of latency variation is generally very small and quickly disappears for non-trivial tasks.

Touch-sensitive surfaces offer an intuitive and flexible form of interaction, for example through gesture input. Despite being the primary input modality for mobile devices, they hardly find application in desktop settings. At the same time, the computer mouse is still the most efficient and accurate input device for pointing. Consequently, keeping the unmatched functionality of a mouse but extending it with new input options via a touch-sensitive surface, is a promising approach. While research prototypes and niche products for multi-touch mice exist, the concept has not yet become established. In this work, we follow a user-centered approach towards touch interaction on computer mice. In a user study (n=12), we identified which areas on the mouse are suitable for touch input. Further, we explored potential usage scenarios in a diary study (n=11) and intuitive gestures in an elicitation study (n=10). We compile our findings into a gesture set which future research can build upon to implement touch interaction on computer mice.

There is a rich history of foot-operated machinery, ranging from treadmill-powered lathes to modern vehicles. Even though early HCI research explored general-purpose foot input, most modern day research on this topic focuses either on gesture-based interaction or very specific application scenarios. Today, working on a desktop or laptop computer relies almost exclusively on users’ hands, leaving their feet unoccupied. Therefore, we explore how foot-operated input devices can be incorporated in modern office workflows. We first gathered design requirements and use cases in multiple focus groups sessions. Based on our findings, we conducted a five-day diary study with twelve participants incorporating off-the-shelf footswitches with customizable functions into their usual workflows. Throughout the study, all participants continued to use the footswitches mainly for secondary tasks, such as controlling media playback or triggering shortcuts. We conclude that even simple foot-operated input devices with customizable functions can improve user experience and help users work more efficiently.

All parts of an audio processing chain introduce latency. Previous studies have shown that high audio latency may negatively impact human performance in different scenarios, e.g., when performing live music or when interacting with real-time human-computer systems. However, is not yet known where the human perception threshold for audio latency lies, i.e., what the lowest amount of latency is that musicians might notice. Therefore, we conducted a user study (n=37) using the PEST method to estimate the just noticeable difference (JND) for audio latency under different base latency settings. Our results suggest that base latency influences the perception threshold in a non-linear manner: Participants achieved a mean JND of 49 ms for a base latency of 0 ms, 27 ms for a base latency of 64 ms, and 77 ms for a base latency of 512 ms. Furthermore, the JND was lower for participants with high musical sophistication.

Tables are focus points for social interactions and support everyday activities, such as learning, crafting, or dining. These physical interactions on and around the table may be augmented with digital information and tools projected onto the tabletop. For interaction with such projected information, touch input suffers from technical and interactional limitations. Pen input is a more robust alternative that does not suffer from Midas-touch problems. We developed a system for tracking the position of an IR-emitting pen tip on a planar surface with sub-millimeter resolution and an end-to-end latency of less than 30 ms. Distinguishing between drawing and hovering states is done by combining a stereoscopic camera setup and a machine-learning classifier. We demonstrate practical performance, uses and limitations through multiple studies and examples.

Even though smartphones offer a broad design space for being used as input devices for video games, their form factor makes them less ergonomic than physical gamepads. Previous research suggests that customizable controller layouts and added haptic feedback can improve the quality of smartphone-based gamepads. However, there are no rigorous user studies comparing different types of smartphone controllers to each other. In this paper, we present results of a user study in which we compared three different smartphone-based gamepads: a smartphone controller with a standard layout, a customizable smartphone controller, and a smartphone controller with a haptic case. Additionally, we included a physical gamepad as a reference in our study. Participants used the different controllers to play a racing game and complete pointing tasks. We found that the physical gamepad outperforms smartphone-based controllers in terms of efficiency, but there was no significant difference in effectiveness. Furthermore, our qualitative findings open up design considerations for future improvements of smartphone-based game controllers.

People interact with a multitude of personal digital devices every day. However, transferring data between devices is still surprisingly cumbersome due to technical barriers, such as authentication or device pairing. Due to their clear affordances, physical devices offer a promising design space as mediators for natural interaction techniques. In a workshop and an elicitation study (n = 30), we investigated different interaction techniques for cross-device data transfer using everyday objects. Our results suggest that depending on the use case, extending always-available physical objects might be more beneficial than developing new artifacts. Designing effective interaction techniques requires consideration of an artifact’s physical characteristics, affordances, and situational surroundings. Participants preferred multi-functional objects which are always at hand, such as their smartphone. However, they opted for more impersonal objects in unfamiliar situations. Interaction techniques associated with objects also influenced users’ actions. We provide an overview of factors influencing intuitive interactions and we derived guidelines for user-centered development of interaction techniques with physical objects as mediators for data transfer.

Existing tools for screen sharing and remote control only allow a single user to interact with a system while others are watching. Collaborative editors and whiteboards allow multiple users to work simultaneously, but only offer a limited set of tools. With CoShare, we combine both concepts into a screen sharing tool that gives remote viewers a mouse pointer and a text cursor so that they can seamlessly collaborate within the same desktop environment. We have developed a proof-of-concept implementation that leverages Linux’ multi-pointer support so users can control applications in parallel. It also allows limited sharing of clipboard and dragging files from the remote viewer’s desktop into the video-streamed desktop. In focus groups we gathered user requirements regarding privacy, control, and communication. A qualitative lab study identified further areas for improvement and demonstrated CoShare’s utility.

In interactive systems high latency affects user performance and experience. This is especially problematic in video games. A large number of studies on this topic investigated the effects of constant, high latency. However, in practice, latency is never constant but varies by up to 100 ms due to variations in processing time and delays added by polling between system components. In a large majority of studies, these variations in latency are neither controlled for nor reported. Thus, it is unclear to which degree small, continuous variations in latency affect user performance. If these unreported variations had a significant impact, this might cast into doubt the findings of some studies. To investigate how latency variation affects player performance and experience in games, we conducted an experiment with 28 participants playing a first-person shooter. Participants played with two levels of base latency (50 ms vs. 150 ms) and variation (0 ms vs. 50 ms). As expected, high base latency significantly reduces player performance and experience. However, we found strong evidence that small variations in latency in the order of 50 ms, do not affect player performance significantly. Thus, our findings mitigate concerns that previous latency studies might have systematically ignored a confounding effect.

People interact with a multitude of personal digital devices and in-frastructural hardware every day. Oftentimes, they need to transfer data from one device to another. In many cases this process is still surprisingly cumbersome, requiring additional, non-intuitive steps, such as authentication, device pairing, or network setup. Tangible User Interfaces (TUIs) allow for quick and intuitive physical interaction with digital data. Therefore, they offer a promising design space towards more natural interaction techniques for cross-device data transfer. In a workshop and an elicitation study, we investigated different form factors and interaction techniques using six everyday objects in three different situations. We found that designing effective tangibles requires consideration of various factors which strongly depend on the target group and intended use case.

The latency of a text editor describes how long it takes from pressing a key to the corresponding letter appearing on the screen. It is well known that high latency affects how quickly authors can write and edit texts. In order to quantify the effects of latency on users’ performance and task load, we conducted a study in which 31 participants had to re-type or correct provided texts with a physical keyboard. Each participant completed each task once with a low latency of 20 ms and once with a high latency of 200 ms. We found that latency had no significant effect on users’ performance during the copy task, but correcting texts was affected significantly by high latency. Additionally, our results suggest that fast typers are more likely to notice latency than slow typers. Our findings regarding the effects of text input latency on users’ performance contributes to the existing body of research on latency in interactive systems.

Annotating and proof-reading documents are common tasks. Digital annotation tools provide easily searchable annotations and facilitate sharing documents and remote collaboration with others. On the other hand, advantages of paper, such as creative freedom and intuitive use, can get lost when annotating digitally. There is a large amount of research indicating that paper outperforms digital annotation tools in task time, error recall and task load. However, most research in this field is rather old and does not take into consideration increasing screen resolution and performance, as well as better input techniques in modern devices. We present three user studies comparing different annotation media in the context of proof-reading tasks. We found that annotating on paper is still faster and less stressful than with a PC or tablet computer, but the difference is significantly smaller with a state-of-the-art device. We did not find a difference in error recall, but the used medium has a strong influence on how users annotate.

Latency is an intrinsic property of all human-computer systems. As it can affect user experience and performance, it should be kept as low as possible for real-time applications. To identify the source of latency, measuring partial latencies is necessary. We present a new method for measuring the latency of graphics frameworks on X11-based systems. Our tool measures the time between an input event arriving at the kernel until a pixel is updated in graphics memory. In a systematic evaluation with 36 test applications, we found that our method delivers consistent results for most tested frameworks, and does not add a significant amount of additional end-to-end latency. Even though further investigation is required to explain inconsistencies with Qt-based frameworks, our method measures the latency of graphics frameworks reliably and accurately in all other cases.

Prototyping interactive hardware artifacts is an iterative process that can produce significant amounts of waste. This problem becomes particularly apparent in teaching, when multiple students build the same artifact as an exercise and components can break when used improperly. In the context of a university course on tangible interaction, we explored how material found in the trash could be used as a resource for prototyping interactive artifacts. We could source interesting components and found that a bottom-up prototyping approach based on those components opened up new design spaces. Furthermore, as we relied on trash as a resource, we were able to considerably reduce waste during the course.

Interactive tabletops do not only offer a large surface for collaborative interaction. They also offer quick access to digital tools directly at the table - where a large number of everyday activities take place. Tabletops with an embedded display are generally less flexible and more fragile than ordinary massive tabletops. Physical objects on the tabletop occlude the digital content. In contrast, top-down-projected interfaces using an overhead projector-camera system allow for augmenting arbitrary tables and objects lying on them. However, detecting pointing input only via a camera image captured from above requires robustly recognizing whether a finger or pen touches the tabletop or whether it hovers slightly above it. In this demonstration, we showcase a solution for reliably tracking a pen on arbitrary tabletop surfaces. The pen emits infrared light via a tip made of optical fiber. A camera captures position and shape of the light point on the surface. Our open-source tracking algorithm combines heuristics and a neural network to distinguish between drawing and hovering. This system can be reliably used for drawing and writing on tabletops. However, occlusion by users’ hands can deteriorate tracking of the pen.

Inside-out optical 2D tracking of tangible objects on a surface oftentimes uses a high-resolution pattern printed on the surface. While De-Bruijn-torus patterns offer maximum information density, their orientation must be known to decode them. Determining the orientation is challenging for patterns with very fine details; traditional algorithms, such as Hough Lines, do not work reliably. We show that a convolutional neural network can reliably determine the orientation of quasi-random bitmaps with 6 × 6 pixels per block within 36 × 36 pixel images taken by a mouse sensor. Mean error rate is below 2°. Furthermore, our model outperformed Hough Lines in a test with arbitrarily rotated low-resolution rectangles. This implies that CNN-based rotation-detection might also be applicable for more general use cases.

Despite ever improving digital ink and paper solutions, many people still prefer printing out documents for close reading, proofreading, or filling out forms. However, in order to incorporate paper-based annotations into digital workflows, handwritten text and markings need to be extracted. Common computer-vision and machine-learning approaches require extensive sets of training data or a clean digital version of the document. We propose a simple method for extracting handwritten annotations from laser-printed documents using multispectral imaging. While black toner absorbs infrared light, most inks are invisible in the infrared spectrum. We modified an off-the-shelf flatbed scanner by adding a switchable infrared LED to its light guide. By subtracting an infrared scan from a color scan, handwritten text and highlighting can be extracted and added to a PDF version. Initial experiments show accurate results with high quality on a test data set of 93 annotated pages. Thus, infrared scanning seems like a promising building block for integrating paper-based and digital annotation practices.

Interactive tabletops do not only offer a large surface for collaborative interaction. They also offer quick access to digital tools directly at the table - where a large number of everyday activities take place. Tabletops with an embedded display are generally less flexible and more fragile than ordinary massive tabletops. Physical objects on the tabletop occlude the digital content. In contrast, top-down-projected interfaces using an overhead camera-projector system allow for augmenting arbitrary tables and the object lying on them. However, detecting pointing input only via a camera image captured from above requires robustly recognizing whether a finger or pen touches the tabletop or whether it hovers slightly above it. In this paper, we present a solution for reliably tracking a pen on arbitrary tabletop surfaces. The pen emits infrared light via a tip made of optical fiber. A camera captures position and shape of the light point on the surface. Our open-source tracking algorithm combines heuristics and a machine learning model to distinguish between drawing and hovering. A pilot study with 7 participants shows that that this system can be reliably used for drawing and writing on tabletops. However, occlusion by users’ hands can deteriorate tracking of the pen.