Participants of one-to-many videoconferencing calls often experience a significant loss of affective feedback due to the limitations of the medium. To address this problem, we present Co-Here: a videoconferencing module that renders participant-driven animations that convey group affect without relying on categorical emotion detection or signaling. Co-Here consolidates and visualizes facial expressions and head movements of fellow videoconferencers, providing a sufficient medium for others to meaningfully construct emotional impressions. Our qualitative user study showed that the system helped users feel a sense of alignment with the emotions of others using the system. Co-Here had the most utility to presentation viewers, who reported that the system offered a low-attention alternative to gauging the sentiment of the crowd. Co-Here further enabled a supportive environment that was described as between having cameras on and off. This encouraged users to emote more, and relieved social pressure commonly experienced in group calls.

To collect feedback in the early stages of research when researchers have to work with expert users, we propose a method based on the large language models and careful elaboration of personas. This method enables low-cost simulation of answers to interviews and provides inputs for decisions on preliminary research directions. To evaluate our method, we propose to simulate two existing studies in the aviation domain. These studies focus on concerns and expectations of airline pilots about future single pilot operations and higher level of automation in the cockpit. Our results show similar level of concerns to human participants in these two simulated studies. However, the results of the simulations include approximations, errors in the actual work of pilots and over-and underestimations of some potential problems of single pilot operations. We conclude that our method can help guide the preliminary stages of research if researchers have sufficient prior knowledge of the work domain and if this method is complemented by human-in-the-loop methods.

Our perception of emotion is highly contextual. Changes in the environment can affect our narrative framing, and thus augment our emotional perception of interlocutors. User environments are typically heavily suppressed due to the technical limitations of commercial videoconferencing platforms. As a result, there is often a lack of contextual awareness while participating in a video call, and this affects how we perceive the emotions of conversants. We present a videoconferencing module that visualizes the user's aural environment to enhance awareness between interlocutors. The system visualizes environmental sound based on its semantic and acoustic properties. We found that our visualization system was about 50% effective at eliciting emotional perceptions in users that was similar to the response elicited by environmental sound it replaced.The contributed system provides a unique approach to facilitate ambient awareness on an implicit emotional level in situations where multimodal environmental context is suppressed.

The disruptive nature of smartphone notifications and their negative impact on users’ productivity are well documented. The majority of these results either originate from controlled laboratory studies, or protocols relying on subjective self-reporting, reducing their ecological validity. This paper presents results from a full day in situ study investigating the impact of perceiving one’s smartphone notifications on wrist motion patterns. Through this objective behavioral assessment, we document for the first time the manifestations of notification-induced disruption outside of the lab, independently of user activity and without the need for self-reporting. We identified a decrease in wrist motion activity following the presentation of a notification while the participant was engaged in higher intensity activities, independently of whether the notification is immediately attended to. These findings provide objective support for the claim that notifications have as much potential for disruption when merely perceived as they do when the user actually responds to them.

Conducting psychophysiological investigations outside of lab settings has a lot of potential for academic applications as well as for industries concerned about the quality of their user and customer experience. Prior work employing in-the-wild methodologies often focuses on a limited set of biometrics, which constrains the depth of the insights generated by such investigations. In this work, we field tested a new system for multimodal data acquisition and present exploratory results and insights from two ambulatory data collection sessions, held during public events. Through a spatially grounded analysis, we investigate the feasibility, and practicality of multimodal ambulatory psychophysiological inquiries, and hypothesize on areas of added values for public event organizers.

Modeling with multimodal data in the wild poses similar challenges in human-computer and human-robot interaction (HCI, HRI). This workshop series thus blends HCI and HRI to jointly address a broad range of current topics in multimodal modeling aimed at designing intelligent systems in the wild. From addressing data scarcity in multimodal user state recognition to emotion prediction from EEG while listening to music, our third workshop in this series aims to further stimulate this important multidisciplinary exchange.

In order to effectively generate actionable user insights using biometric data, a deep understanding of the psychophysiological processes involved is required. However, despite a few notable commercial exceptions, psychophysiology remains primarily an academic discipline. Isolating the video game sector as a case study, this work brushes on some of the factors that hold back the adoption of psychophysiology in the industry and introduces an unsupervised approach that aims to facilitate the adoption of multimodal physiological data in product development and decision making.

Appreciating a sensory illusion requires first hand experience with the stimulus. While visual and auditory illusions can be rendered using commonly available hardware, hap-tic illusions often require dedicated mechanical systems. This work introduces a series of haptic illusions and demonstrations that can be fabricated using any FDM 3D printer and assembled by hand. This is a step forward in increasing the accessibility of tactile illusions for educational purposes.

Traditional tacton evaluation studies often rely on pre-defined haptic effects that are specifically tailored to explore a handful of design parameters. To prevent combinatorial explosion, researchers are forced to constrain their exploration to very limited subsets of the parameter space. In this work, we propose a hands-off active sampling strategy grounded in probability and information theory that automatically generates tactons to maximize the perceptual information gain at each stimulus presentation. As a proof of concept of the proposed technique, we present the results from a crowdsourced study investigating the perceived similarity between tactons with over 200 participants. Without researcher intervention in the tacton selection process, our method allowed a set of the most salient features for perception of tacton similarity to emerge naturally from the data. This approach is highly scalable and allows for a more efficient exploration of a larger haptic space than typical laboratory study designs aimed at evaluating perceptual attributes of tactons.

Commercial smartwatches and fitness trackers are integrating increasingly advanced physiological sensors. For optimal performance, such devices need to be firmly coupled to the body, yet also remain comfortable when worn for extended periods of time. Existing solutions for measuring the contact force in order to ensure it is in an optimal tightness range typically depend on direct force measurement, but this adds hardware, and therefore cost, to the devices. This paper presents a novel method for estimating contact force by using only an optical heart rate sensor, as already found in many wearable devices. Initial tests indicate that the proposed method can estimate contact force with a mean absolute error of 0.36 N, on par with FSRs. This new approach has the potential to expand the utility of existing sensors for both researchers and end-users, with anticipated applications not only in optimizing physiological sensing, but also in haptic information delivery.

Voice characteristics are known to influence people's perception of a speaker's professional abilities, often offering an unfair disadvantage to speakers in position of perceived vulnerability. By using a custom speech to haptics synthesis framework, this paper presents results from a user study investigating the influence of haptic speech enhancement on speaker's characteristic perception. A custom speech to haptic system is used to first replicate a study from Klofstad et al. examining how voice pitch influences perception of speaker strength, competence and trustworthiness, and second to explore the impact of multimodal stimulus presentation on these perceived characteristics. Our preliminary findings suggest that the perceived strength of a speaker with a higher voice pitch is enhanced, whereas the outcome is uncertain for competence. Perceived trustworthiness was not affected by the system at all. This work puts forward the potential positive effect that the use of haptic-enhanced communication system could have in social and professional communications, but also outlines its limitations.

Our smartphones are constantly fighting to capture our attention, oftentimes causing significant disruption in professional and social contexts. In contrast with prior smart notification systems work focused on external contextual information (e.g., environment, user activity, etc.), my research explores how the notification experience could be enhanced by providing smartphones with a better awareness of their user's psycho-physiological state both prior to, but more importantly immediately after the presentation of alerts. This paper first summarizes findings from the evaluation of a novel notification perception classification technique based on wearable physiological sensing, and a non-intrusive mobile journaling mechanisms adapted to modern smartphone usage. From there, a tentative sequence of studies is presented, aiming to answer the project's remaining research questions.

As haptics have become an ingrained part of our wearable experience, particularly through phones, smartwatches, and fitness trackers, significant research effort has been conducted to find new ways of using wearable haptics to convey information, especially while we are on-the-go. In this paper, instead of focusing on aspects of haptic information design, such as tacton encoding methods, actuators, and technical fabrication of devices, we address the more general recurring issues and "gotchas" that arise when moving from core haptic perceptual studies and in-lab wearable experiments to real world testing of wearable vibrotactile haptic systems. We summarize key issues for practitioners to take into account when designing and carrying out in-the-wild wearable haptic user studies, as well as for user studies in a lab environment that seek to simulate real-world conditions. We include not only examples from published work and commercial sources, but also hard-won illustrative examples derived from issues and failures from our own haptic studies. By providing a broad-based, accessible overview of recurring issues, we expect that both novice and experienced haptic researchers will find suggestions that will improve their own mobile wearable haptic studies.

Studies suggest that imbalances in speaking opportunities during meetings often lead to sub-optimal meeting outcomes. These imbalances can be due to a variety of reasons, including people’s perception of speakers and their voice. Indeed, speakers with higher pitched voices were shown to be perceived as having lower leadership ability. In an attempt at countering such voice-pitch related biases, this work introduces BarryWhaptics, a real-time speech-to-haptics conversion system that leverages multimodal perception to alter the listener’s perception of a speaker. The system operates by augmenting human speech with vibration, applying more intense vibrations to voices that would ordinarily be considered low in dominance. Results from a pilot study assessing the influence of the system in a decision-making task demonstrate that it can meaningfully influence how users choose to follow instructions given by one speaker over another.

Today's smartphone notification systems are incapable of determining whether a notification has been successfully perceived without explicit interaction from the user. If the system incorrectly assumes that a notification has not been perceived, it may repeat it redundantly, disrupting the user and others (e.g., phone ringing). Or, if it incorrectly assumes that a notification was perceived, and therefore fails to repeat it, the notification will be missed altogether (e.g., text message). Results from a laboratory study confirm, for the first time, that both vibrotactile and auditory smartphone notifications induce skin conductance responses (SCR), that the induced responses differ from that of arbitrary stimuli, and that they could be employed to predict perception of smartphone notifications after their presentation using wearable sensors.

Operating power tools over extended periods of time can pose significant risks to humans, due to the strong forces and vibrations they impart to the limbs. Telemanipulation systems can be employed to minimize these risks, but may impede effective task performance due to the reduced sensory cues they typically convey. To address this shortcoming, we explore the benefits of augmenting these cues with the addition of audition, vibration, and force feedback, and evaluate them on users' performance in a VR mechanical assembly task employing a simulated impact wrench. Our research focuses on the utility of vibrotactile feedback, rendered as a simplified and attenuated version of the vibrations experienced while operating an actual impact wrench. We investigate whether such feedback can serve to enhance the operator's awareness of the state of the tool, as well as a proxy for the forces experienced during collisions and coupling, while operating the tool an actual impact wrench. Results from our user study comparing feedback modalities confirm that the introduction of vibrotactile, in addition to auditory feedback can significantly improve user performance as assessed by completion time. However, the addition of force feedback to these two modalities did not further improve performance.

How firmly a haptic device, such as a smartwatch, is coupled to the body can change how its haptic effects are perceived. However, hapticians often rely on vague subjective coupling characteristics such as "strapped snugly" or "tight yet comfortable". Achieving consistent strap tightness across body sites and between participants can be challenging, since even if strap tension is consistent, differences in limb circumference alter the resulting normal force under the haptic actuator in potentially unintuitive ways. Furthermore, when participants must attach the devices on their own, e.g., during a longitudinal in-the-wild study, they may not use the same tightness each day without guidance. We present HaptiStrap, a low-cost, easily fabricated tool, as a contribution towards a standard method for ensuring that wearable haptic studies do better than vague and subjective "tight yet comfortable" guidelines.

Experience Sampling Methods (ESM) allow the timely collection of subjective self-reports that would otherwise be impossible to measure accurately in ecologically valid scenarios. Recent work suggests that unlock journaling allowed the collection of more data points per day, was faster and perceived as being less intrusive by participants than notification-based ESM. This work extends the unlock journaling field by introducing a novel lockscreen data collection mechanism harnessing an increasingly popular authentication mechanism: the fingerprint sensor. Results collected during a twelve-day user study with fingerprint sensor users show that fingerprint sensor gesture reporting compares favorably to Slide-to-X approaches. The proposed gestural interface was subjectively perceived as being the fastest, least intrusive, and overall most preferred interface, in addition to offering the highest response compliance. By offering a reporting mechanism better aligned with modern smartphone unlocking habits, this work encourages the deployment of unlock journaling in the wild.

Using technology to convey information and feelings between people is a key goal of many interactive systems, typically with the highest connection fidelity possible. However, the choices made during design and implementation inevitably impact how the communication is perceived. As part of the Empathy Mirror project [4], we explore using technology to instead invert the expressed physical aggression of one participant into a soothing massage for another. Participants take out their aggression on a punching bag. The system detects the magnitude of the blows, and processes them into vibrations rendered via a massage seat to a second participant. Participants reflect on how technology can subvert our intentions, such that the receiver's perception may be very different from what the sender originally communicated. In this case, the most aggressive action is to remove themselves from the exhibit, leaving the receiver with no positive vibes, effectively nullifying the ability to be hostile.