This paper presents the efforts made towards the creation of a client interface to be used with Hello-Robot Stretch. The goal is to create an interface that is accessible to allow for the best user experience. This interface enables users to control Stretch with basic commands through several modalities. To make this interface accessible, a simple and clear web interface was crafted so users of differing abilities can successfully interact with Stretch. A voice activated option was also added to further increase the range of possible interactions.

New mobile manipulator platforms, like the Hello Robot Stretch, have made the idea of long-term in-home robotic assistance feasible. However, existing autonomous capabilities for such robots in unstructured, highly-varied environments are still not available. Instead, using robots with human tele-operation can have huge immediate impact. For these robots to serve populations that need them the most, their interfaces need to be accessible to people with mobility limitations. In this paper we explore the utility, usability, and accessibility of a tele-operated Stretch robot in the home. We first describe a browser-based interface for controlling the Stretch robot designed with accessibility in mind. We then present findings from a study (N=18) in which participants used the interface to remotely control the robot to perform realistic tasks in a kitchen, demonstrating the feasibility of tele-operated assistance and revealing challenges and opportunities. Next, we present a study with individuals with mobility limitations (N=3) identifying additional accessibility requirements for the interface. Participants in both studies agree to the utility of the robot despite current limitations.

Reasoning about object handover configurations allows an assistive agent to estimate the appropriateness of handover for a receiver with different arm mobility capacities. While there are existing approaches for estimating the effectiveness of handovers, their findings are limited to users without arm mobility impairments and to specific objects. Therefore, current state-of-the-art approaches are unable to hand over novel objects to receivers with different arm mobility capacities. We propose a method that generalises handover behaviours to previously unseen objects, subject to the constraint of a user's arm mobility levels and the task context. We propose a heuristic-guided hierarchically optimised cost whose optimisation adapts object configurations for receivers with low arm mobility. This also ensures that the robot grasps consider the context of the user's upcoming task, i.e., the usage of the object. To understand preferences over handover configurations, we report on the findings of an online study, wherein we presented different handover methods, including ours, to 259 users with different levels of arm mobility. We find that people's preferences over handover methods are correlated to their arm mobility capacities. We encapsulate these preferences in a statistical relational learner (SRL) that is able to reason about the most suitable handover configuration given a receiver's arm mobility and upcoming task. Using our SRL model, we obtained an average handover accuracy of 90.8% when generalising handovers to novel objects.

Cursor-based tele-operation interfaces for manipulators can enable widely available and accessible control of robots to make many near term applications possible. However, their efficiency is restricted by the challenge of controlling 6 Degrees-of-Freedom (DoF) with 2D input from the cursor. Existing interfaces make use of different strategies to tackle this challenge, including viewpoint constraints, mode switching, and visual overlays, but it is unclear how these strategies impact the efficiency and accessibility of the interface. In this paper we characterize the design space of cursor-based robot control interfaces and compare alternatives in two user studies. Study 1 (N=216) compares nine alternative interfaces focusing on control of 3 DoFs to understand the differences of the interfaces at the basic level and examine the impact of task parameters on efficiency. Study 2 (N=60) compares a subset of the interfaces integrated into a system that allows full control of a robot manipulator from three orthogonal views. We also present a framework for heuristically evaluating accessibility of these interfaces and discuss the efficiency and accessibility trade-off with recommendations.

Introduction Point-of-injury (POI) care requires immediate specialized assistance but delays and expertise lapses can lead to complications. In such scenarios, telementoring can benefit health practitioners by transmitting guidance from remote specialists. However, current telementoring systems are not appropriate for POI care. This article clinically evaluates our System for Telementoring with Augmented Reality (STAR), a novel telementoring system based on an augmented reality head-mounted display. The system is portable, self-contained, and displays virtual surgical guidance onto the operating field. These capabilities can facilitate telementoring in POI scenarios while mitigating limitations of conventional telementoring systems. Methods Twenty participants performed leg fasciotomies on cadaveric specimens under either one of two experimental conditions: telementoring using STAR; or without telementoring but reviewing the procedure beforehand. An expert surgeon evaluated the participants' performance in terms of completion time, number of errors, and procedure-related scores. Additional metrics included a self-reported confidence score and postexperiment questionnaires. Results STAR effectively delivered surgical guidance to nonspecialist health practitioners: participants using STAR performed fewer errors and obtained higher procedure-related scores. Conclusions This work validates STAR as a viable surgical telementoring platform, which could be further explored to aid in scenarios where life-saving care must be delivered in a prehospital setting.

BACKGROUND:The surgical workforce particularly in rural regions needs novel approaches to reinforce the skills and confidence of health practitioners. Although conventional telementoring systems have proven beneficial to address this gap, the benefits of platforms of augmented reality-based telementoring in the coaching and confidence of medical personnel are yet to be evaluated. METHODS:A total of 20 participants were guided by remote expert surgeons to perform leg fasciotomies on cadavers under one of two conditions: (1) telementoring (with our System for Telementoring with Augmented Reality) or (2) independently reviewing the procedure beforehand. Using the Individual Performance Score and the Weighted Individual Performance Score, two on-site, expert surgeons evaluated the participants. Postexperiment metrics included number of errors, procedure completion time, and self-reported confidence scores. A total of six objective measurements were obtained to describe the self-reported confidence scores and the overall quality of the coaching. Additional analyses were performed based on the participants' expertise level. RESULTS:Participants using the System for Telementoring with Augmented Reality received 10% greater Weighted Individual Performance Score (P = .03) and performed 67% fewer errors (P = .04). Moreover, participants with lower surgical expertise that used the System for Telementoring with Augmented Reality received 17% greater Individual Performance Score (P = .04), 32% greater Weighted Individual Performance Score (P < .01) and performed 92% fewer errors (P < .001). In addition, participants using the System for Telementoring with Augmented Reality reported 25% more confidence in all evaluated aspects (P < .03). On average, participants using the System for Telementoring with Augmented Reality received augmented reality guidance 19 times on average and received guidance for 47% of their total task completion time. CONCLUSION:Participants using the System for Telementoring with Augmented Reality performed leg fasciotomies with fewer errors and received better performance scores. In addition, participants using the System for Telementoring with Augmented Reality reported being more confident when performing fasciotomies under telementoring. Augmented Reality Head-Mounted Display-based telementoring successfully provided confidence and coaching to medical personnel.

Telementoring platforms can help transfer surgical expertise remotely. However, most telementoring platforms are not designed to assist in austere, pre-hospital settings. This paper evaluates the system for telementoring with augmented reality (STAR), a portable and self-contained telementoring platform based on an augmented reality head-mounted display (ARHMD). The system is designed to assist in austere scenarios: a stabilized first-person view of the operating field is sent to a remote expert, who creates surgical instructions that a local first responder wearing the ARHMD can visualize as three-dimensional models projected onto the patient’s body. Our hypothesis evaluated whether remote guidance with STAR could lead to performing a surgical procedure better, as opposed to remote audio-only guidance. Remote expert surgeons guided first responders through training cricothyroidotomies in a simulated austere scenario, and on-site surgeons evaluated the participants using standardized evaluation tools. The evaluation comprehended completion time and technique performance of specific cricothyroidotomy steps. The analyses were also performed considering the participants’ years of experience as first responders, and their experience performing cricothyroidotomies. A linear mixed model analysis showed that using STAR was associated with higher procedural and non-procedural scores, and overall better performance. Additionally, a binary logistic regression analysis showed that using STAR was associated to safer and more successful executions of cricothyroidotomies. This work demonstrates that remote mentors can use STAR to provide first responders with guidance and surgical knowledge, and represents a first step towards the adoption of ARHMDs to convey clinical expertise remotely in austere scenarios.

Augmented Reality (AR) benefits telementoring by enhancing the communication between the mentee and the remote mentor with mentor authored graphical annotations that are directly integrated into the mentee's view of the workspace. An important problem is conveying the workspace to the mentor effectively, such that they can provide adequate guidance. AR headsets now incorporate a frontfacing video camera, which can be used to acquire the workspace. However, simply providing to the mentor this video acquired from the mentee's first-person view is inadequate. As the mentee moves their head, the mentor's visualization of the workspace changes frequently, unexpectedly, and substantially. This paper presents a method for robust high-level stabilization of a mentee first-person video to provide effective workspace visualization to a remote mentor. The visualization is stable, complete, up to date, continuous, distortion free, and rendered from the mentee's typical viewpoint, as needed to best inform the mentor of the current state of the workspace. In one study, the stabilized visualization had significant advantages over unstabilized visualization, in the context of three number matching tasks. In a second study, stabilization showed good results, in the context of surgical telementoring, specifically for cricothyroidotomy training in austere settings.

A robot-assisted feeding system can potentially help a user with upper-body mobility impairments eat independently. However, autonomous assistance in the real world is challenging because of varying user preferences, impairment constraints, and possibility of errors in uncertain and unstructured environments. An autonomous robot-assisted feeding system needs to decide the appropriate strategy to acquire a bite of hard-to-model deformable food items, the right time to bring the bite close to the mouth, and the appropriate strategy to transfer the bite easily. Our key insight is that a system should be designed based on a user's preference about these various challenging aspects of the task. In this work, we explore user preferences for different modes of autonomy given perceived error risks and also analyze the effect of input modalities on technology acceptance. We found that more autonomy is not always better, as participants did not have a preference to use a robot with partial autonomy over a robot with low autonomy. In addition, participants' user interface preference changes from voice control during individual dining to web-based during social dining. Finally, we found differences on average ratings when grouping the participants based on their mobility limitations (lower vs. higher) that suggests that ratings from participants with lower mobility limitations are correlated with higher expectations of robot performance.

Background: While there has been abundant research concerning neurological responses to gesture generation, the time course of gesture processing is not well understood. Specifically, it is not clear if or how particular characteristics within the kinematic execution of gestures capture attention and aid in the classification of gestures with communicative intent. If indeed key features of gestures with perceptual saliency exist, such features could help form the basis of a compact representation of the gestures in memory. Methods: This study used a set of available gesture videos as stimuli. The timing for salient features of performed gestures was determined by isolating inflection points in the hands' motion trajectories. Participants passively viewed the gesture videos while continuous EEG data was collected. We focused on mu oscillations (10 Hz) and used linear regression to test for associations between the timing of mu oscillations and inflection points in motion trajectories. Results: Peaks in the EEG signals at central and occipital electrodes were used to isolate the salient events within each gesture. EEG power oscillations were detected 343 and 400ms on average after inflection points at occipital and central electrodes, respectively. A regression model showed that inflection points in the motion trajectories strongly predicted subsequent mu oscillations (R^2=0.961, p<.01). Conclusion: The results suggest that coordinated activity in the visual and motor cortices are highly correlated with key motion components within gesture trajectories. These points may be associated with neural signatures used to encode gestures in memory for later identification and even recognition.

This paper presents the initial efforts towards developing a robotic limb repositioning system. We aim to combine programming by demonstration and end-user programming in a tele-manipulation system that includes the user in the loop. We propose an approach based on a general-purpose mobile manipulator and a web-based interface where a user can select, edit, preview and execute different repositioning exercises based on the selected limb. This approach shows the potential to empower people who have mobility impairments to be more involved in an activity of daily living.

Understanding people's attitudes towards robots and how those attitudes are affected by exposure to robots is essential to the effective design and development of social robots. Although researchers have been studying attitudes towards robots among adults and even children for more than a decade, little has been explored assessing attitudes among teens-a highly vulnerable population that presents unique opportunities and challenges for social robots. Our work aims to close this gap. In this paper we present findings from several participatory robot interaction and design sessions with 136 teenagers who completed a modified version of the Negative Attitudes Towards Robots Scale (NARS) before participation in a robot interaction. Our data reveal that most teens are 1) highly optimistic about the helpfulness of robots, 2) do not feel nervous talking with a robot, but also 3) do not trust a robot with their data. Ninety teens also completed a post-interaction survey and reported a significant change in the motional attitudes subscale of the NARS. We discuss the implications of our findings on the design of social robots for teens.

Maintaining independence and dignity is a primary goal of successful aging for older adults around the globe. Robots can support this goal in various ways by assisting everyday tasks that become challenging due to aging-related deterioration in physical and mental abilities. While a growing body of research tackles challenges in creating such robots, most work has focused on older adults with high socio-economic status in the developed world. In most cases, the price of these robots alone prohibits their potential use in the developing world. Further, socio-cultural differences in the developing world will limit the usability and chance of adoption of a robot designed based on users in the developed world. Our work aims to close this gap. In this paper we present findings from the user-centered design and development process of a low-cost assistive robot for older adults in the developing world named SHEBA, which is a Bengali term for care. We first interviewed 37 older adults and 21 caregivers in assisted and independent living settings in Dhaka, Bangladesh to gather requirements and understand priorities. We then developed a prototype focused on medication management and delivery and we brought it to an assisted living center to interact with potential older adult users. We interviewed 23 older adults and 5 caregivers who interacted with or observed our prototype to gather feedback. We present quantitative and qualitative data obtained in these interviews, identifying key requirements for robots designed for older adults in the developing world.

Introduction Surgical telementoring connects expert mentors with trainees performing urgent care in austere environments. However, such environments impose unreliable network quality, with significant latency and low bandwidth. We have developed an augmented reality telementoring system that includes future step visualization of the medical procedure. Pregenerated video instructions of the procedure are dynamically overlaid onto the trainee's view of the operating field when the network connection with a mentor is unreliable. Methods Our future step visualization uses a tablet suspended above the patient's body, through which the trainee views the operating field. Before trainee use, an expert records a "future library" of step-by-step video footage of the operation. Videos are displayed to the trainee as semitransparent graphical overlays. We conducted a study where participants completed a cricothyroidotomy under telementored guidance. Participants used one of two telementoring conditions: conventional telestrator or our system with future step visualization. During the operation, the connection between trainee and mentor was bandwidth throttled. Recorded metrics were idle time ratio, recall error, and task performance. Results Participants in the future step visualization condition had 48% smaller idle time ratio (14.5% vs. 27.9%, P < 0.001), 26% less recall error (119 vs. 161, P = 0.042), and 10% higher task performance scores (rater 1 = 90.83 vs. 81.88, P = 0.008; rater 2 = 88.54 vs. 79.17, P = 0.042) than participants in the telestrator condition. Conclusions Future step visualization in surgical telementoring is an important fallback mechanism when trainee/mentor network connection is poor, and it is a key step towards semiautonomous and then completely mentor-free medical assistance systems.

This poster presents the design, implementation, and evaluation of a method for robust high-level stabilization of mentees first-person video in augmented reality (AR) telementoring. This video is captured by the front-facing built-in camera of an AR headset and stabilized by rendering from a stationary view a planar proxy of the workspace projectively texture mapped with the video feed. The result is stable, complete, up to date, continuous, distortion free, and rendered from the mentee's default viewpoint. The stabilization method was evaluated in two user studies, in the context of number matching and for cricothyroidotomy training, respectively. Both showed a significant advantage of our method compared with unstabilized visualization.

Combat trauma injuries require urgent and specialized care. When patient evacuation is infeasible, critical life-saving care must be given at the point of injury in real-time and under austere conditions associated to forward operating bases. Surgical telementoring allows local generalists to receive remote instruction from specialists thousands of miles away. However, current telementoring systems have limited annotation capabilities and lack of direct visualization of the future result of the surgical actions by the specialist. The System for Telementoring with Augmented Reality (STAR) is a surgical telementoring platform that improves the transfer of medical expertise by integrating a full-size interaction table for mentors to create graphical annotations, with augmented reality (AR) devices to display surgical annotations directly onto the generalist’s field of view. Along with the explanation of the system’s features, this paper provides results of user studies that validate STAR as a comprehensive AR surgical telementoring platform. In addition, potential future applications of STAR are discussed, which are desired features that state-of-the-art AR medical telementoring platforms should have when combat trauma scenarios are in the spotlight of such technologies.

Robot-assisted feeding (RAF) systems offer enormous potential benefits to community-centered care-giving environments. However, developers of RAF technologies often focus on evaluating their standard transactional functionality, omitting the impact of such technologies in contexts that extend past the interaction of the robot and food receiver. RAF technologies have complex social, cultural and self-identity implications, since a "meal" extends well beyond the simple provisioning of nourishment. To better understand these implications we conducted a contextual inquiry in an assisted-living community with five potential care recipients and five caregivers, as well as interviews with fifteen domain experts including occupational therapists and feeding specialists. Based on our findings from these studies, we developed a new framework for RAF technologies that formulates this vital task as a community-centered relational service. We then use this framework to qualitatively and quantitatively assess three existing feeding systems and identify areas of improvement. Our work reveals new insights about stakeholders of RAF technologies and provides a roadmap for technology developers to better serve the needs of these stakeholders.

Objective: This study investigates the benefits of a surgical telementoring system based on an augmented reality head-mounted display (ARHMD) that overlays surgical instructions directly onto the surgeon's view of the operating field, without workspace obstruction. Summary Background Data: In conventional telestrator-based telementoring, the surgeon views annotations of the surgical field by shifting focus to a nearby monitor, which substantially increases cognitive load. As an alternative, tablets have been used between the surgeon and the patient to display instructions; however, tablets impose additional obstructions of surgeon's motions. Methods: Twenty medical students performed anatomical marking (Task1) and abdominal incision (Task2) on a patient simulator, in 1 of 2 telementoring conditions: ARHMD and telestrator. The dependent variables were placement error, number of focus shifts, and completion time. Furthermore, workspace efficiency was quantified as the number and duration of potential surgeontablet collisions avoided by the ARHMD. Results: The ARHMD condition yielded smaller placement errors (Task1: 45%, P < 0.001; Task2: 14%, P = 0.01), fewer focus shifts (Task1: 93%, P < 0.001; Task2: 88%, P = 0.0039), and longer completion times (Task1: 31%, P < 0.001; Task2: 24%, P = 0.013). Furthermore, the ARHMD avoided potential tablet collisions (4.8 for 3.2 seconds in Task1; 3.8 for 1.3 seconds in Task2). Conclusion: The ARHMD system promises to improve accuracy and to eliminate focus shifts in surgical telementoring. Because ARHMD participants were able to refine their execution of instructions, task completion time increased. Unlike a tablet system, the ARHMD does not require modifying natural motions to avoid collisions.

Most common approaches to one-shot gesture recognition have leveraged mainly conventional machine learning solutions and image based data augmentation techniques, ignoring the mechanisms that are used by humans to perceive and execute gestures, a key contextual component in this process. The novelty of this work consists on modeling the process that leads to the creation of gestures, rather than observing the gesture alone. In this approach, the context considered involves the way in which humans produce the gestures - the kinematic and biomechanical characteristics associated with gesture production and execution. By understanding the main "modes" of variation we can replicate the single observation many times. Consequently, the main strategy proposed in this paper includes generating a data set of human-like examples based on "naturalistic" features extracted from a single gesture sample while preserving fundamentally human characteristics like visual saliency, smooth transitions and economy of motion. The availability of a large data set of realistic samples allows the use state-of-the-art classifiers for further recognition. Several classifiers were trained and their recognition accuracies were assessed and compared to previous one-shot learning approaches. An average recognition accuracy of 95% among all classifiers highlights the relevance of keeping the human "in the loop" to effectively achieve one-shot gesture recognition.

An experiment was conducted where a robotic platform performs artificially generated gestures and both trained classifiers and human participants recognize. Classification accuracy is evaluated through a new metric of coherence in gesture recognition between humans and robots. Experimental results showed an average recognition performance of 89.2% for the trained classifiers and 92.5% for the participants. Coherence in one-shot gesture recognition was determined to be gamma = 93.8%. This new metric provides a quantifier for validating how realistic the robotic generated gestures are.