Iterative Repair of Social Robot Programs from Implicit User Feedback via Bayesian Inference

robotics science and systems, 2020.

Cited by: 0|Views16
Weibo:
This paper presented an iterative program repair approach for creating robust and fluent social robot programs without painstakingly tuning program parameters

Abstract:

Creating natural and autonomous interactions with social robots requires rich, multi-modal sensory input from the user. Writing interactive robot programs that make use of this input can demand tedious and error-prone tuning of program parameters, such as tuning thresholds on noisy sensory streams for detecting whether the robot's user is...More

Code:

Data:

0
Full Text
Bibtex
Weibo
Introduction
  • The past decade has been marked by the rise of social robots, with soaring interest from start-ups, large corporations, and researchers alike.
  • Programming a social robot for a particular application still requires specialized software development skills.
  • A children’s hospital that wants to incorporate social robots into autismrelated therapy can purchase a robot platform, but it will struggle to find qualified programmers to customize the robot for their specific needs and incur additional costs every time the robots need re-programming.
Highlights
  • The past decade has been marked by the rise of social robots, with soaring interest from start-ups, large corporations, and researchers alike
  • (1) The programmer creates a program sketch representing a finite state machine, or Finite State Machines, (2) the program is executed on the robot, and a user interacts with the robot, (3) the program is automatically repaired based on annotations over the interaction trace that are either provided by the programmer or derived from implicit feedback in user inputs, (4) steps 2-3 are repeated until a satisfactory program is obtained
  • We present SoRTSketch (Social Robot Program Transition Sketch Language), a domainspecific language for sketching social robot behaviors based on finite state machines (FSMs) with transitions that are not fully specified
  • This paper presented an iterative program repair approach for creating robust and fluent social robot programs without painstakingly tuning program parameters
  • Our approach helps programmers to implement robot programs without complete low-level details and incrementally repair programs over time using corrective feedback provided by the programmer or the robot’s user
  • We examined the feasibility and effectiveness of our approach via two experiments involving human simulators and 10 real human users across three representative social robot use cases; results demonstrate the utility and potential of the proposed approach
Methods
  • The authors first simulated 100 input traces and ground truth state traces as a test dataset.
  • The authors simulated a new FSM input trace and desired state trace pair and applied the IterativeRepair (Alg. 2) and IterativeBayesRepair (Alg. 3) to acquire a set of repaired parameters.
  • The 10 participants were recruited from the University of Washington (UW) campus community through mailing lists
  • Their average age was 23.2 (SD = 5.58).
  • All transition parameters were initialized to make the robot not respond to users except when they tapped buttons like “Go back.”
Results
  • IterativeRepair and IterativeBayesRepair algorithms performed in terms of interaction quality(Fig. 5left)
  • Both algorithms’ average percentage overlaps increased monotonically over the five iterations, reached average overlaps above 95% by the third iteration, and produced standard deviations lower than 9% by the final iteration in all three tasks.
  • Some participants frequently moved out of the robot’s field of view, while others did not tap the “Go back” button when the robot skipped to the question; both behaviors caused the algorithm to find ineffective or even counterproductive parameters
Conclusion
  • DISCUSSION AND FUTURE

    WORK

    The authors believe the results demonstrate the usefulness of the proposed approach, with some limitations.

    In the human experiment results, the authors observed that three users did not fix the incorrect transition that occurred during the open-ended Q&A interaction.
  • Finding a more scalable inference algorithm or one that does not require any prior distributions for hole variables may increase usability and applicability.This paper presented an iterative program repair approach for creating robust and fluent social robot programs without painstakingly tuning program parameters.
  • The authors examined the feasibility and effectiveness of the approach via two experiments involving human simulators and 10 real human users across three representative social robot use cases; results demonstrate the utility and potential of the proposed approach
Summary
  • Introduction:

    The past decade has been marked by the rise of social robots, with soaring interest from start-ups, large corporations, and researchers alike.
  • Programming a social robot for a particular application still requires specialized software development skills.
  • A children’s hospital that wants to incorporate social robots into autismrelated therapy can purchase a robot platform, but it will struggle to find qualified programmers to customize the robot for their specific needs and incur additional costs every time the robots need re-programming.
  • Methods:

    The authors first simulated 100 input traces and ground truth state traces as a test dataset.
  • The authors simulated a new FSM input trace and desired state trace pair and applied the IterativeRepair (Alg. 2) and IterativeBayesRepair (Alg. 3) to acquire a set of repaired parameters.
  • The 10 participants were recruited from the University of Washington (UW) campus community through mailing lists
  • Their average age was 23.2 (SD = 5.58).
  • All transition parameters were initialized to make the robot not respond to users except when they tapped buttons like “Go back.”
  • Results:

    IterativeRepair and IterativeBayesRepair algorithms performed in terms of interaction quality(Fig. 5left)
  • Both algorithms’ average percentage overlaps increased monotonically over the five iterations, reached average overlaps above 95% by the third iteration, and produced standard deviations lower than 9% by the final iteration in all three tasks.
  • Some participants frequently moved out of the robot’s field of view, while others did not tap the “Go back” button when the robot skipped to the question; both behaviors caused the algorithm to find ineffective or even counterproductive parameters
  • Conclusion:

    DISCUSSION AND FUTURE

    WORK

    The authors believe the results demonstrate the usefulness of the proposed approach, with some limitations.

    In the human experiment results, the authors observed that three users did not fix the incorrect transition that occurred during the open-ended Q&A interaction.
  • Finding a more scalable inference algorithm or one that does not require any prior distributions for hole variables may increase usability and applicability.This paper presented an iterative program repair approach for creating robust and fluent social robot programs without painstakingly tuning program parameters.
  • The authors examined the feasibility and effectiveness of the approach via two experiments involving human simulators and 10 real human users across three representative social robot use cases; results demonstrate the utility and potential of the proposed approach
Tables
  • Table1: Percentage overlaps before and after the repair, and the number of human inputs measured in the behavior adaptation experiment
Download tables as Excel
Related work
  • The robotics community has long been interested in applying formal methods, such as verification and synthesis, to robotics problems. Techniques in formal verification have been applied to assess the correctness of concurrent and time-critical programs [12] that check both general and application-specific properties [26]. Researchers also used program synthesis to find a plan or a controller for navigation [13], mobile manipulation [27], or multi-robot planning problems [40] that satisfy specifications often expressed in a temporal logic language. Other work explored alternative specification languages, such as structured natural languages [20] or adaptations of existing robot programs in new environments interactively with a programmer, e.g., for robot soccer [18] or tabletop manipulation [7]. Most recently, Hammond et al introduced a system that can automatically recover from errors incurred while running end-user created mobile service robot programs [15]. Our work most closely relates to that of Holtz et al on automatically repairing robot-soccer playing programs, which are represented as FSMs, using sparse state corrections from a programmer with a MaxSMT solver [18, 19]. However, we focus on repairing social robot programs using Bayesian inference and feedback provided by the human who is interacting with the robot.
Funding
  • This work was supported by the National Science Foundation, Awards IIS-1552427 “CAREER: End-User Programming of General-Purpose Robots” and IIS-1925043 “NRI: INT: COLLAB: Program Verification and Synthesis for Collaborative Robots.” We thank Rajesh P.N
Reference
  • Philipp Allgeuer and Sven Behnke. Hierarchical and state-based architectures for robot behavior planning and control. arXiv preprint arXiv:1809.11067, 2018.
    Findings
  • Sean Andrist, Xiang Zhi Tan, Michael Gleicher, and Bilge Mutlu. Conversational gaze aversion for humanlike robots. In International Conference on Human-Robot Interaction, pages 25–3ACM, 2014.
    Google ScholarLocate open access versionFindings
  • Emilia I Barakova, Jan CC Gillesen, Bibi EBM Huskens, and Tino Lourens. End-user programming architecture facilitates the uptake of robots in social therapies. Robotics and Autonomous Systems, 61(7):704–713, 2013.
    Google ScholarLocate open access versionFindings
  • Jonathan Bohren and Steve Cousins. The smach highlevel executive [ros news]. Robotics & Automation Magazine, 17(4):18–20, 2010.
    Google ScholarLocate open access versionFindings
  • Dan Bohus and Eric Horvitz. Managing human-robot engagement with forecasts and... um... hesitations. In International Conference on Multimodal Interaction, pages 2–9. ACM, 2014.
    Google ScholarLocate open access versionFindings
  • Rafael H Bordini, Michael Fisher, and Maarten Sierhuis. Formal verification of human-robot teamwork. In International Conference on Human-Robot Interaction, pages 267–268. ACM/IEEE, 2009.
    Google ScholarLocate open access versionFindings
  • Adrian Boteanu, Jacob Arkin, Siddharth Patki, Thomas Howard, and Hadas Kress-Gazit. Robot-initiated specification repair through grounded language interaction. arXiv preprint arXiv:1710.01417, 2017.
    Findings
  • Rodney Brooks. A robust layered control system for a mobile robot. Journal on robotics and automation, 2(1): 14–23, 1986.
    Google ScholarLocate open access versionFindings
  • Sebastian G Brunner, Franz Steinmetz, Rico Belder, and Andreas Domel. Rafcon: A graphical tool for engineering complex, robotic tasks. In International Conference on Intelligent Robots and Systems, pages 3283–3290. IEEE, 2016.
    Google ScholarLocate open access versionFindings
  • Crystal Chao and Andrea L Thomaz. Controlling social dynamics with a parametrized model of floor regulation. Journal of Human-Robot Interaction, 2(1):4–29, 2013.
    Google ScholarLocate open access versionFindings
  • James Diprose, Bruce MacDonald, John Hosking, and Beryl Plimmer. Designing an api at an appropriate abstraction level for programming social robot applications. Journal of Visual Languages & Computing, 39:22–40, 2017.
    Google ScholarLocate open access versionFindings
  • Bernard Espiau, Konstantinos Kapellos, and Muriel Jourdan. Formal verification in robotics: Why and how? In Robotics Research, pages 225–236.
    Google ScholarLocate open access versionFindings
  • Georgios E Fainekos, Hadas Kress-Gazit, and George J Pappas. Temporal logic motion planning for mobile robots. In International Conference on Robotics and Automation, pages 2020–2025. IEEE, 2005.
    Google ScholarLocate open access versionFindings
  • Dylan F Glas, Takayuki Kanda, and Hiroshi Ishiguro. Human-robot interaction design using interaction composer eight years of lessons learned. In International Conference on Human-Robot Interaction, pages 303– 310. ACM/IEEE, 2016.
    Google ScholarLocate open access versionFindings
  • Jenna Claire Hammond, Joydeep Biswas, and Arjun Guha. Automatic failure recovery for end-user programs on service mobile robots. arXiv preprint arXiv:1909.02778, 2019.
    Findings
  • Guy Hoffman. Evaluating fluency in human–robot collaboration. IEEE Transactions on Human-Machine Systems, 49(3):209–218, 2019.
    Google ScholarLocate open access versionFindings
  • Guy Hoffman, Maya Cakmak, and Crystal Chao. Timing in human-robot interaction. In International Conference on Human-robot interaction, pages 509–510. ACM/IEEE, 2014.
    Google ScholarLocate open access versionFindings
  • Jarrett Holtz, Arjun Guha, and Joydeep Biswas. Interactive robot transition repair with smt. In International Joint Conference on Artificial Intelligence, pages 4905– 4911.
    Google ScholarLocate open access versionFindings
  • Jarrett Holtz, Arjun Guha, and Joydeep Biswas. Smt-based robot transition repair. arXiv preprint arXiv:2001.04397, 2020.
    Findings
  • Hadas Kress-Gazit, Georgios E Fainekos, and George J Pappas. Translating structured english to robot controllers. Advanced Robotics, 22(12):1343–1359, 2008.
    Google ScholarLocate open access versionFindings
  • Alap Kshirsagar, Hadas Kress-Gazit, and Guy Hoffman. Specifying and synthesizing human-robot handovers. In International Conference on Intelligent Robots and Systems. IEEE/RSJ, 2019.
    Google ScholarLocate open access versionFindings
  • Alyssa Kubota, Emma IC Peterson, Vaishali Rajendren, Hadas Kress-Gazit, and Laurel D Riek. Jessie: Synthesizing social robot behaviors for personalized neurorehabilitation and beyond.
    Google ScholarFindings
  • Wenchao Li, Dorsa Sadigh, S Shankar Sastry, and Sanjit A Seshia. Synthesis for human-in-the-loop control systems. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 470–484, 2014.
    Google ScholarLocate open access versionFindings
  • Marco Manca, Fabio Paterno, and Carmen Santoro. Analyzing trigger-action programming for personalization of robot behaviour in iot environments. In International Symposium on End User Development, pages 100–114, 2019.
    Google ScholarLocate open access versionFindings
  • Spyros Maniatopoulos, Philipp Schillinger, Vitchyr Pong, David C Conner, and Hadas Kress-Gazit. Reactive highlevel behavior synthesis for an atlas humanoid robot. In International Conference on Robotics and Automation, pages 4192–4199. IEEE, 2016.
    Google ScholarLocate open access versionFindings
  • Alvaro Miyazawa, Pedro Ribeiro, Wei Li, Ana Cavalcanti, and Jon Timmis. Automatic property checking of robotic applications. In International Conference on Intelligent Robots and Systems, pages 3869–3876. IEEE/RSJ, 2017.
    Google ScholarLocate open access versionFindings
  • Srinivas Nedunuri, Sailesh Prabhu, Mark Moll, Swarat Chaudhuri, and LE Kavraki. Smt-based synthesis of integrated task and motion plans for mobile manipulation. In International Conference on Robotics and Automation. IEEE, 2014.
    Google ScholarLocate open access versionFindings
  • Hai Nguyen, Matei Ciocarlie, Kaijen Hsiao, and Charles C Kemp. Ros commander (rosco): Behavior creation for home robots. In International Conference on Robotics and Automation, pages 467–474. IEEE, 2013.
    Google ScholarLocate open access versionFindings
  • Scott Niekum, Sarah Osentoski, George Konidaris, Sachin Chitta, Bhaskara Marthi, and Andrew G Barto. Learning grounded finite-state representations from unstructured demonstrations. The International Journal of Robotics Research, 34(2):131–157, 2015.
    Google ScholarLocate open access versionFindings
  • D Oved. Real-time human pose estimation in the browser with tensorflow.js. TensorFlow Medium, May, 2018.
    Google ScholarFindings
  • David Porfirio, Allison Sauppe, Aws Albarghouthi, and Bilge Mutlu. Authoring and verifying human-robot interactions. In Symposium on User Interface Software and Technology, pages 75–86. ACM, 2018.
    Google ScholarLocate open access versionFindings
  • David Porfirio, Evan Fisher, Allison Sauppe, Aws Albarghouthi, and Bilge Mutlu. Bodystorming human-robot interactions. In Symposium on User Interface Software and Technology, 2019.
    Google ScholarLocate open access versionFindings
  • Emmanuel Pot, Jerome Monceaux, Rodolphe Gelin, and Bruno Maisonnier. Choregraphe: a graphical tool for humanoid robot programming. In The International Symposium on Robot and Human Interactive Communication, pages 46–51. IEEE, 2009.
    Google ScholarLocate open access versionFindings
  • Mattia Racca, Ville Kyrki, and Maya Cakmak. Interactive tuning of robot program parameters via expected divergence maximization.
    Google ScholarFindings
  • Charles Rich, Brett Ponsler, Aaron Holroyd, and Candace L Sidner. Recognizing engagement in humanrobot interaction. In International Conference on HumanRobot Interaction, pages 375–382. ACM/IEEE, 2010.
    Google ScholarLocate open access versionFindings
  • Allison Sauppeand Bilge Mutlu. Robot deictics: How gesture and context shape referential communication. In International Conference on Human-Robot Interaction, pages 342–349. ACM, 2014.
    Google ScholarLocate open access versionFindings
  • Philipp Schillinger, Stefan Kohlbrecher, and Oskar von Stryk. Human-robot collaborative high-level control with application to rescue robotics. In International Conference on Robotics and Automation, pages 2796– 2802. IEEE, 2016.
    Google ScholarLocate open access versionFindings
  • Armando Solar-Lezama. Program sketching. International Journal on Software Tools for Technology Transfer, 15(5-6):475–495, 2013.
    Google ScholarLocate open access versionFindings
  • Andre Staltz. Cycle.js. https://cycle.js.org/, 2016. Accessed:2019-10-1.
    Findings
  • Jana Tumova and Dimos V Dimarogonas. Multi-agent planning under local ltl specifications and event-based synchronization. Automatica, 70:239–248, 2016.
    Google ScholarLocate open access versionFindings
  • Matt Webster, Clare Dixon, Michael Fisher, Maha Salem, Joe Saunders, Kheng Lee Koay, Kerstin Dautenhahn, and Joan Saez-Pons. Toward reliable autonomous robotic assistants through formal verification: a case study. Transactions on Human-Machine Systems, 46(2):186– 196, 2015.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments