Humans form sequences of event models—representations of the immediate situation—to predict how activity will unfold. Multiple mechanisms have been proposed for how the cognitive system determines when to segment the stream of behavior and switch from one active event model to another. Here we use a large-scale naturalistic dataset to compare two candidate gating mechanisms for model updating: prediction uncertainty and prediction error. We constructed a computational model combining a recurrent neural network for short-term dynamics with Bayesian inference over event types for event-to-event transitions. This architecture learns event schemas representing knowledge about event types and uses them, along with observed perceptual information, to construct a series of event models. This architecture was trained on one pass through an 18-hour corpus of naturalistic human activity. Another 3.5 hours of activities were used to test each variant for agreement with human segmentation and categorization. The architecture was able to learn to predict human activity, and it developed human-like segmentation and categorization. We then compared two variants of this architecture designed to better emulate human event segmentation: one transitioned when the active event schema produced high uncertainty in its prediction; the other transitioned when the active event schema produced a large prediction error. The variant that transitioned from one active event schema to another based on prediction uncertainty provided the closest match to human segmentation and forming human-like event categories—despite being given no feedback about segmentation or categorization. These results establish that event model transitioning based on prediction uncertainty can naturally reproduce two important features of human event comprehension.

This paper introduces the notion of danger awareness in the context of Human-Robot Interaction (HRI), which decodes whether a human is aware of the existence of the robot, and illuminates whether the human is willing to engage in enforcing the safety. This paper also proposes a method to quantify this notion as a single binary variable, so-called danger awareness coefficient. By analyzing the effect of this coefficient on the human's actions, an online Bayesian learning method is proposed to update the belief about the value of the coefficient. It is shown that based upon the danger awareness coefficient and the proposed learning method, the robot can build a predictive human model to anticipate the human's future actions. In order to create a communication channel between the human and the robot, to enrich the observations and get informative data about the human, and to improve the efficiency of the robot, the robot is equipped with a danger signaling system. A predictive planning scheme, coupled with the predictive human model, is also proposed to provide an efficient and Probabilistically safe plan for the robot. The effectiveness of the proposed scheme is demonstrated through simulation studies on an interaction between a self-driving car and a pedestrian.

This article introduces the notion of carelessness level into robot action planners such that the safety and efficiency are optimized. The core idea is to make the robot’s plan less sensitive to the behavior of careless humans who may inattentively violate safety constraints and degrade efficiency. More precisely, our planner reduces the opportunities given to the careless humans to put themselves in danger and hamper the efficiency of the robot’s plan. The effectiveness of the proposed planner is demonstrated through simulation studies on a packaging line and on a collaborative assembly line. Results show that the proposed scheme can improve efficiency and safety in both examples.

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

The explicit reference governor (ERG) is an add-on unit that provides constraint handling capabilities to prestabilized systems. The basic idea behind this approach is to translate state and input constraints into an upper-bound on the value of the Lyapunov function, which is then enforced by suitably manipulating the derivative of the auxiliary reference. This paper extends the ERG approach to deal with time-varying linear constraints. In particular, it is shown that under a certain condition on the rate of change of the constraints and the strength of stability of the prestabilized system (i.e., the speed of the dynamics of the prestabilized system), it is possible to deploy the ERG approach to handle time-varying constraints. To do so, components of the ERG approach are redefined, and constraints satisfaction and convergence properties are proven analytically. The effectiveness of the proposed scheme is demonstrated through a simulation study on filming a movie scene with a drone.

This work introduces an affordance characterization employing mechanical wrenches as a metric for predicting and planning with workspace affordances. Although affordances are a commonly used high-level paradigm for robotic task-level planning and learning, the literature has been sparse regarding how to characterize the agent in this object-agent-environment framework. In this work, we propose decomposing a behavior into a vocabulary of characteristic requirements and capabilities that are suitable to predict the affordances of various parts of the workspace. Specifically, we investigate mechanical wrenches as a viable representation of these affordance requirements and capabilities. We then use this vocabulary in a planning system to compose complex motions from simple behavior types in continuous space. The utility of the framework for complex planning is demonstrated on example scenarios both in simulation and with real-world industrial manipulators.

•SIN parses structured activity sequence (or time series data in general).•SIN is not a time sliced graphical model.•SIN is equivalent to a left-right segmental model (HSMM) and allows exact inference.

Segmenting and tracking multiple humans is a challenging problem in complex situations in which extended occlusion, shadow and/or reflection exists. We tackle this problem with a 3D model-based approach. This method includes two stages, segmentation (detection) and tracking. Human hypotheses are generated by shape analysis of the foreground blobs using human shape model. The segmented human hypotheses are tracked with a Kalman filter with explicit handling of occlusion. Hypotheses are verified while they are tracked for the first second or so. The verification is done by walking recognition using an articulated human walking model. We propose a new method to recognize walking using motion template and temporal integration. Experiments show that our approach works robustly in very challenging Sequences.

We present a general system with a focus on addressing three events of the 2013 DARPA Robotics Challenge (DRC) trials: debris clearing, door opening, and wall breaking. Our hardware platform is DRC‐HUBO, a redesigned model of the HUBO2+ humanoid robot developed by KAIST and Rainbow, Inc. Our system allowed a trio of operators to coordinate a 32 degree‐of‐freedom robot on a variety of complex mobile manipulation tasks using a single, unified approach. In addition to descriptions of the hardware and software, and results as deployed on the DRC‐HUBO platform, we present some qualitative analysis of lessons learned from this demanding and difficult challenge.

We address the problem of predicting the physical state of a an agent performing a known activity. In particular we are interested in predicting human movement during complex composite activities. Our proposed framework combines a graphical model that extends the Sequential Interval Network (SIN) [1] for modeling global temporal structure of activities with a low level dynamic system for modeling the dynamics of the physical state. Specifically, two sets of new hidden state variables are added: one with respect to the temporal structure and one with respect to time. A mapping factor is defined to ensure these variables values remain consistent and hence allows fusing the two sources of information. We then derive an inference algorithm for computing the posterior densities of the hidden variables. The system can run in an on-line predictive mode to recognize on-going activity and make predictions arbitrarily far in the future during execution of the activity. Experiments illustrate that the long term prediction performance benefits from the knowledge about the temporal structure of the activity while short term prediction performance is improved by incorporating the dynamics of physical state.

A representation for structured activities is developed that allows a robot to probabilistically infer which task actions a human is currently performing and to predict which future actions will be executed and when they will occur. The goal is to enable a robot to anticipate collaborative actions in the presence of uncertain sensing and task ambiguity. The system can represent multi-path tasks where the task variations may contain partially ordered actions or even optional actions that may be skipped altogether. The task is represented by an AND-OR tree structure from which a probabilistic graphical model is constructed. Inference methods for that model are derived that support a planning and execution system for the robot which attempts to minimize a cost function based upon expected human idle time. We demonstrate the theory in both simulation and actual human-robot performance of a two-way-branch assembly task. In particular we show that the inference model can robustly anticipate the actions of the human even in the presence of unreliable or noisy detections because of its integration of all its sensing information along with knowledge of task structure.

This paper introduces the concept of rationally safe robot behavior as behavior which is safe with respect to a rational model of human behavior. We discuss how to model rational human behavior by fully accounting for a range of acceptable human variations. We propose modelling the system as a form of robust controller, and frame the robot’s goal as an optimal control problem where the human’s possible actions are probabilistic and/or possibilistic. A simple example is provided which applies the framework to a linear quadratic problem.

We propose a probabilistic method for parsing a temporal sequence such as a complex activity defined as composition of sub-activities/actions. The temporal structure of the high-level activity is represented by a string-length limited stochastic context-free grammar. Given the grammar, a Bayes network, which we term Sequential Interval Network (SIN), is generated where the variable nodes correspond to the start and end times of component actions. The network integrates information about the duration of each primitive action, visual detection results for each primitive action, and the activity's temporal structure. At any moment in time during the activity, message passing is used to perform exact inference yielding the posterior probabilities of the start and end times for each different activity/action. We provide demonstrations of this framework being applied to vision tasks such as action prediction, classification of the high-level activities or temporal segmentation of a test sequence, the method is also applicable in Human Robot Interaction domain where continual prediction of human action is needed.

A novel representation for the human component of multi-step, human-robot collaborative activity is presented. The goal of the system is to predict in a probabilistic manner when the human will perform different subtasks that may require robot assistance. The representation is a graphical model where the start and end of each subtask is explicitly represented as a probabilistic variable conditioned upon prior intervals. This formulation allows the inclusion of uncertain perceptual detections as evidence to drive the predictions. Next, given a cost function that describes the penalty for different wait times, we develop a planning algorithm which selects robot-actions that minimize the expected cost based upon the distribution over predicted human-action timings. We demonstrate the approach in assembly tasks where the robot must provide the right part at the right time depending upon the choices made by the human operator during the assembly.

We present a method by which a robot learns to predict effective push-locations as a function of object shape. The robot performs push experiments at many contact locations on multiple objects and records local and global shape features at each point of contact. The robot observes the outcome trajectories of the manipulations and computes a novel push-stability score for each trial. The robot then learns a regression function in order to predict push effectiveness as a function of object shape. This mapping allows the robot to select effective push locations for subsequent objects whether they are previously manipulated instances, new instances from previously encountered object classes, or entirely novel objects. In the totally novel object case, the local shape property coupled with the overall distribution of the object allows for the discovery of effective push locations. These results are demonstrated on a mobile manipulator robot pushing a variety of household objects on a tabletop surface.

We present a method by which a robot learns to predict effective contact locations for pushing as a function of object shape. The robot performs push experiments at many contact locations on multiple objects and records local and global shape features at each point of contact. Each trial attempts to either push the object in a straight line or to rotate the object to a new orientation. The robot observes the outcome trajectories of the manipulations and computes either a push-stability or rotate-push score for each trial. The robot then learns a regression function for each score in order to predict push effectiveness as a function of object shape. With this mapping, the robot can infer effective push locations for subsequent objects from their shapes, regardless of whether they belong to a previously encountered object class. These results are demonstrated on a mobile manipulator robot pushing a variety of household objects on a tabletop surface.