In this paper we present a framework to learn skills from human demonstrations in the form of geometric nullspaces, which can be executed using a robot. We collect data of human demonstrations, fit geometric nullspaces to them, and also infer their corresponding geometric constraint models. These geometric constraints provide a powerful mathematical model as well as an intuitive representation of the skill in terms of the involved objects. To execute the skill using a robot, we combine this geometric skill description with the robot's kinematics and other environmental constraints, from which poses can be sampled for the robot's execution. The result of our framework is a system that takes the human demonstrations as input, learns the underlying skill model, and executes the learnt skill with different robots in different dynamic environments. We evaluate our approach on a simulated industrial robot, and execute the final task on the iCub humanoid robot.

This paper proposed a motion planning and control approach for real-time collision avoidance of high degree-of-freedom (DoF) robots. It constructs a 3D visibility graph by representing external obstacles with discrete polyhedrons and searches for the shortest path on it. To consider the complex internal structure of high DoF robots and take full advantage of redundancy, inverse kinematics is solved as an optimization problem which allows for flexible cost functions and constraints. The distance between the robot and external obstacles is treated as an extra constraint in the optimization which works together with visibility graph for collision avoidance. At the end, a pick-place task was successfully performed among obstacles by a simulated dual-arm quadrupedal robot considering the whole-body kinematics and stability, which demonstrated the feasibility of the proposed method.

Vitreoretinal (VR) surgery is typical microsurgery with delicate and complex surgical procedures. The vision-based navigation for robot-assisted VR surgery has not been fully exploited because of the challenges that arise from illumination, high precision, and safety assessments. This paper presents a novel method to estimate the 6DOF needle pose specifically for the application of robotic intraocular needle navigation using optical coherence tomography (OCT) volumes. The key ingredients of the proposed method are (1) 3D needle point cloud segmentation in OCT volume and (2) needle point cloud 6DOF pose estimation using a modified iterative closest point (ICP) algorithm. To address the former, a voting mechanism with geometric features of the needle is utilized to robustly segment the needle in OCT volume. Afterward, the CAD model of the needle point cloud is matched with the segmented needle point cloud to estimate the 6DOF needle pose with a proposed shift-rotate ICP (SR-ICP). This method is evaluated by the existing ophthalmic robot on ex-vivo pig eyes. The quantitative and qualitative results are evaluated and presented for the proposed method.

The sense of touch is arguably the first human sense to develop. Empowering robots with the sense of touch may augment their understanding of interacted objects and the environment beyond standard sensory modalities (e.g., vision). This paper investigates the effect of hybridizing touch and sliding movements for tactile-based texture classification. We develop three machine-learning methods within a framework to discriminate between surface textures; the first two methods use hand-engineered features, whilst the third leverages convolutional and recurrent neural network layers to learn feature representations from raw data. To compare these methods, we constructed a dataset comprising tactile data from 23 textures gathered using the iCub platform under a loosely constrained setup, i.e., with nonlinear motion. In line with findings from neuroscience, our experiments show that a good initial estimate can be obtained via touch data, which can be further refined via sliding; combining both touch and sliding data results in 98% classification accuracy over unseen test data.

This paper introduces an end-to-end learning approach based on Reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) for a multi-layered spiking neural network (SNN). As a case study, a snake-like robot is used as an agent to perform target tracking tasks on the basis of our proposed approach. Since the key of R-STDP is to use rewards to modulate synapse strengthens, we first propose a general way to propagate the reward back through a multi-layered SNN. Upon the proposed approach, we build up an SNN controller that drives a snake-like robot for performing target tracking tasks. We demonstrate the practicability and advantage of our approach in terms of lateral tracking accuracy by comparing it to other state-of-the-art learning algorithms for SNNs based on R-STDP.

Spiking neural networks (SNNs) offer many advantages over traditional artificial neural networks (ANNs) such as biological plausibility, fast information processing, and energy efficiency. Although SNNs have been used to solve a variety of control tasks using the Spike-Timing-Dependent Plasticity (STDP) learning rule, existing solutions usually involve hard-coded network architectures solving specific tasks rather than solving different kinds of tasks generally. This results in neglecting one of the biggest advantages of ANNs, i.e., being general-purpose and easy-to-use due to their simple network architecture, which usually consists of an input layer, one or multiple hidden layers and an output layer. This paper addresses the problem by introducing an end-to-end learning approach of spiking neural networks constructed with one hidden layer and reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) synapses in an all-to-all fashion. We use the supervised reward-modulated Spike-Timing-Dependent-Plasticity learning rule to train two different SNN-based sub-controllers to replicate a desired obstacle avoiding and goal approaching behavior, provided by pre-generated datasets. Together they make up a target-reaching controller, which is used to control a simulated mobile robot to reach a target area while avoiding obstacles in its path. We demonstrate the performance and effectiveness of our trained SNNs to achieve target reaching tasks in different unknown scenarios.

In this paper, we show that visual servoing tasks can be integrated into a prioritized constraint-based torque control framework. This framework can achieve real time control of robots performing strict prioritized motion and forces tasks. A null-space projector is used to combine the tasks with different priorities. The visual servoing is considered just another task formulated with constraints in the whole body control framework. Several relevant constraints (i.e., motion constraints, joint limits) are tested to evaluate the control framework. Further, we evaluate the proposed approach in typical industrial robotics applications: grasping of cylindrical objects, and two position/force control applications (Erasing and Peg-in-Hole).

Recent research progresses in speech recognition, text-to-speech, natural language understanding, or dialog management components are improving the way humans interact with advanced robot machines. However, far from being solved, we are just starting the process of creating meaningful multimodal platforms that can allow operators to use and control industrial robots through spoken dialogue. This paper describes our ongoing efforts on creating a modular platform that combines different technologies to cover typical requirements in an industrial setting, i.e. robust speech recognition, low level skill functions to operate the robot, recommendations and validation procedures to setup parameters, combination of audio-visual information for challenging environments, integration of domain-knowledge by means of an ontology, a flexible definition of the dialog model and natural language rules, as well as a test and control interface to quickly check the functionality of each module during development and operation. All platform modules are intercommunicated by the ROS operative system which allows the integration of external plugins and modules easily. Finally, a preliminary user study with IT experts simulating a welding task has been doing giving us clues on what should be the focus of our next developments.

In this paper, we propose a framework for prioritized constraint-based specification of robot tasks. This framework is integrated with a cognitive robotic system based on semantic models of processes, objects, and workcells. The target is to enable intuitive (re-)programming of robot tasks, in a way that is suitable for non-expert users typically found in SMEs. Using CAD semantics, robot tasks are specified as geometric inter-relational constraints. During execution, these are combined with constraints from the environment and the workcell, and solved in real-time. Our constraint model and solving approach supports a variety of constraint functions that can be non-linear and also include bounds in the form of inequalities, e.g., geometric inter-relations, distance, collision avoidance and posture constraints. It is a hierarchical approach where priority levels can be specified for the constraints, and the nullspace of higher priority constraints is exploited to optimize the lower priority constraints. The presented approach has been applied to several typical industrial robotic use-cases to highlight its advantages compared to other state-of-the-art approaches.

We present an approach to control a 6-degree-of-freedom (DOF) manipulator using an uncalibrated visual servoing (VS) approach that addresses the challenges of choosing proper image features for target objects and designing a VS controller to enhance the tracking performance. The main contribution of this paper is the definition of a new virtual visual space (image space). A novel stereo camera model employing virtual orthogonal cameras is used to map 6-D poses from Cartesian space to this virtual visual space. Each component of the 6-D pose vector defined in this virtual visual space is linearly independent, leading to a full-rank 6 × 6 image Jacobian matrix, which allows avoiding classical problems, such as image space singularities and local minima. Furthermore, the control for rotational and translational motion of robot is decoupled due to the diagonal image Jacobian. Finally, simulation results with an eye-to-hand robotic system confirm the improvement in controller stability and motion performance with respect to conventional VS approaches. Experimental results on a 6-DOF industrial robot are provided to illustrate the effectiveness of the proposed method and the feasibility of using this method in practical scenarios.

In this paper, an approach for matching of primitive shapes detected from point clouds, to boundary representations of primitive shapes contained in CAD models of objects/workpieces is presented. The primary target application is object detection and pose estimation from noisy RGBD sensor data. This approach can also be used to determine incomplete object poses, including those of symmetrical objects. Detection and reasoning about these under-specified object poses is useful in several practical applications such as robotic manipulation, which are also presented in this paper.

To synthesize whole-body behaviors interactively, multiple tasks and constraints need to be simultaneously satisfied, including those that guarantee the constraints imposed by the robot's structure and the external environment. In this paper, we present a prioritized, multiple-task control framework that is able to control forces in systems ranging from humanoids to industrial robots. Priorities between tasks are accomplished through null-space projection. Several relevant constraints (i.e., motion constraints, joint limits, force control) are tested to evaluate the control framework. Further, we evaluate the proposed approach in two typical industrial robotics applications: grasping of cylindrical objects and welding.

This paper introduces 6 new image features to provide a solution to the open problem of uncalibrated 6D image-based visual servoing for robot manipulators, where the goal is to control the 3D position and orientation of the robot end-effector using visual feedback. One of the main contributions of this article is a novel stereo camera model which employs virtual orthogonal cameras to map 6D Cartesian poses defined in the Task space to 6D visual poses defined in a Virtual Visual space (Image space). This new model is used to compute a full-rank square Image Jacobian matrix (J img ), which solves several common problems exhibited by the classical image Jacobians, e.g., Image space singularities and local minima. This Jacobian is a fundamental key for the image-based controller design, where a chattering-free adaptive second order sliding mode is employed to track 6D visual motions for a robot manipulator. Exponential convergence of errors in both spaces without local minima are demonstrated. The complete control system is experimentally evaluated on a real industrial robot. The robustness of the control scheme is evaluated for cases where the extrinsic parameters of the uncalibrated stereo camera system are changed on-line and unknown when the stereo system is manually moved to obtain a clearer view of the task.

In this paper, we present an uncalibrated position-based fixed-camera Visual Servoing for robot manipulators, where the goal is to track the 3D position and orientation of the target. The stereo system with 2 USB cameras is uncalibrated with respect to the robot base frame and the transformation between them is estimated on-line while performing the task. Dynamic impedance control is designed to generate a dynamic trajectory for the robot manipulator considering the dynamic environment constraints, such as: robot singularities avoidance and (self-/obstacle-) collision avoidance. Experiments have been carried out to verify performance of the proposed system on a real industrial robot, where the calibration estimation process and handling of all uncertainties in the environment are demonstrated. Moreover the uncalibrated stereo camera system can be manually moved while performing the task in order to obtain a clearer view and the re-calibration is performed automatically and on-line.

In this paper, an object recognition and pose estimation approach based on constraints from primitive shape matching is presented. Additionally, an approach for primitive shape detection from point clouds using an energy minimization formulation is presented. Each primitive shape in an object adds geometric constraints on the object’s pose. An algorithm is proposed to find minimal sets of primitive shapes which are sufficient to determine the complete 3D position and orientation of a rigid object. The pose is estimated using a linear least squares solver over the combination of constraints enforced by the primitive shapes. Experiments illustrating the primitive shape decomposition of object models, detection of these minimal sets, feature vector calculation for sets of shapes and object pose estimation have been presented on simulated and real data.

Industrial robotics is currently witnessing a phase where a lot of effort is directed towards applications of standard industrial robots in smaller industries with short production lines, where the environment is rather unstructured and rapidly changing. Standard industrial robot systems face limitations in their ability to adapt to these environments, and with the complexity of some tasks which seem relatively easier to humans. We present a framework for intuitive symbiotic human robot collaboration in industrial scenarios, where the differing capabilities of human and robot can be combined in a way which enhances the overall effectiveness of the process.

In this paper, a combination of perception modules and reasoning engines is used for scene understanding in typical Human-Robot Interaction(HRI) scenarios. The major contribution of this work lies in a 3D object detection, recognition and pose estimation module, which can be trained using CAD models and works for noisy data, partial views and in cluttered scenes. This perception module is combined with first-order logic reasoning to provide a semantic description of scenes, which is used for process planning. This abstraction of the scene is an important concept in the design of intelligent robotic systems which can adapt to unstructured and rapidly changing environments since it provides a separation of the process planning problem from its execution and scenario-specific parameters. This work is aimed at HRI applications in industrial settings and has been evaluated in several experiments and demonstration scenarios for autonomous process plan execution, humanrobot interaction and co-operation.

In this paper, an intuitive interface for collaborative tasks involving a human and a standard industrial robot is presented. The target for this interface is a worker who is experienced in manufacturing processes but has no experience in conventional industrial robot programming. Physical Human-Robot Interaction (pHRI) and interactive GUI control using hand gestures offered by this interface allows this novice user to instruct industrial robots with ease. This interface combines state of the art perception capabilities with first order logic reasoning to generate semantic description of the process plan. This semantic representation creates the possibility of including human and robot tasks in the same plan and also reduces the complexity of problem analysis by allowing process planning at semantic level, thereby isolating the problem description and analysis from the execution and scenario-specific parameters.

In this paper, a scene perception and recognition module aimed at use in typical industrial scenarios is presented. The major contribution of this work lies in a 3D object detection, recognition and pose estimation module, which can be trained using CAD models and works for noisy data, partial views and in cluttered scenes. This algorithm was qualitatively and quantitatively compared with other state-of-art algorithms. Scene perception and recognition is an important aspect in the design of intelligent robotic systems which can adapt to unstructured and rapidly changing environments. This work has been used and evaluated in several experiments and demonstration scenarios for autonomous process plan execution, human-robot interaction and co-operation.

This paper introduces a new comprehensive solution for the open problem of uncalibrated 3D image-based stereo visual servoing for robot manipulators. One of the main contributions of this article is a novel 3D stereo camera model to map positions in the task space to positions in a new 3D Visual Cartesian Space (a visual feature space where 3D positions are measured in pixels). This model is used to compute a full-rank Image Jacobian Matrix (J img ), which solves several common problems presented on the classical image Jacobians, e.g., image space singularities and local minima. This Jacobian is a fundamental key for the image-based control design, where uncalibrated stereo camera systems can be used to drive a robot manipulator. Furthermore, an adaptive second order sliding mode control is designed to track 3D visual motions using the 3D trajectory errors defined in the Visual Cartesian Space, where a Torque to Position Model is designed to allow the implementation of joint torque control techniques on joint position-controlled robots. This approach has been experimentally implemented on a real industrial robot where exponential convergence of errors in the Visual Cartesian Space and Task space without local minima are demonstrated. This approach offers a proper solution for the common problem of visual occlusion, since the stereo system can be moved manually to obtain a clear view of the task at any time.