This paper investigates the benefits of immersive and intuitive 3D virtual reality (VR) control versus conventional 2D screen-based control for teleoperating a mobile manipulator in hazardous scenarios. We implemented two functionally similar control methods on a ruggedized response robot: (1) a mouse-driven 2D GUI with multiple live video streams and a 2D projection of environment and robot, and (2) a Godot-based walkable VR environment operated with a commercial VR headset, featuring an interactive digital twin and overlaid sensor data. In a within-subjects experiment with nine participants (with robot control experience) performing inspection and manipulation tasks, we measured objective performance (time on task, accuracy) and subjective responses with standardized questionnaires (SSQ, SEQ, ASQ, UMUX, NASA-TLX). Both control methods were well suited for the given tasks, but participants predominantly preferred the VR control. Without inducing excessive simulator sickness, VR control yielded statistically significant 12% faster task completion, 45% lower mental workload, and higher usability ratings. These results demonstrate that intuitive VR interfaces can enhance expert operators’ performance and the usability of response robots with a manipulator in time-critical scenarios.

Robotic systems performing end-user oriented autonomous exploration can be deployed in different scenarios which not only require mapping but also simultaneous inspection of regions of interest for the end-user. In this work, we propose a novel Next-Best-View (NBV) planner which can perform full exploration and user-oriented exploration with inspection of the regions of interest using a mobile manipulator robot. We address the exploration-inspection problem as an instance of Multi-Objective Optimization (MOO) and propose a weighted-sum-based information gain function for computing NBVs for the RGB-D camera mounted on the arm. For both types of exploration tasks, we compare our approach with an existing state-of-the-art exploration method as the baseline and demonstrate our improvements in terms of total volume mapped and lower computational requirements. The real experiments with a mobile manipulator robot demonstrate the practicability and effectiveness of our approach outdoors.

: Grasping of objects is not always feasible for robot manipulators, e.g

Unmanned ground vehicles (UGV) with differing degrees of autonomy are increasingly used for routine tasks, albeit still widely controlled through teleoperation in safety-critical contexts. We pro-pose to complement a traditional approach of showing camera views on a 2D screen with an immersive Virtual Environment (VE), displaying camera data with depth cues around a model of the robot. Combining both approaches with a wearable sensor jacket, the operator can choose the best one for the current task. Preliminary experiments showed that even untrained operators can successfully solve pick and place tasks with high accuracy with our system.

Rescue vehicles accidentally driving into non-traversable terrains increase the risk of delays during disaster response. This paper investigates the problem of coordinating heterogeneous robots to find traversable regions for rescue vehicles under limited communication constraint. The communication range is strictly limited such that no inter-robot communication is possible outside the range. The proposed coordination strategy is called Simultaneous Exploration And Information Delivery (SEAID) strategy and is implemented as a centralised planner. The results demonstrate that SEAID planner completely explores and discovers all the traversable and non-traversable regions to guide rescue vehicles. It also provides regular updates to the base station on the current status of exploration and traversable terrains discovered which is particularly useful during rescue operations.

Gesture-based control potentially eliminates the need for wearisome physical controls and facilitates easy interaction between a human and a robot. At the same time, it is intuitive and enables a natural means of control. In this paper, we present and evaluate a framework for gesture recognition using four wearable Inertial Measurement Units (IMUs) to indirectly control a mobile robot. Six gestures involving different hand and arm motions are defined. A novel algorithm based on an Online Lazy Neighborhood Graph (OLNG) search is used to recognise and classify the gestures online. A software framework is developed to control a robotic platform through integrating our gesture recognition algorithm with a Robot Operating System (ROS), which is in turn used to trigger predefined robot behaviours. Experiments show that the framework is able to correctly detect and classify six different gestures in real time with average success rates of 81.61 % and 81.67 %, while keeping the false-positive rate low by designing and using only 126 training samples.

Many path planning systems for mobile robots today rely on a combination of way point navigation and reactive collision avoidance for motion control Generally, such systems are notable to reliably produce smooth and predictable navigation behavior as it is required, e.g., when robots and humans operate in the same environment In contrast, spline-based motion planning techniques can generate smooth goal-directed trajectories But for most robot drives a subsequent control step is required that actually steers the robot along the desired trajectory In this article we propose a control approach that is based on the Dynamic Window Approach(DWA) Classical DWA controls the robot's velocities in order to steer it to a given goal location, avoiding collisions along the way Here, the approach is used to follow a given spline path as close as possible, while collision avoidance is already handled in the spline-based motion planning Thus expensive collision checks can be omitted from the velocity search of the DWA and the gained processing time allows computing velocity controls several time-steps ahead, resulting in improved path following, We demonstrate the benefit of this approach in simulation experiments and show its applicability for a real robot.

Teleoperation is still the de-facto mode of operation for robotic manipulators in hazardous and unknown environments. The objective is to move the manipulator under the influence of a plenitude of constraints, mainly following the human operator's commands, but also the avoidance of adverse effects such as joint limits or the exertion of external forces. A classic approach to incorporate such non-instantaneous behavior into the instantaneous motion of the kinematic chain is the Closed-Loop Inverse Kinematics (CLIK) control scheme.In this paper, we present PV-CLIK, a novel CLIK realization that for the first time practically applies the Popov-Vereshchagin (PV) hybrid dynamics solver to map the instantaneous constraints to motion commands. By relying on the PV solver, PV-CLIK offers several benefits over traditional CLIK implementations such as linear runtime complexity, handling constraints on the dynamics level or fostering composable software architectures. In the experimental evaluation, we show that our implementation of PV-CLIK outperforms existing kinematics solvers in Cartesian trajectory-following tasks at high velocities.

This article introduces a novel approach to the online complete- coverage path planning (CCPP) problem that is specically tailored to the needs of skid-steer tracked robots. In contrast to most of the current state-of-the-art algorithms for this task, the proposed algorithm reduces the number of turning maneuvers, which are responsible for a large part of the robot's energy consumption. Nevertheless, the approach still keeps the total distance traveled at a competitive level. The algorithm operates on a grid-based environment representation and uses a 3x3 prioritization matrix for local navigation decisions. This matrix prioritizes cardinal di- rections leading to a preference for straight motions. In case no progress can be achieved based on a local decision, global path planning is used to choose a path to the closest known unvisited cell, thereby guaranteeing completeness of the approach. In an extensive evaluation using simulation experiments, we show that the new algorithm indeed generates competi- tively short paths with largely reduced turning costs, compared to other state-of-the-art CCPP algorithms. We also illustrate its performance on a real robot.

This paper proposes a novel framework for the segmentation and classification of 3D point cloud which jointly uses spatial, temporal and semantic information. It improves the classification performance by reducing undersegmentation errors. The presented framework, which can determine the number and label of objects in each spatially extracted blob, is decomposed into three steps to acquire spatial, temporal and semantic cues. For the spatial features, blobs are extracted spatially with a neighborhood system on an occupancy grid representation. A smoothed motion field is estimated for the acquisition of temporal cue, where the grid cells are tracked using individual Kalman filters and estimated velocities are transformed to one dimensional movement directions. A support vector machine (SVM) classifier is trained to discriminate the classes of interest for the semantic information of the blobs. A confidence metric is defined to probabilistically compare the volume of each classified blob with the volume of an average object for that class. If this metric is below a predefined threshold, a sequential variant of distance dependent Chinese restaurant process (s-ddCRP) performs the final partition in this blob by using spatial and temporal information. If the s-ddCRP approach splits the blob, the partitioned sub-blobs are afterwards reassigned to new objects by the classifier. Otherwise, the queried blob remains the same. This procedure iteratively continues while searching each blob in the scene at each time frame. Experiments on data obtained with a Velodyne HDL64 scanner in real traffic scenarios illustrate that the proposed framework improves the classification performance of an SVM classifier by reducing under-segmentation errors.

Visual scene decomposition into semantic entities is one of the major challenges when creating a reliable object grasping system. Recently, we introduced a bottom-up hierarchical clustering approach which is able to segment objects and parts in a scene. In this paper, we introduce a transform from such a segmentation into a corresponding, hierarchical saliency function. In comprehensive experiments we demonstrate its ability to detect salient objects in a scene. Furthermore, this hierarchical saliency defines a most salient corresponding region (scale) for every point in an image. Based on this, an easy-to-use pick and place manipulation system was developed and tested exemplarily.

Visual grouping is a key mechanism in human scene perception. There, it belongs to the subconscious, early processing and is key prerequisite for other high level tasks such as recognition. In this paper, we introduce an efficient, realtime capable algorithm which likewise agglomerates a valuable hierarchical clustering of a scene, while using purely local appearance statistics. To speed up the processing, first we subdivide the image into meaningful, atomic segments using a fast Watershed transform. Starting from there, our rapid, agglomerative clustering algorithm prunes and maintains the connectivity graph between clusters to contain only such pairs, which directly touch in the image domain and are reciprocal nearest neighbors (RNN) wrt. a distance metric. The core of this approach is our novel cluster distance: it combines boundary and surface statistics both in terms of appearance as well as spatial linkage. This yields state-of-the art performance, as we demonstrate in conclusive experiments conducted on BSDS500 and Pascal-Context datasets.

This paper proposes a novel hybrid segmentation method for 3D Light Detection and Ranging (Lidar) data. The presented approach gains robustness against the under-segmentation issue, i.e., assigning several objects to one segment, by jointly using spatial and temporal information to discriminate nearby objects in the data. When an autonomous vehicle has a complex dynamic environment, such as pedestrians walking close to their nearby objects, determining if a segment consists of one or multiple objects can be difficult with spatial features alone. The temporal cues allow us to resolve such ambiguities. In order to get temporal information, a motion field of the environment is estimated for subsequent 3D Lidar scans based on an occupancy grid representation. Then we propose a hybrid approach using the mean-shift method and the distance dependent Chinese Restaurant Process (ddCRP). After the segmentation blobs are spatially extracted from the scene, the mean-shift seeks the number of possible objects in the state space of each blob. If the mean-shift algorithm determines an under-segmented blob, the ddCRP performs the final partition in this blob. Otherwise, the queried blob remains the same and it is assigned as a segment. The computational time of the hybrid method is below the scanning period of the Lidar sensor. This enables the system to run in real time.

This paper proposes a novel object segmentation method for 3D Light Detection and Ranging (LIDAR) data that is particularly useful for the traffic scene analysis of self-driving vehicles. The novel method gains robustness against under-segmentation, i.e. the problem of assigning several objects to one segment, by jointly using geometrical features and motion field information to discriminate even spatially close objects in the data. The approach maps point cloud data to an occupancy grid representation and estimates the motion field using Kalman filter based tracking of grid cells. A non-parametric Bayesian clustering approach based on a sequential distance dependent Chinese Restaurant Process (s-ddCRP) utilizes this information in order to sample possible data segmentation hypotheses and decide on the most probable one. The computational efficiency of the approach is improved by exploiting the sequential nature of the problem and initializing the required Gibbs sampler using data from the previous time step and fixing found clusters of objects in super grid cells which can be tracked jointly. Experiments carried out on data obtained with a Velodyne HDL64 scanner in a real traffic scenario illustrate the performance of the approach.

This paper proposes a novel method for integrated tracking and segmentation of 3D Light Detection and Ranging (LIDAR) data. The conventional processing pipeline of object tracking methods performs the segmentation and tracking modules consecutively. They apply a connected component algorithm on a grid for object segmentation. This results in an under-segmentation and in turn wrong tracking estimates when there are spatially close objects. We present a new approach in which segmentation and tracking modules profit from each other to resolve ambiguities in complex dynamic scenes. A non-parametric Bayesian method, the sequential distance dependent Chinese Restaurant Process (s-ddCRP), enables us to combine segmentation and tracking components. After a pre-processing step which maps measurements to a grid representation, the proposed method tracks each grid cell and segments the environment in an integrated way. A smoothing algorithm is applied to the estimated grid cell velocities for better motion consistency of neighboring dynamic grid cells. Experiments on data obtained with a Velodyne HDL64 scanner in real traffic scenarios illustrate that the proposed approach has a encouraging detection performance and conclusive motion consistency between consecutive time frames.

We present a fast method of detecting loop closure opportunities through the use of similarity measures on histograms extracted from 3-D LIDAR data. We avoid computationally expensive features and compute histograms over simple global statistics of the LIDAR scans. The resulting histograms encode sufficient information to detect spatially close scans with high precision and recall and can be computed at rates faster than data acquisition on modest consumer-grade hardware. Our approach is able to match previously established results in LIDAR loop closure detection with less computational overhead.

This paper proposes a novel method to obtain robust and accurate object segmentations from 3D Light Detection and Ranging (LIDAR) data points. The method exploits motion information simultaneously estimated by a tracking algorithm in order to resolve ambiguities in complex dynamic scenes. Typical approaches for tracking multiple objects in LIDAR data follow three steps; point cloud segmentation, object tracking, and track classification. A large number of errors is due to failures in the segmentation component, mainly because segmentation and tracking are performed consecutively and the segmentation step solely relies on geometrical features. This article presents a 3D LIDAR based object segmentation method that exploits the motion information provided by a tracking algorithm and spatial features in order to discriminate spatially close objects. After a pre-processing step that maps LIDAR measurements to an occupancy grid representation, the motions of grid cells are estimated using independent Kalman filters. A distance dependent Chinese Restaurant Process based Markov chain Monte Carlo approach is applied to generate different segmentation hypotheses and decide on the most probable segments by using motion and spatial features together.