Vehicle teleoperation is gaining relevance not only as a fallback solution for autonomous vehicles but also as an independent mode of vehicle control. While a common design goal of teleoperation interfaces is to increase immersive qualities and the telepresence experience, it remains unclear whether this merely improves user experience or if telepresence is also related to operational outcomes, such as increased situation and spatial awareness and remote driving behaviours. This test-track study compared two immersive features - vestibular feedback via a motion platform and surround-view via video-stitching - against a baseline remote control version and in-car driving to manipulate the level of presence and examine its relation to user experience and operational outcomes. Both immersive features increased the users' presence experience. Presence experience was related to spatial awareness, marginally related to situation awareness, and strongly related to risky driving behaviours, specifically the number of harsh brakings. Anecdotal observations also supported a "video game effect," where individual differences were observed in how participants treated the remote setup: while some acted with the utmost caution, aware of real-world consequences, others treated the setup like a video game and appeared indifferent even when they crashed. Overall, the study implies that presence experience is a meaningful design target not only to enhance user experience but also to mitigate risky control behaviours. We advocate for the inclusion of risk perception in the remote environment as a dimension of presence experience.

Vehicle teleoperation is gaining relevance not only as a fallback solution for autonomous vehicles but also as an independent mode of vehicle control. While a common design goal of teleoperation interfaces is to increase immersive qualities and the telepresence experience, it remains unclear whether this merely improves user experience or if telepresence is also related to operational outcomes, such as increased situation and spatial awareness and remote driving behaviours. This test-track study compared two immersive features - vestibular feedback via a motion platform and surround-view via video-stitching - against a baseline remote control version and in-car driving to manipulate the level of presence and examine its relation to user experience and operational outcomes. Both immersive features increased the users' presence experience. Presence experience was related to spatial awareness, marginally related to situation awareness, and strongly related to risky driving behaviours, specifically the number of harsh brakings. Anecdotal observations also supported a "video game effect," where individual differences were observed in how participants treated the remote setup: while some acted with the utmost caution, aware of real-world consequences, others treated the setup like a video game and appeared indifferent even when they crashed. Overall, the study implies that presence experience is a meaningful design target not only to enhance user experience but also to mitigate risky control behaviours. We advocate for the inclusion of risk perception in the remote environment as a dimension of presence experience.

Video streaming based teleoperation often faces a trade-off between bandwidth consumption and the need for high-fidelity telepresence. Higher image resolution or a wider field of view (FOV) substantially increases bandwidth requirements. In this paper, we propose a novel telepresence model for teleoperated vehicles operating in bandwidth-constrained environments. Our approach employs a LiDAR-fused 3D Gaussian Splatting (3DGS) as a compact scene representation to efficiently generate remote views. Initially, a static point cloud map is constructed using LiDAR-based semantic mapping, which serves as the initial Gaussians for optimizing the 3DGS model. During teleoperation, the prebuilt 3DGS is then rendered on the teleoperation platform, while only safety-critical information, such as vehicle pose and dynamic objects, is transmitted from the vehicle to the teleoperator in real-time. The proposed telepresence model significantly reduces data transmission requirements while maintaining photorealistic telepresence, enabling reliable and effective teleoperation even under stringent bandwidth constraints. This capability ensures safe and efficient vehicle teleoperation under challenging environments without relying on traditional high-bandwidth communication, thereby broadening the applicability of teleoperation technology to more demanding and diverse operational scenarios. Real-world experimental results show that the developed system can provide immersive teleoperation experiences at Kbps-level bandwidth consumption.

Worker shortages and safety incidents in warehouses call for innovative solutions. This study investigated the usability of a teleoperation prototype for forklifts and its impact on warehouse assistants' job well-being and satisfaction. Six warehouse assistants tested a prototype teleoperation system repeatedly, with interviews conducted before and after the trial. Findings revealed mixed attitudes: While participants appreciated improved safety from physical separation, they expressed concerns about reduced work efficiency. Audio feedback was deemed essential for situational awareness, while opinions on motion feedback were mixed. Challenges included difficulties estimating fork positions from camera views, highlighting the need for additional cameras. The game-like character of the system raised concerns about potential impacts on risk perception. Despite limited impacts on career choice, remote control technology can offer safety and comfort benefits, supporting worker health and satisfaction.

Traditional teleoperation technologies based on video streaming are facing several challenges in practical applications, including limited bandwidth, constrained spatial awareness, and sensitivity to illumination. Existing studies have not adequately addressed these issues. This paper presents a novel non-video based teleoperation framework for autonomous vehicles operating in bandwidth-limited environments. To reduce the amount of data being transmitted, a persistent-transient environment model is proposed for telepresence. Initially, a digital twin of the environment is preconstructed, containing only persistent environmental information. Subsequently, transient information captured by onboard sensors, such as vehicle state and dynamic objects, necessitate real-time transmission. Based on this model, a 3D virtual scene is rendered in front of the teleoperator, offering any desired virtual viewpoint to enhance spatial awareness. This telepresence model only requires real-time transmission of minimal data, i.e., vehicle state and detected objects, and remains unaffected by illumination conditions, enabling teleoperation even in applications with Kbps-level bandwidth constraints. Experimental results showcase the substantial potential of the proposed framework in bandwidth-limited settings.

Simultaneous Localization and Mapping (SLAM) is moving towards a robust perception age. However, LiDAR- and visual- SLAM may easily fail in adverse conditions (rain, snow, smoke and fog, etc.). In comparison, SLAM based on 4D Radar, thermal camera and IMU can work robustly. But only a few literature can be found. A major reason is the lack of related datasets, which seriously hinders the research. Even though some datasets are proposed based on 4D radar in past four years, they are mainly designed for object detection, rather than SLAM. Furthermore, they normally do not include thermal camera. Therefore, in this paper, NTU4DRadLM is presented to meet this requirement. The main characteristics are: 1) It is the only dataset that simultaneously includes all 6 sensors: 4D radar, thermal camera, IMU, 3D LiDAR, visual camera and RTK GPS. 2) Specifically designed for SLAM tasks, which provides fine-tuned ground truth odometry and intentionally formulated loop closures. 3) Considered both low-speed robot platform and fast-speed unmanned vehicle platform. 4) Covered structured, unstructured and semi-structured environments. 5) Considered both middle- and large- scale outdoor environments, i.e., the 6 trajectories range from 246m to 6.95km. 6) Comprehensively evaluated three types of SLAM algorithms. Totally, the dataset is around 17.6km, 85mins, 50GB and it will be accessible from this link: https://github.com/junzhang2016/NTU4DRadLM

Teleoperation or remote operation is an important part of research in autonomous vehicles and robots. It serves as a useful backup option in case the autonomous system encounters any issue and human intervention is needed. In the case of heavy-duty vehicles for off-road applications, teleoperation is also a good alternative even without autonomy, as it can help improve the working conditions of the operators. In either case, teleoperation can be quite challenging due a number of factors such as network connectivity, lack of natural visual perspective and the inability of the remote operator to judge the vehicle's accelerations and movements. These challenges cause significant safety issues. We have developed a high-fidelity teleoperation system that is designed to overcome these challenges and make remote operation easy and natural. The prototypes have been developed and significantly tested during proof of concept trials with various types of vehicles to demonstrate the effectiveness of our system.

In this work, we present the design, fabrication, and experimental validation of a lightweight, low inertia dual-arm manipulator with a center of gravity (COG) balancing mechanism, specifically designed for aerial manipulation missions. The developed system, consisting of the dual-arm base and two arms with 6 degrees of freedom (DOFs) each, weighs 2.5 kg in total with a maximum payload of 1.0 kg per arm. The dual-arm system is designed such that it can be attached to different multirotors without making major design modifications. In addition, the design of the arms is conceived to decrease the inertia of the arms by utilizing a timing belt-based transmission mechanism. Moreover, the proposed dual-arm design employs prismatic joints to introduce the following distinctive features: 1) ability of each arm to dynamically adjust its COG for better flight performance; 2) fully independent control of each arm for performing different tasks simultaneously; 3) extended workspace and reach of the arms for enhancing operational capability and improving safety during aerial manipulation missions. Since the proposed design has low inertia and a compensation mechanism to deal with the disturbances caused by the COG change, the developed dual-arm system can be mounted on multirotors equipped with standard autopilots, opening the door for the widespread use of dual-arm manipulators with the commercially available UAV platforms. To ensure the robustness of the mechanical structure, an analysis using finite element methods (FEM) is conducted. Extensive experimental flight tests are performed to evaluate the proposed dual-arm design with a hexarotor equipped with a common off-the-shelf autopilot. The experimental results show that the influence of the arms motion over the hexarotor stability is minor in contactless flight due to the low weight and inertia of the proposed dual-arm design. Furthermore, better hovering performance is achieved by exploiting the ability of the proposed design to compensate the dual-arm COG displacement.

In this work, a learning model-free control method is proposed for accurate trajectory tracking and safe landing of unmanned aerial vehicles (UAVs). A realistic scenario is considered where the UAV commutes between stations at high-speeds, experiences a single motor failure while surveying an area, and thus requires to land safely at a designated secure location. The proposed challenge is viewed solely as a control problem. A hybrid control architecture – an artificial neural network (ANN)-assisted proportional-derivative controller – is able to learn the system dynamics online and compensate for the error generated during different phases of the considered scenario: fast and agile flight, motor failure, and safe landing. Firstly, it deals with unmodelled dynamics and operational uncertainties and demonstrates superior performance compared to a conventional proportional-integral-derivative controller during fast and agile flight. Secondly, it behaves as a fault-tolerant controller for a single motor failure case in a coaxial hexacopter thanks to its proposed sliding mode control theory-based learning architecture. Lastly, it yields reliable performance for a safe landing at a secure location in case of an emergency condition. The tuning of weights is not required as the structure of the ANN controller starts to learn online, each time it is initialised, even when the scenario changes – thus, making it completely model-free. Moreover, the simplicity of the neural network-based controller allows for the implementation on a low-cost low-power onboard computer. Overall, the real-time experiments show that the proposed controller outperforms the conventional controller.

In this work, we address fast and agile manoeuvre control problem of unmanned aerial vehicles (UAVs) using an artificial neural network (ANN)-assisted conventional controller. Whereas the need for having almost perfect control accuracy for UAVs pushes the operation to boundaries of the performance envelope, safety and reliability concerns enforce researchers to be more conservative in tuning their controllers. As an alternative solution to the aforementioned trade-off, a reliable yet accurate controller is designed for the trajectory tracking of UAVs by learning system dynamics online over the trajectory. What is more, the proposed online learning mechanism helps us to deal with unmodelled dynamics and operational uncertainties. Experimental results validate the proposed approach and show the superiority of our method compared to the conventional controller for fast and agile manoeuvres, at speeds as high as 20m/s. An onboard implementation of the sliding mode control theory-based adaptation rules for the training of the proposed ANN is computationally efficient which allows us to learn system dynamics and operational variations instantly using a low-cost and low-power computer.

In this paper, we present a novel onboard robust visual algorithm for long-term arbitrary 2D and 3D object tracking using a reliable global-local object model for unmanned aerial vehicle (UAV) applications, e.g., autonomous tracking and chasing a moving target. The first main approach in this novel algorithm is the use of a global matching and local tracking approach. In other words, the algorithm initially finds feature correspondences in a way that an improved binary descriptor is developed for global feature matching and an iterative Lucas–Kanade optical flow algorithm is employed for local feature tracking. The second main module is the use of an efficient local geometric filter (LGF), which handles outlier feature correspondences based on a new forward-backward pairwise dissimilarity measure, thereby maintaining pairwise geometric consistency. In the proposed LGF module, a hierarchical agglomerative clustering, i.e., bottom-up aggregation, is applied using an effective single-link method. The third proposed module is a heuristic local outlier factor (to the best of our knowledge, it is utilized for the first time to deal with outlier features in a visual tracking application), which further maximizes the representation of the target object in which we formulate outlier feature detection as a binary classification problem with the output features of the LGF module. Extensive UAV flight experiments show that the proposed visual tracker achieves real-time frame rates of more than thirty-five frames per second on an i7 processor with 640 × 512 image resolution and outperforms the most popular state-of-the-art trackers favorably in terms of robustness, efficiency and accuracy.

Ground plane detection is essential for successful navigation of vision based mobile robots. We introduce a very simple but robust ground plane detection method based on depth information obtained using an RGB-Depth sensor. We present two different variations of the method: the simplest one is robust in setups where the sensor pitch angle is fixed and has no roll, whereas the second one can handle changes in pitch and roll angles. Our comparisons show that our approach performs better than the vertical disparity approach. It produces accurate ground plane-obstacle segmentation for difficult scenes, which include many obstacles, different floor surfaces, stairs, and narrow corridors.

Ground plane detection is essential for successful navigation of vision based mobile robots. We introduce a novel and robust ground plane detection algorithm using depth information acquired by a Kinect sensor. Unlike similar methods from the literature, we do not assume that the ground plane covers the largest area in the scene. Furthermore our algorithm handles two different conditions: fixed and changing view angle of the sensor. We show that the algorithm is robust if the view angle is fixed whereas an additional procedure handles different view angles satisfactorily.

This paper presents a novel vision based obstacle detection algorithm that is adapted from a powerful background subtraction algorithm: ViBe (VIsual Background Extractor). We describe an adaptive obstacle detection method using monocular color vision and an ultrasonic distance sensor. Our approach assumes an obstacle free region in front of the robot in the initial frame. However, the method dynamically adapts to its environment in the succeeding frames. The adaptation is performed using a model update rule based on using ultrasonic distance sensor reading. Our detailed experiments validate the proposed concept and ultrasonic sensor based model update.