Despite the rapid technological progress, autonomous vehicles still face a wide range of complex driving situations that require human intervention. Teleoperation technology offers a versatile and effective way to address these challenges. The following work puts existing ideas into a modern context and introduces a novel technical implementation of the trajectory guidance teleoperation concept. The presented system was developed within a high-fidelity simulation environment and experimentally validated, demonstrating a realistic ride-hailing mission with prototype autonomous vehicles and onboard passengers. The results indicate that the proposed concept can be a viable alternative to the existing remote driving options, offering a promising way to enhance teleoperation technology and improve overall operation safety.

Teleoperated Driving (ToD) is a widely acknowledged concept applied to handle edge-case situations in automated vehicles. In ToD, a human operator judges and resolves these situations based on video streams. Due to varying network coverage, the compression level of these video streams and therefore the resulting image quality (IQ) are adjusted dynamically. In the presented work, the effect of IQ on task performance is investigated. We hypothesize that IQ impacts the operator’s reaction time to dynamic obstacles, and therefore influences safety. We conducted a user study to test this hypothesis. Subjective and objective data were collected. The results reveal that IQ has a significant influence on the operator’s task performance.

Automated vehicles and their operational range are getting better over time. Due to the complexity of urban traffic, it might not be feasible for an automated vehicle to handle all occasions. To maintain vehicle operation, a remote operator can connect to the vehicle via the cellular network to resolve those situations. This method, called Teleoperated Driving (ToD), introduces new safety challenges to the vehicle. Due to fluctuations in network performance and missing physical connection to the vehicle, there is no guarantee that the operator is continuously able to safely control the vehicle. Therefore, a concept is proposed that enables the vehicle to return to a safe state if certain system requirements are not met. This concept relies on a safe trajectory, which is implicitly planned by the operator as long as all system requirements are met.

In parallel with the advancement of Automated Driving (AD) functions, teleoperation has grown in popularity over recent years. By enabling remote operation of automated vehicles, teleoperation can be established as a reliable fallback solution for operational design domain limits and edge cases of AD functions. Over the years, a variety of different teleoperation concepts as to how a human operator can remotely support or substitute an AD function have been proposed in the literature. This paper presents the results of a literature survey on teleoperation concepts for road vehicles. Furthermore, due to the increasing interest within the industry, insights on patents and overall company activities in the field of teleoperation are presented.

Teleoperation is becoming an essential feature in automated vehicle concepts, as it will help the industry overcome challenges facing automated vehicles today. Teleoperation follows the idea to get humans back into the loop for certain rare situations the automated vehicle cannot resolve. Teleoperation therefore has the potential to expand the operational design domain and increase the availability of automated vehicles. This is especially relevant for concepts with no backup driver inside the vehicle. While teleoperation resolves certain issues an automated vehicle will face, it introduces new challenges in terms of safety requirements. While safety and regulatory approval is a major research topic in the area of automated vehicles, it is rarely discussed in the context of teleoperated road vehicles. The focus of this paper is to systematically analyze the potential hazards of teleoperation systems. An appropriate hazard analysis method (STPA) is chosen from literature and applied to the system at hand. The hazard analysis is an essential part in developing a safety concept (e.g., according to ISO26262) and thus far has not been discussed for teleoperated road vehicles.

Teleoperation allows a human operator to remotely interact with and control a mobile robot in a dangerous or inaccessible area. Besides well-known applications such as space exploration or search and rescue operations, the application of teleoperation in the area of automated driving, i.e., teleoperated driving (ToD), is becoming more popular. Instead of an in-vehicle human fallback driver, a remote operator can connect to the vehicle using cellular networks and resolve situations that are beyond the automated vehicle (AV)'s operational design domain. Teleoperation of AVs, and unmanned ground vehicles in general, introduces different problems, which are the focus of ongoing research. This paper presents an open source ToD software stack, which was developed for the purpose of carrying out this research. As shown in three demonstrations, the software stack can be deployed with minor overheads to control various vehicle systems remotely.

Vehicles with autonomous driving capabilities are present on public streets. However, edge cases remain that still require a human in-vehicle driver. Assuming the vehicle manages to come to a safe state in an automated fashion, teleoperated driving technology enables a human to resolve the situation remotely by a control interface connected via a mobile network. While this is a promising solution, it also introduces technical challenges, one of them being the necessity to transmit video data of multiple cameras from the vehicle to the human operator. In this paper, an adaptive video streaming framework specifically designed for teleoperated vehicles is proposed and demonstrated. The framework enables automatic reconfiguration of the video streams of the multi-camera system at runtime. Predictions of variable transmission service quality are taken into account. With the objective to improve visual quality, the framework uses so-called rate-quality models to dynamically allocate bitrates and select resolution scaling factors. Results from deploying the proposed framework on an actual teleoperated driving system are presented.

Automated driving has started to be used commercially for individual mobility and public transport in recent years. As soon as automated driving is commercially exploited, automated vehicle fleets require assistance which can be provided by an operational control center as known from air traffic, public transport or process technology. The need for assistance is shown by two aspects. The aspects are human interaction and efficiency of journeys. Based on that, a control center is proposed to address those aspects. Further, a concept for a control center is derived. For that, the methods of interviewing experts, observation and literature research are used. Specific tasks for the control center are defined. These tasks were examined and categorized into three service categories. The three service categories are emergency service, fleet service and teleoperation service. Due to the categories, future-built control centers will be scalable and adaptable to its demand. Whereas two of three categories are well covered by industry or research, the teleoperation service as the essential problem solving technique needs further development. Moreover, further research will be required to quantify the control center demand.

Due to the challenges of autonomous driving, backup options like teleoperation become a relevant solution for critical scenarios an automated vehicle might face. To enable teleoperated systems, two main problems have to be solved: Safely controlling the vehicle under latency, and presenting the sensor data from the vehicle to the operator in such a way, that the operator can easily understand the vehicles environment and the vehicles current state. While most of the teleoperation systems face similar challenges, the teleoperation of automated vehicles is unique in its scale, safety requirements and system constraints. Two major constraints are the round-trip-latency and the maximum upload-bandwidth. While the latency mainly influences the controllability and safety of the vehicle, the upload-bandwidth affects the amount of transmittable sensor data and therefore operators situation awareness, as well as the running costs of the whole system. The focus of this paper is measuring and reducing the end-to-end latency for a teleoperation setup. Therefore the latency is separated into actuator and sensor latency. For each part the different components and settings are analyzed in order to find a realistic minimal end-to-end latency for the teleoperation of automated vehicles. Therefore new measurement methods are developed and existing methods adapted.

Future Cooperative Intelligent Transport Systems (C-ITS) require an integrated functional framework that provides cloud-based services to automated vehicles and other traffic participants. The goal is to process, store and share relevant information in order to continually assure and improve the efficiency, safety and comfort of the CITS. This paper introduces a first conceptual hypothesis for such a framework that is developed in the project UNICARagil, funded by the German Federal Ministry for Education and Research (BMBF). Three main components of this framework, the Collective Environment Model, the Collective Memory and the Collective Behavior, are presented. Open challenges associated with current and future technology are discussed.