Taking into account information across the temporal domain helps to improve environment perception in autonomous driving. However, it has not been studied so far whether temporally fused neural networks are vulnerable to deliberately generated perturbations, i.e. adversarial attacks, or whether temporal history is an inherent defense against them. In this work, we study whether temporal feature networks for object detection are vulnerable to universal adversarial attacks. We evaluate attacks of two types: imperceptible noise for the whole image and locally-bound adversarial patch. In both cases, perturbations are generated in a white-box manner using PGD. Our experiments confirm, that attacking even a portion of a temporal input suffices to fool the network. We visually assess generated perturbations to gain insights into the functioning of attacks. To enhance the robustness, we apply adversarial training using 5-PGD. Our experiments on KITTI and nuScenes datasets demonstrate, that a model robustified via K-PGD is able to withstand the studied attacks while keeping the mAP-based performance comparable to that of an unattacked model.

Detecting 3D traffic participants in the surrounding of an autonomous vehicle is a challenging task. But as long as the vehicle is equipped with expensive light detection and ranging (LiDAR) sensors and the weather conditions are sufficient, the task seems to be easily solvable. As soon as one of these preconditions is no longer satisfied, camera sensors become an important factor. Obtaining 3D spatial information from a 2D image is a challenging task and most 3D object detectors do not deliver adequate results as long as they solely depend on camera images. For this reason, more input information needs to be provided to such systems. But as long as there are no other sensors available, it is not trivial to gain additional information. As depth reconstruction from monocular images started to provide solid results, camera based 3D detection algorithms began to use these generated depth maps as additional input information. However, this comes at the cost of an additional algorithm that requires valuable on-board resources. To circumvent this issue, we propose an architecture that learns the concept of depth during training based on additional ground truth. In operational mode, the 3D object detector no longer relies on depth maps as input. On the KITTI 3D object detection dataset the proposed architecture achieves comparable results to architectures using additional depth inputs and outperforms image only methods.

AbstractThe mixture-of-experts (MoE) architecture is an approach to aggregate several expert components via an additional gating module, which learns to predict the most suitable distribution of the expert’s outputs for each input. An MoE thus not only relies on redundancy for increased robustness—we also demonstrate how this architecture can provide additional interpretability, while retaining performance similar to a standalone network. As an example, we train expert networks to perform semantic segmentation of the traffic scenes and combine them into an MoE with an additional gating network. Our experiments with two different expert model architectures ( and ) reveal that the MoE is able to reach, and for certain data subsets even surpass, the baseline performance and also outperforms a simple aggregation via ensembling. A further advantage of an MoE is the increased interpretability—a comparison of pixel-wise predictions of the whole MoE model and the participating experts’ help to identify regions of high uncertainty in an input.

For reliable environment perception, the use of temporal information is essential in some situations. Especially for object detection, sometimes a situation can only be understood in the right perspective through temporal information. Since image-based object detectors are currently based almost exclusively on CNN architectures, an extension of their feature extraction with temporal features seems promising. Within this work we investigate different architectural components for a CNN-based temporal information extraction. We present a Temporal Feature Network which is based on the insights gained from our architectural investigations. This network is trained from scratch without any ImageNet information based pre-training as these images are not available with temporal information. The object detector based on this network is evaluated against the non-temporal counterpart as baseline and achieves competitive results in an evaluation on the KITTI object detection dataset.

The use of deep neural networks (DNNs) in safety-critical applications like mobile health and autonomous driving is challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability to problems with malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from safety concerns. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. This work provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our paper addresses both machine learning experts and safety engineers: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern ML methods. We moreover hope that our contribution fuels discussions on desiderata for ML systems and strategies on how to propel existing approaches accordingly.

Not only correct scene understanding, but also ability to understand the decision making process of neural networks is essential for safe autonomous driving. Current work mainly focuses on uncertainty measures, often based on Monte Carlo dropout, to gain at least some insight into a models confidence. We investigate a mixture of experts architecture to achieve additional interpretability while retaining comparable result quality.By being able to use both the overall model output as well as retaining the possibility to take into account individual expert outputs, the agreement or disagreement between those individual outputs can be used to gain insights into the decision process. Expert networks are trained by splitting the input data into semantic subsets, e.g. corresponding to different driving scenarios, to become experts in those domains. An additional gating network that is also trained on the same input data is consequently used to weight the output of individual experts. We evaluate this mixture of expert setup on the A2D2 dataset and achieve similar results to a baseline FRRN network trained on all available data, while getting additional information.

For self driving cars one of the current key technologies are deep neural networks. Especially in camera based environment perception they are absolutely irreplaceable. The currently developed network models are usually executed on high end consumer or server GPUs. Also the verification of the real-time properties is mostly based on these GPUs. However, if these models are to be used in near-series applications, the question arises whether they can also be used on significantly reduced hardware. To address this question, we conduct a case study with a camera based traffic light detection system [1]. Promising optimization techniques are adapted and applied to the model to investigate potential performance gains achievable with these techniques in the context of self driving car environment perception. In particular, the trade-off between quality and speed is to be examined in detail.

Current state-of-the-art object detection algorithms still suffer the problem of imbalanced distribution of training data over object classes and background. Recent work introduced a new loss function called focal loss to mitigate this problem, but at the cost of an additional hyperparameter. Manually tuning this hyperparameter for each training task is highly time-consuming. With automated focal loss we introduce a new loss function which substitutes this hyperparameter by a parameter that is automatically adapted during the training progress and controls the amount of focusing on hard training examples. We show on the COCO benchmark that this leads to an up to 30 % faster training convergence. We further introduced a focal regression loss which on the more challenging task of 3D vehicle detection outperforms other loss functions by up to 1:8 AOS and can be used as a value range independent metric for regression.

In autonomous driving, the detection of objects like surrounding vehicles based on monocular RGB images is usually performed by 2D bounding box detectors. The resulting 2D objects can be used for a first coarse 3D position estimate but for a precise location, additional sensor data has to be taken into account. For further use in sensor fusion systems and environment maps it is preferable to detect objects, their orientation and dimensions directly in 3D coordinates. To address this 3D object detection task, we propose a direct 3D bounding box estimator which is realized as CNN decoder module and can be connected to most 2D object detectors like SSD[1], OverFeat[2], YOLO[3] and RetinaNet[4] or directly to CNN feature extractors like VGG [2] and ResNet [5]. The 3D parameters of the objects such as dimension and orientation are directly predicted by the CNN module. To successfully train this complex MultiNet architecture, a combination and modification of current loss functions is proposed. The fastest of the proposed network module combinations is capable of detecting objects in 3D camera coordinates at a frame rate of 28 fps.

The representation of objects as 2D bounding boxes in monocular RGB images limits the faculty of current computer vision systems to 2D object detection. It fails to provide crucial information such as the orientation of other vehicles, which is vital for autonomous driving. At the same time, real-time performance is essential to qualify an approach for deployment in a productive environment. In order to tackle this problem, we present an approach that predicts several key points selected from a virtual 3D bounding box around a vehicle instead of a pure 2D bounding box. These key points can be interpreted as a bounding shape. With this novel representation we can calculate the actual 3D bounding box of the corresponding object. Thanks to the straightforward implementation of bounding shape in any current state-of-the-art 2D object detector both for singleshot frameworks like YOLO or SSD as well as for two-stage detectors like Faster-RCNN with a minimum of computational overhead, it is able to be run in real-time while providing additional useful information for vehicle detection. We exemplify the extension of SSD to Bounding Shape SSD (BS3D) and evaluate our approach using the challenging KITTI as well as the novel VIPER dataset.

We present a system capable of detecting and associating traffic participants in camera images with different points of view to a street scene. Our system is based on a multitask CNN architecture and detection as well as association are performed within the network. The association between different images is estimated without explicit knowledge of scene geometry and camera calibration information. One of the main applications of our system are currently test areas for autonomous vehicles. For this use case ground truth information for testing the environment perception as well as external environment information sent to autonomous vehicles via car-to-infrastructure is of great importance. This is particularly interesting in complex scenarios like big intersections. Our system shows promising results on the association task of big intersections taken from 8 different points of view.

While most approaches to semantic reasoning have focused on improving performance, in this paper we argue that computational times are very important in order to enable real time applications such as autonomous driving. Towards this goal, we present an approach to joint classification, detection and semantic segmentation using a unified architecture where the encoder is shared amongst the three tasks. Our approach is very simple, can be trained end-to-end and performs extremely well in the challenging KITTI dataset. Our approach is also very efficient, allowing us to perform inference at more then 23 frames per second. Training scripts and trained weights to reproduce our results can be found here: https://github.com/MarvinTeichmann/MultiNet

This paper presents the concept, realization and evaluation of a flexible and scalable setup for smart infrastructure at the example of the Test Area Autonomous Driving Baden-Württemberg. In verification and validation of autonomous driving systems, there exists a gap between virtual validation and real road tests: Simulation provides an easy and efficient way to assess a system's performance under a variety of environmental constraints, but is restricted to model assumptions and scenarios, which might ignore important aspects. Whereas expensive real road tests promise an unexpected environment for statistical evaluation of traffic scenarios, but lack of observability. Our setup for smart infrastructure is supposed to close the gap by tackling this issue by observing and providing reference data of traffic scenarios for application in different testing and evaluation settings. We present the approach of implementing a distributed intelligent infrastructure capable of handling traffic light states, road topology and especially information about locally observed traffic participants. The data is provided online via Vehicle-to-X (V2X) communication for live testing and sensor range extension as well as offline via a backend for high-precision analysis and application of machine learning techniques. To obtain information about traffic participants, a camera based object tracking was realised. To cope with the high amount of information to be transmitted via V2X and to use the available bandwidth optimally, the standard for broadcasting vehicle information is modified by applying a form of data compression through prioritization. The setup is initially evaluated at a large intersection in Karlsruhe, Germany.

Reliable traffic light detection is one crucial key component for autonomous driving in urban areas. This includes the extraction of direction arrows contained within the traffic lights as an autonomous car will need this information for selecting the traffic light corresponding to its current lane. Current state of the art traffic light detection systems are not able to provide such information. Within this work we present a hierarchical traffic light detection algorithm, which is able to detect traffic lights and determine their state and contained direction information within one CNN forward pass. This Hierarchical DeepTLR (HDTLR) outperforms current state of the art traffic light detection algorithms in state aware detection and can detect traffic lights with direction information down to a size of 4 pixel in width at a frequency of 12 frames per second.

Navigation and obstacle avoidance are two problems that are not easily incorporated into direct control of autonomous vehicles solely based on visual input. However, they are required if lane following given proper lane markings is not enough to incorporate trained systems into larger architectures. We present a method to allow for obstacle avoidance while driving using a single, front-facing camera as well as navigation capabilities such as taking turns at junctions and lane changes by feeding turn indicator signals into a Convolutional Neural Network. Both situations share the difficulty intrinsic to single camera setups of limited field of views. This problem is handled by using a spatial history of input images to extend the field of view regarding static obstacles. The trained model, referred to as DriveNet, is evaluated in real world driving scenarios, using the same model for lateral vehicle control to both dynamically drive around obstacles as well as perform lane changing and turning in intersections.

We present a reinforcement learning approach using Deep Q-Networks to steer a vehicle in a 3D physics simulation. Relying solely on camera image input the approach directly learns steering the vehicle in an end-to-end manner. The system is able to learn human driving behavior without the need of any labeled training data. An action-based reward function is proposed, which is motivated by a potential use in real world reinforcement learning scenarios. Compared to a naive distance-based reward function, it improves the overall driving behavior of the vehicle agent. The agent is even able to reach comparable to human driving performance on a previously unseen track in our simulation environment.

We propose an integrated approach of combining end-to-end learned trajectory proposals with a probabilistic sampling based planning algorithm for autonomous driving. A convolutional neural network is trained based on monocular image data to predict prospective steering angles. By using a local history of image data, we achieve an implicit spatial representation of parked cars or other obstacles commonly found in urban and residential areas. Through this local history, calculated using the vehicle's velocity data, the trajectory proposals are not only capable of lane following, but also comfortably circumnavigate obstacles. Training data is collected by recording video data and the vehicles CAN bus during human driving, thus imitating human behavior. The integration of end-to-end learning into a modularized architecture allows for additional safety constraints and complementary sensor information to be combined with intuitive steering. Our first results take a promising step towards general architectures for autonomous vehicles that combine deep learning with factorized probabilistic modeling.

In this paper, we present a method to estimate abstract parameters of high definition (HD) maps from sensor data. Parameters we estimate include the distance from ego-vehicle to road boundary, orientation of the ego-vehicle with respect to lanes, number of lanes, and street type. Our method is realized as a Convolutional Neural Network (CNN) that takes pre-processed sensor information in the form of grid map images as input. The estimated parameters of the network can then either be used for localization or to validate existing map data. To generate ground truth training samples, we use a semi-automatic procedure based on a good localization method to align the HD map with the sensor information from the vehicle. Our experiments yield some first promising results of the concept.