We present a flow-matching planner for autonomous driving that directly outputs actionable control trajectories defined by acceleration and curvature profiles. The model is conditioned on a bird's-eye-view (BEV) raster of the surrounding scene and generates control sequences in a small number of Ordinary Differential Equations (ODE) integration steps, enabling low-latency inference suitable for real-time closed-loop re-planning. We train exclusively on urban scenarios (real urban city streets, intersections and roundabouts of the city of Parma, Italy) collected from a 2D traffic simulator with reactive agents, and evaluate in closed-loop on both in-distribution and markedly out-of-distribution environments, including multi-lane highways and unseen urban scenarios. Our results show that the model generalizes reliably to these unseen conditions, maintaining stable closed-loop control and successfully completing scenarios that differ substantially from the training distribution. We attribute this to the BEV representation, which provides a geometry-centric view of the scene that is inherently less sensitive to distributional shifts, and to the flow-matching formulation, which learns a smooth vector field that degrades gracefully under distribution shift. We provide video demonstrations of closed-loop behavior at https://marcelloceresini.github.io/DirectControlFlowMatching.

In the typical autonomous driving stack, planning and control systems represent two of the most crucial components in which data retrieved by sensors and processed by perception algorithms are used to implement a safe and comfortable self-driving behavior. In particular, the planning module predicts the path the autonomous car should follow taking the correct high-level maneuver, while control systems perform a sequence of low-level actions, controlling steering angle, throttle and brake. In this work, we propose a model-free Deep Reinforcement Learning Planner training a neural network that predicts both acceleration and steering angle, thus obtaining a single module able to drive the vehicle using the data processed by localization and perception algorithms on board of the self-driving car. In particular, the system that was fully trained in simulation is able to drive smoothly and safely in obstacle-free environments both in simulation and in a real-world urban area of the city of Parma, proving that the system features good generalization capabilities also driving in those parts outside the training scenarios. Moreover, in order to deploy the system on board of the real self-driving car and to reduce the gap between simulated and real-world performances, we also develop a module represented by a tiny neural network able to reproduce the real vehicle dynamic behavior during the training in simulation.

Navigating through intersections is one of the main challenging tasks for an autonomous vehicle. However, for the majority of intersections regulated by traffic lights, the problem could be solved by a simple rule-based method in which the autonomous vehicle behavior is closely related to the traffic light states. In this work, we focus on the implementation of a system able to navigate through intersections where only traffic signs are provided. We propose a multi-agent system using a continuous, model-free Deep Reinforcement Learning algorithm used to train a neural network for predicting both the acceleration and the steering angle at each time step. We demonstrate that agents learn both the basic rules needed to handle intersections by understanding the priorities of other learners inside the environment, and to drive safely along their paths. Moreover, a comparison between our system and a rule-based method proves that our model achieves better results especially with dense traffic conditions. Finally, we test our system on real world scenarios using real recorded traffic data, proving that our module is able to generalize both to unseen environments and to different traffic conditions.

Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. In this work, we investigate these problems in the autonomous driving field, especially for a maneuver planning module for roundabout insertions. In particular, we present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios. Finally, we analyze techniques aimed at reducing the gap between simulated and real-world data showing that this increased the generalization capabilities of the system both on unseen and real-world scenarios.

Novel advance driver assistance systems, such as emergency braking and adaptive cruise control require the most reliable detection algorithms. Furthermore, in the recent years, the use of computer vision approaches in these type of applications is becoming more frequent. However, when dealing with these technologies, reliability is a very important factor that still requires improvement. On this paper, it is presented a tracking algorithm which aims in improving the accuracy of these applications, based on computer vision and modern Probability Hypothesis Density (PHD) Filter technique. The tracking is performed on the features detected within the bounding box provided by a computer video based vehicle detection algorithm. The features tracked are combined in a last stage, providing accurate monocular camera tracking. Test provided, allowed to identify the best method for feature combination. Furthermore, it was proved that under the proper visibility conditions, the PHD filter design is able to improve current methods such as Unscented Kalman Filter. (C) 2017 Elsevier Ltd. All rights reserved.

This paper presents a synthesis of techniques enabling vision-based autonomous Unmanned Aerial Vehicle (UAV) systems. A full stack of computer vision processing modules are used to exploit visual information to simultaneously perceive obstacles and refine localization in a GPS-denied environment. An omni-directional stereo-vision based setup is used to build a 3D representation of the surroundings. A fully 3D local obstacle grid, maintained through multiple frames and updated accordingly to the UAV movement, is built accumulating multiple observations coming from the 360 stereo vision sensing suite. Visual data is also used to extract information regarding the drone attitude and position while exploring the environment. Sparse optical flow collected from both front and down facing stereo cameras is used to estimate UAV movement through multiple frames. The down-looking stereo pair is also used to estimate the drone height from the ground and to refine the pose estimation in a Simultaneous Localization and Mapping (SLAM) fashion. An improved A* planning algorithm exploits both the 3D representation of the surroundings and precise localization information in order to find the shortest path and reach the goal through a three dimensional safe trajectory.

Video sensors based on CMOS technology represent the vast majority of the adopted solutions for image acquisition systems equipped in unmanned vehicles and Advanced Driver Assistance Systems. These sensors typically employ a rolling shutter readout scheme, which causes image distortion in case of motion. This paper investigates how rolling shutter effects influence the performance of Semi-Global Matching, a state of the art dense stereo matching algorithm. In order to exhaustively evaluate those effects, the algorithm is computed over sequences acquired with a dedicated image acquisition system composed by two calibrated and synchronized stereo cameras, the first provided with global shutter sensors and the second with rolling shutter sensors. A pixel-wise comparison between the computed Disparity Space Images allows to determine the error in a field automotive test. An effective correction method is then applied to the rolling shutter Disparity Space Image, aiming to reduce the error and therefore to improve accurate position estimation of obstacles in automotive scenarios.

Obstacles detection is used nowdays for a number of road safety applications, increasing the drivers awareness in potential dangerous situations. A reliable and robust obstacles detection continues to be largely investigated and still remains an open challenge, especially for difficult scenarios and in general cases, with loosened constraints and multiple simultaneous use-cases. This work presents an obstacles detection, tracking and fusion algorithm which allows to reconstruct the environment surrounding the vehicle. While the techniques used for the detection are well-known in literature, the improvements introduced by this paper regard the data association and tracking approach of heterogeneous sensors observations. An innovative multi-dimensional structure based on association costs originating from a classifier provides an optimal solution to the association problem with respect to the total association cost. An Unscented Kalman Filter (UKF) managing a variable number of observations, arbitrarily composable, allows to correctly address the combined tracking and fusion challenge. The results, obtained on a public benchmark, show improvements with respect to state of the art systems.

Three-dimensional environment perception is one of the most important tasks for an autonomous vehicle. Map-based approaches play a fundamental role in the representation of vehicle surroundings, allowing several perception features, such as obstacle detection or road classification. However, benchmarks available in literature do not allow to evaluate the accuracy of these discrete representations, focusing only on the results downstream the maps. The proposed system uses a stochastic approach to evaluate a generic discrete representation of a three-dimensional world. The evaluation process consists in comparing a local perceived representation with the corresponding previously computed ground truth. The ground truth is automatically generated exploiting either accurate depth sensing and precise localization information. A test case is proposed, using stereo vision data and Digital Elevation Maps.

This chapter describes the emerging robotics application field of intelligent vehicles - motor vehicles that have autonomous functions and capabilities. The chapter is organized as follows. Section 62.1 provides a motivation for why the development of intelligent vehicles is important, a brief history of the field, and the potential benefits of the technology. Section 62.2 describes the technologies that enable intelligent vehicles to sense vehicle, environment, and driver state, work with digital maps and satellite navigation, and communicate with intelligent transportation infrastructure. Section 62.3 describes the challenges and solutions associated with road scene understanding a key capability for all intelligent vehicles. Section 62.4 describes advanced driver assistance systems, which use the robotics and sensing technologies described earlier to create new safety and convenience systems for motor vehicles, such as collision avoidance, lane keeping, and parking assistance. Section 62.5 describes driver monitoring technologies that are being developed to mitigate driver fatigue, inattention, and impairment. Section 62.6 describes fully autonomous intelligent vehicles systems that have been developed and deployed. The chapter is concluded in Sect. 62.7 with a discussion of future prospects, while Sect. 62.8 provides references to further reading and additional resources.

The objective of this article is to study the problem of pedestrian classification across different light spectrum domains (visible and far-infrared (FIR)) and modalities (intensity, depth and motion). In recent years, there has been a number of approaches for classifying and detecting pedestrians in both FIR and visible images, but the methods are difficult to compare, because either the datasets are not publicly available or they do not offer a comparison between the two domains. Our two primary contributions are the following: (1) we propose a public dataset, named RIFIR , containing both FIR and visible images collected in an urban environment from a moving vehicle during daytime; and (2) we compare the state-of-the-art features in a multi-modality setup: intensity, depth and flow, in far-infrared over visible domains. The experiments show that features families, intensity self-similarity (ISS), local binary patterns (LBP), local gradient patterns (LGP) and histogram of oriented gradients (HOG), computed from FIR and visible domains are highly complementary, but their relative performance varies across different modalities. In our experiments, the FIR domain has proven superior to the visible one for the task of pedestrian classification, but the overall best results are obtained by a multi-domain multi-modality multi-feature fusion.

This paper presents an autonomous driving test held in Parma on urban roads and freeways open to regular traffic. During this test, the vehicle not only performed simple maneuvers, but it had to cope with complex driving scenarios as well, including roundabouts, junctions, pedestrian crossings, freeway junctions, and traffic lights. The test demonstrated the ability of the current technology to manage real situations and not only the well-structured and predictable ones. A comparison of milestones, challenges, and key results in autonomous driving is presented to highlight the novelty and the specific purpose of the test. The whole system is described: the vehicle; the software architecture; details about high-, medium-, and low-level control; and details about perception algorithms. A conclusion highlights the achieved results and draws possible directions for future development.

Abstract In this work an overview of the local motion planning and dynamic perception framework within the V-Charge project is presented. This framework enables the V-Charge car to autonomously navigate in dynamic mixed-traffic scenarios. Other traffic participants are detected, classified and tracked from a combination of stereo and wide-angle monocular cameras. Predictions of their future movements are generated utilizing infrastructure information. Safe motion plans are acquired with a system-compliant sampling-based local motion planner. We show the navigation performance of this vision-only autonomous vehicle in both simulation and real-world experiments.

The keynote will focus on the latest developments in terms of perception on board of vehicles. Many prototype vehicles have been design and implemented worldwide, each with its own sensor suite; vision is playing an ever increasing role since cameras performance is improving year after year and processing power is becoming available at a low cost.The presentation will highlight the benefits of using vision as the main perception technology and will describe some ongoing experiments on full 360-degree 3D reconstruction in real time.

One of the main challenges in intelligent vehicles concerns pedestrian detection for driving assistance. Recent experiments have showed that state-of-the-art descriptors provide better performances on the far-infrared (FIR) spectrum than on the visible one, even in daytime conditions, for pedestrian classification. In this paper, we propose a pedestrian detector with on-board FIR camera. Our main contribution is the exploitation of the specific characteristics of FIR images to design a fast, scale-invariant and robust pedestrian detector. Our system consists of three modules, each based on speeded-up robust feature (SURF) matching. The first module allows generating regions-of-interest (ROI), since in FIR images of the pedestrian shapes may vary in large scales, but heads appear usually as light regions. ROI are detected with a high recall rate with the hierarchical codebook of SURF features located in head regions. The second module consists of pedestrian full-body classification by using SVM. This module allows one to enhance the precision with low computational cost. In the third module, we combine the mean shift algorithm with inter-frame scale-invariant SURF feature tracking to enhance the robustness of our system. The experimental evaluation shows that our system outperforms, in the FIR domain, the state-of-the-art Haar-like Adaboost-cascade, histogram of oriented gradients (HOG)/linear SVM (linSVM) and MultiFtrpedestrian detectors, trained on the FIR images.

This paper presents the design, the setup, and the architecture of a new class of autonomous vehicles prototype. The car is equipped with 26 cameras, divided into 13 stereo pairs. Four stereo systems are dedicated to the reconstruction of the near area surrounding the vehicle, other nine are dedicated to the 3D reconstruction of the far driving area. Additionally four laserscanners, and a high performance GPS/IMU unit provide more information on the ground truth to measure the validity of the data obtained with vision. The autonomous driving system, the perception system, and the processing system have been installed in the vehicle taking extra care for an as-clean-as-possible integration with te aim of setting a new state of the art level for this kind of setup. A Human Machine Interface, allows the driver to control all the functions of the systems using a touch screen. A high precision synchronization system coupled with a custom software architecture allows to obtain recordings from all sensors installed.

The reliability of a safety sensor is strongly dependent on the range the sensor is able to cover. Safety devices for electric automatic gates, mainly photodetectors, with their reduced sensing capabilities, can not provide the complete avoidance of injuries so that to guarantee protection for people crossing the danger area additional devices must be introduced. In this paper we evaluate as safety sensor a solo vision system fitting sliding and swing gates, the most widespread gate sorts for residential, commercial and industrial automations. Two cameras are placed at the opposite ends of the gate doorway looking at the same area to be monitored. The evaluation is carried out on both virtual and real image sequences in order to verify the practical suitability of the system after the theoretical trials exploitation.

The presence of autonomous vehicles on public roads is becoming a reality. In the last 10 years, autonomous prototypes have been confined in controlled or isolated environments, but new traffic regulations for testing and direct automotive companies interests are moving autonomous vehicles tests on real roads. This paper presents a test on public urban roads and freeways that was held in Parma on July 12, 2013. This was the first test in open public urban roads with nobody behind the steering wheel: the vehicle had to cope with roundabouts, junctions, pedestrian crossings, freeway junctions, traffic lights, and regular traffic. The vehicle setup, the software architecture, and the route are here presented together with some results and possible future improvements.