Camera models play a crucial role in robot vision applications. Yet their diversity poses a challenge when working with data captured with cameras calibrated using different models. In this paper, we address this issue by introducing a mathematical framework that enables conversion between various camera projection models. This approach allows algorithms designed for a specific model to process data from cameras calibrated with other models, eliminating the need for recalibration and enabling the reuse of pre-existing datasets that do not provide access to calibration images. We present the general conversion method for state-of-the-art camera models that we derive for three new camera model conversions, covering various camera types, including fisheye and catadioptric systems. Quantitative evaluation is conducted with respect to well-known calibration methods. We compare our method on image undistortion, as well as in practical applications such as SLAM, visual servoing, and visual odometry. The results demonstrate that our conversion approach achieves performances comparable to calibration without the need for explicit calibration. This work contributes to a more flexible and adaptive use of cameras in robot applications. The proposed camera model conversion framework is implemented in the open-source libPeR library.

Unmanned Aerial Vehicles (UAVs) are popular robotic platforms nowadays and they are gradually entering our everyday lives. For normal operation, a two-way communication link between a UAV and a ground station, has to be established. While radio-frequency communication is a simple and common option, it suffers from major drawbacks, such as interference and multi-path errors. Moreover, it is prone to spoofing/jamming attacks which might have catastrophic consequences in denselypopulated or sensitive areas. To overcome these limitations, in this paper, we propose a compact LED-event camera system for fast optical communication via a new optimized dynamic $N$-pulse protocol. A learning-based method guarantees the robust detection and tracking of the active markers and it makes the application of standard 3D pose estimation algorithms possible. The effectiveness of the proposed system is demonstrated via hardware experiments with a DJI Matrice 600 Pro hexarotor and a Prophesee Gen. 4.1 HD camera.

This paper addresses the problem of omnidirectional video interpolation, which plays an essential role in applications such as virtual reality and immersive video enhancement. Existing video interpolation methods are not well-suited for spherical videos, as they have difficulty handling severe distortions close to the poles. To address this issue, we propose SVI360, a dual-branch framework that combines the image frame and its rotated orthogonal view to deal with these distortions. The core methodological aspect of the approach is to reinforce equivariance of the flow displacements between the original and orthogonal views to improve intermediate frame prediction. Experiments show that our method outperforms state-of-the-art approaches in interpolation quality while maintaining accurate optical flow in four different public benchmarks. Code and pre-trained models are available at: https://icb-vision-ai.github.io/video360_interpolation/

The advent of event-based cameras, with their low latency, high dynamic range, and reduced power consumption, marked a turning point in machine perception and robotic vision. In particular, the combination of these neuromorphic sensors with widely available passive or active optical markers (e.g. AprilTags and arrays of blinking LEDs) has recently opened up a new field of opportunities. This survey article provides a comprehensive review of event-based optical marker systems (EBOMS). We analyze the underlying principles and technologies on which these systems are based, with a special focus on the systems’ asynchronous operation and robustness against challenging lighting conditions. We also describe the most relevant applications of EBOMS, including object detection and tracking, pose estimation, and optical communication. The article concludes with a discussion of possible future research directions in this rapidly emerging and multidisciplinary area.

In many autonomous applications requiring real-time localization, active marker-based systems are preferred due to their low latency and ease of deployment compared to computationally demanding feature-based methods. Event offer high temporal resolution and minimal delay and are commonly used with active LED markers for robust real-time localization. Existing methods typically rely on Perspective-n-Point (PnP) solvers for pose estimation. However, structured marker layouts can be challenging to deploy in space-constrained scenarios, while partial self-motion information (e.g., gravity direction and altitude) is readily available from onboard sensors. We derive a robust and accurate minimal solver that estimates camera pose from only two LED markers by incorporating known tilt angle and camera height measured by an onboard sensor, such as an IMU or an altimeter. The proposed formulation uniquely determines the camera pose through both a closed-form and a linear least-squares solution. We further analyze degenerate configurations and characterize the conditions under which height information does not contribute to rotation estimation. For evaluation, we developed an event-based active marker system to collect real-world data with ground truth from a motion capture system. Experiments on both synthetic and real data demonstrate improved accuracy over the state-of-the-art P2P solver and competitive performance relative to P3P.

In this paper, we present a new compact vision sensor consisting of two fisheye event cameras mounted back-to-back, which offers a full 360-degree view of the surrounding environment. We describe the optical design, projection model and practical calibration using the incoming stream of events, of the novel stereo camera, called SFERA. The potential of SFERA for real-time target tracking is evaluated using a Bayesian estimator adapted to the geometry of the sphere. Real-world experiments with a prototype of SFERA, including two synchronized Prophesee EVK4 cameras and a DJI Mavic Air 2 quadrotor, show the effectiveness of the proposed system for aerial surveillance.

Autonomous driving systems require robust object detection in complex environments. Event cameras outperform RGB cameras under challenging lighting conditions, but face limitations due to the scarcity of available datasets and lack of specialized training. To narrow the gap between RGB- and event-based detection accuracy and avoid the high complexity of real-time RGB-event fusion, in this paper, we propose a knowledge distillation framework. Our approach uses both modalities during training but relies solely on sparse event data at inference and transfers knowledge from a robust RGB-based teacher model. We build on the success of DETR (DEtection TRansformer) and we leverage an event-aware masked knowledge distillation mechanism, to boost event-based detection accuracy. Experiments on the DSEC-DET dataset demonstrate that our method not only excels in challenging driving scenarios where RGB images are unreliable, but also surpasses the state-of-the-art in event-based object detection.

In this paper, we study the problem of estimating the orientation of an event omnidirectional camera mounted on a robot and observing 3D parallel lines in a man-made environment (Manhattan world). We present Gyrevento, the first event-based omnidirectional visual gyroscope. Gyrevento does not require any initialization, provides certifiably globally optimal solutions, and is scalable, since the size of the nonlinear least-squares cost function is independent of the number of lines. Thanks to the Cayley-Gibbs-Rodrigues parameterization of a 3D rotation, this cost function is a degree-four rational function in three variables, which can be efficiently minimized via off-the-shelf polynomial optimization software. Numerical simulations and real-world experiments with a robot manipulator show the effectiveness of our visual gyroscope and elucidate the impact of camera velocity on the attitude estimation error.

Autonomous driving systems rely on accurate perception and localization of the ego car to ensure safety and reliability in challenging real-world driving scenarios. Public datasets play a vital role in benchmarking and guiding advancement in research by providing standardized resources for model development and evaluation. However, potential inaccuracies in sensor calibration and vehicle poses within these datasets can lead to erroneous evaluations of downstream tasks, adversely impacting the reliability and performance of the autonomous systems. To address this challenge, we propose a robust optimization method based on Neural Radiance Fields (NeRF) to refine sensor poses and calibration parameters, enhancing the integrity of dataset benchmarks. To validate improvement in accuracy of our optimized poses without ground truth, we present a thorough evaluation process, relying on reprojection metrics, Novel View Synthesis rendering quality, and geometric alignment. We demonstrate that our method achieves significant improvements in sensor pose accuracy. By optimizing these critical parameters, our approach not only improves the utility of existing datasets but also paves the way for more reliable autonomous driving models. To foster continued progress in this field, we make the optimized sensor poses publicly available, providing a valuable resource for the research community.

As part of the development of many robotic systems for the forestry sector, forest scene understanding requires the use of computer vision algorithms. However, this dense and unstructured environment is complex and puts conventional detection approaches to the test. In the case of tree instance segmentation, the presence of closely spaced or even intertwined trees, their highly variable shapes, and complex masks due to their branches and leaves are just some of the challenges to be overcome. For this, specific learning of tree boundaries is required to better distinguish one from another. In this letter, we propose ConvexMask, a convolutional neural network for real-time instance segmentation. ConvexMask opts for a label representation approach with a convex exterior polygon, defined by tree extremities, and a binary mask to handle the detail and occlusions that the label may contain. Experiments conducted on the SynthTree43k dataset show that ConvexMask distinguishes tree extremities better than state-of-the-art networks, resulting in better-quality masks.

About a decade ago the idea of cooperation has been introduced to self-driving with the aim to enhance safety in dangerous places such as intersections. Infrastructure-based cooperative systems emerged very recently bringing a new point of view of the scene and more computation power. In this paper, we want to go beyond the framework presented in the vehicle-to-infrastructure (V2I) cooperation by including the vehicle's point of view in the perception of the environment. To keep the cost low, we decided to use only two-dimensional bounding boxes, thus depriving ourselves of depth information that contrasts with state-of-the-art methods. With this in-the-scene point-of-view, we propose a new framework to generate a cooperative evidential occupancy grid based on the Dempster-Shafer Theory and which employs a Monte Carlo framework to incorporate position noise in our algorithm. We also provide a new cooperative dataset generator based on the CARLA simulator. Finally, we provide an extended review of our new cooperative occupancy grid map generation method which improves the state-of-the-art techniques.

Self-supervised monocular depth prediction is a widely researched field that aims to provide a better scene understanding. However, most existing methods prioritize prediction accuracy over computation cost, which can hinder the deployment of these methods in real-world applications. Our objective is to propose a solution that efficiently compresses the depth map while maintaining a high level of accuracy for navigation purpose. The proposed method is an expansion of the work presented in N-QGN, which utilizes a quadtree representation for compression. This approach has already shown promising results, but we aim to improve it further by making it more accurate, faster, and easier to train. Therefore, we introduce a new method that directly predicts the quadtree structure, resulting in a more consistent prediction, and we revise the network architecture to be lighter and produce state-of-the-art accuracy results, depending on the data compression rate. The new implementation is also faster, making it more suitable for real-time applications. Experiments have been conducted on various scene configuration highlighting the capability of the method to efficiently predicting a reliable quadtree depth representation of the scene at low computation cost and high accuracy.

Reliable multimodal sensor fusion algorithms require accurate spatiotemporal calibration. Recently, targetless calibration techniques based on implicit neural representations have proven to provide precise and robust results. Nevertheless, such methods are inherently slow to train given the high computational overhead caused by the large number of sampled points required for volume rendering. With the recent introduction of 3D Gaussian Splatting as a faster alternative to implicit representation methods, we propose to leverage this new rendering approach to achieve faster multi-sensor calibration. We introduce 3DGS-Calib, a new calibration method that relies on the speed and rendering accuracy of 3D Gaussian Splatting to achieve multimodal spatiotemporal calibration that is accurate, robust, and with a substantial speed-up compared to methods relying on implicit neural representations. We demonstrate the superiority of our proposal with experimental results on sequences from KITTI-360, a widely used driving dataset.

On the one hand, cameras of conventional field-of-view usually considered in computer vision and robotics are very often modeled as a pinhole plus possibly a distortion model. On the other hand, there is a large variety of models for panoramic cameras. Many camera models have been proposed for fisheye cameras, catadioptric cameras, and super fisheye cameras. But in both cases, few models offer the possibility of converting them into another model.This paper contributes to filling this gap in, to allow an algorithm designed with a projection model to accept data of a camera calibrated with another model. So, a pre-existing data set can be used without having to recalibrate the camera. We provide the methodology and mathematical developments for three conversions considering three different types of cameras that are evaluated with respect to calibration and within a visual Simultaneous Localization And Mapping benchmark. The source code of the camera model conversions studied in this paper is shared within the libPeR library for Perception in Robotics: https://github.com/PerceptionRobotique/libPeRbase.

Graphs have been used in different fields of research for performing structural analysis of various systems. In order to compare the structure of two systems, the correspondence between their graphs has to be verified. The problem of graph matching, especially subgraph isomorphism (SI), has been well studied in case of static graphs. However, many applications require incorporating temporal information, making the corresponding graphs dynamic. In this paper, we apply SI to detect dynamic patterns in dynamic graphs. We propose an algorithm for induced SI to detect all the matchings for a given pattern graph while considering snapshot-based representation of dynamic graphs and taking into account the chronological order of these snapshots. This is the novelty of the proposed approach since the existing state-of-the-art algorithms model dynamic graphs using an aggregated model with time-stamped edges. To the best of our knowledge, there does not exist another approach which considers snapshot-based representation of dynamic pattern and dynamic target graphs for this problem. We discussed the time complexity of our algorithm and tested its performance while comparing it with two existing algorithms using the real-world datasets. It was found that our algorithm is the second best overall in terms of the execution time. The results are promising given the fact that the choice of dynamic graph model affects the algorithmic design for solving the problem of SI. For the applications where aggregated model of dynamic graphs is not applicable and snapshot-based representation is indispensable, our algorithm can be directly applied as opposed to the existing ones.

With the recent advances in autonomous driving and the decreasing cost of LiDARs, the use of multimodal sensor systems is on the rise. However, in order to make use of the information provided by a variety of complimentary sensors, it is necessary to accurately calibrate them. We take advantage of recent advances in computer graphics and implicit volumetric scene representation to tackle the problem of multi-sensor spatial and temporal calibration. Thanks to a new formulation of the Neural Radiance Field (NeRF) optimization, we are able to jointly optimize calibration parameters along with scene representation based on radiometric and geometric measurements. Our method enables accurate and robust calibration from data captured in uncontrolled and unstructured urban environments, making our solution more scalable than existing calibration solutions. We demonstrate the accuracy and robustness of our method in urban scenes typically encountered in autonomous driving scenarios.

Semantic image segmentation is an essential task for autonomous vehicles and self-driving cars where a complete and real-time perception of the surroundings is mandatory. Convolutional Neural Network approaches for semantic segmentation standout over other state-of-the-art solutions due to their powerful generalization ability over unknown data and end-to-end training. Fisheye images are important due to their large field of view and ability to reveal information from broader surroundings. Nevertheless, they pose unique challenges for CNNs, due to object distortion resulting from the Fisheye lens and object position. In addition, large annotated Fisheye datasets required for CNN training is rather limited. In this paper, we investigate the use of Deformable convolutions in accommodating distortions within Fisheye image segmentation for fully residual U-net by learning unknown geometric transformations via variable shaped and sized filters. The proposed models and integration strategies are exploited within two main paradigms: single(front)-view and multi-view Fisheye images segmentation. The validation of the proposed methods is conducted on synthetic and real Fisheye images from the WoodScape and the SynWoodScape datasets. The results validate the significance of the Deformable fully residual U-Net structure in learning unknown geometric distortions in both paradigms, demonstrate the possibility in learning view-agnostic distortion properties when trained on the multi-view data and shed light on the role of surround-view images in increasing segmentation performance relative to the single view. Finally, our experiements suggests that Deformable convolutions are a powerful tool that can increase the efficiency of fully residual U-Nets for semantic segmentation of automotive fisheye images.

In this paper, we present a relative pose estimation algorithm based on lines knowing the vertical direction associated to each image. We demonstrate that a closed-form solution requiring only eight lines between three views is possible. As a linear solution, it is shown that our approach outperforms the standard trifocal estimation based on 13 triplets of lines and can be efficiently inserted into an hypothesize-and-test framework such as RANSAC. We also study our approach on different singular configurations of lines. The method is evaluated on both synthetic data and real-world sequences from KITTI and the Zürich Urban Micro Aerial Vehicle datasets. Our method is compared to 13 lines algorithm as well to points based methods such as 7-points, 5-points and 3-points.

Surround-view cameras are a primary sensor for automated driving, used for near-field perception. It is one of the most commonly used sensors in commercial vehicles primarily used for parking visualization and automated parking. Four fisheye cameras with a $190^{\circ }$ field of view cover the $360^{\circ }$ around the vehicle. Due to its high radial distortion, the standard algorithms do not extend easily. Previously, we released the first public fisheye surround-view dataset named WoodScape. In this work, we release a synthetic version of the surround-view dataset, covering many of its weaknesses and extending it. Firstly, it is not possible to obtain ground truth for pixel-wise optical flow and depth. Secondly, WoodScape did not have all four cameras annotated simultaneously in order to sample diverse frames. However, this means that multi-camera algorithms cannot be designed to obtain a unified output in birds-eye space, which is enabled in the new dataset. We implemented surround-view fisheye geometric projections in CARLA Simulator matching WoodScape’s configuration and created SynWoodScape. We release 80 k images from the synthetic dataset with annotations for 10+ tasks. We also release the baseline code and supporting scripts.