Building a Stereo and Wide-View Hybrid RGB/FIR Imaging System for Autonomous Vehicle

In this work, a novel paradigm is proposed to build a hybrid RGB/FIR camera, which outputs pixel-wisely aligned and temporally synchronized RGB and FIR frames simultaneously. Compared with the existing schemes, we innovate in several ways to make it compact, practical, and extendable for real-world applications. First, we propose a new structure layout design of the hybrid camera, and we also design a removable adjuster to detach the hybrid camera from the base platform. For RGB and FIR image registration, we have achieved a pixel-wise spatial registration of the FIR and RGB frames by a two-step alignment, where we propose a ghost-effect removal scheme to remove the ghost effect from the rolling-shutter-based FIR camera. Finally, multiple hybrid cameras are extended to a hybrid camera array, through which we can obtain wide-view spatially aligned FIR, RGB and disparity images simultaneously. The experimental results demonstrate the accuracy of the hybrid camera and the effectiveness of the designed stereo wide-view hybrid camera system.