SoC-SLAM: FPGA-Based Hardware/Software Co-Design for Real-Time Visual SLAM Front-End and Back-End Acceleration

ORB-SLAM3 is a state-of-the-art visual SLAM system, but its computational complexity poses major challenges for real-time deployment on embedded platforms. While prior work has largely focused on accelerating front-end tasks like feature extraction, back-end stages such as bundle adjustment remain less explored due to their algorithmic complexity and memory-intensive nature. Furthermore, most existing solutions accelerate specific modules without integrating them into a complete SLAM framework. In this work, we present SoC-SLAM, a novel FPGA-based hardware/software co-design that accelerates both the front-end and back-end stages of ORB-SLAM3 within a unified framework. Profiling identifies ORB feature extraction and local bundle adjustment as the primary performance bottlenecks, both critical for maintaining real-time system responsiveness. To address these, we develop modular FPGA-based accelerators for ORB extraction and key bundle adjustment solver steps, including Schur elimination, Cholesky decomposition, and back substitution, while retaining the remaining pipeline in software. Since the workload predominantly consists of operations on sparse block matrices, we develop and combine several optimization techniques, including matrix partitioning and pipelined block processing, to efficiently handle sparsity and maximize parallelism. Evaluation on the EuRoC MAV dataset shows 8x and 7.4x speedups for ORB extraction and local bundle adjustment, resulting in 2.4x and 3.3x improvements in the Tracking and Local Mapping threads, respectively. The system operates at 222 MHz, consumes 4.276W, and achieves an RMSE of 0.02832 m. Our fully integrated pipeline demonstrates competitive performance and power efficiency compared to prior FPGA, ASIC, and GPU-based solutions. The proposed architecture is scalable and generalizable to other bundle adjustment modules, such as global bundle adjustment, welding bundle adjustment, and essential graph optimization, offering an extensible hardware acceleration design for real-time visual SLAM on resource-constrained platforms.