Multimodal Road Sign Interpretation for Autonomous Vehicles.

Autonomous vehicles (AVs) are becoming increasingly prevalent. However, current AVs are unable to handle unexpected traffic signs (e.g., construction zones, road closures) encountered on the roads. To address this limitation, we propose MOSER, a Multimodal rOad Sign intERpretation system, to enable automated detection and interpretation of diverse road signs. Our system consists of a pipeline architecture with three main components, including perception, text processing, and planning. The perception component detects arbitrary road signs and extracts the sign text into proper groups and orders. The text processing component then identifies the high-level semantics of the text and determines whether any actions are required for the autonomous vehicle. Based on the interpretation of the signs, the planning component provides navigation guidance, such as instructing the vehicle to stop at a specific location or adding rules to its internal map. To the best of our knowledge, this is the first attempt to address the interpretation of arbitrary road signs using a multimodal processing strategy. Our work provides important insights and capabilities to support Level 4 autonomous vehicles, ensuring their safety and smoothness of operation.