THAT-Net: Two-layer hidden state aggregation based two-stream network for traffic accident prediction

Traffic accident prediction based on dashboard cameras is of great significance for ensuring the guaranteed-safety of self-driving systems and reducing the occurrence of accidents. Due to complexity of traffic scenarios and wide range of object's motion, early prediction of accidents remains a challenge. However, the existing methods process all traffic objects and all video frames indiscriminately, thus leading to poor prediction performance. To address this issue, we propose a novel traffic accident prediction framework, namely two-layer hidden state aggregation based two-stream network (THAT-Net). The proposed method first fuses the spatial and temporal flow to capture complementary spatio-temporal information, which filters out irrelevant objects in traffic scenes to reduce the influence of the object's motion state. Furthermore, a two-layer hidden state aggregation structure is designed to reintegrate the hidden state weights of gated recurrent units. It captures contextual information through frame-level and segment-level aggregation and decreases the influence of irrelevant frames to reduce the complexity of traffic scenes. Experiments based on two real-world datasets show the state-of-the-art performance of THAT -Net. Our proposed method achieves the highest accuracy by predicting accidents 0.48 sec to 2.8 sec earlier compared to the baseline methods in more challenging situations. Our code is available at: https://github .com /redeyezt /THAT-Net.