The joint detection and classification model for spatiotemporal action localization of primates in a group

Analysis of primate behavior is crucial for neuroscience research and drug evaluation. Although many methods of automatically recording animal behavior have been proposed, none of them can meet the requirements of both speed and accuracy. In order to implement real-time and high-precision automatic recording of primate behavior, we proposed a novel Joint Detection and Classification (JDC) model to predict the location, identity and actions of monkeys simultaneously. Different from the existing complex non-end-to-end models, our model is the first end-to-end method in this field. In order to explore how to efficiently fuse spatiotemporal information, we constructed the JDC model with a single frame or different fusion approaches. In addition, we collected a new dataset named Spatiotemporal Action Localization of Monkeys in a Group (SALMG), which is the first one containing the location, identity and actions of monkeys in a group. The JDC model with middle fusion approach (JDC-MF) on the SALMG dataset outperforms all compared methods. The F1 score of the JDC-MF is 81.4%, which is 15.3% and 10.6% higher than the Separate Detection and Classification model and the two-stage model, respectively. Moreover, the JDC-MF achieves the highest accuracy of 99.1 % on public Pig Novelty Preference Behavior dataset, which is 4.1% higher than the second-best model. The JDC-MF only takes 0.027 s for a clip on a Nvidia GeForce RTX 2080 Ti. Therefore, the JDC-MF can realize real-time and high-precision spatiotemporal action localization of monkeys, and provide an effective reference scheme for automatic recording and analysis of animal behavior. Code has been made available at: https://github.com/Kewei-Liang/JDC-MF .