Actions as points: a simple and efficient detector for skeleton-based temporal action detection

Temporal action detection, aiming to determine the fragment and category of a human action simultaneously from continuous data stream, is still a challenge issue in the field of human–robot interaction, somatosensory game and security monitoring. In this paper, we present a novel one-stage skeleton-based TAD method, Action-CenterNet(ACNet) with a simple anchor-free and fully convolutional encoder-decoder pipeline. Our approach encodes skeleton position and motion data sequence from multiple persons into multi-channel skeleton images which are subsequently preprocessed by view invariant transform and translation-scale invariant. ACNet models each action fragment as a center point along the time dimension and generates a keypoint heatmap to locate and classify action fragments. To ensure the accurate temporal coordinates, the discretization error caused by the output stride of network is also learned. Compared with two-stage methods, ACNet is end-to-end differential and flexible. ACNet is also an anchor-free method avoiding the drawbacks of anchor boxes used in anchor-based TAD methods. Experimental results on PKU-MMD dataset, NTU RGB-D dataset and HITvs dataset reveal the excellent performance of our approach.