Position Puzzle Network and Augmentation: localizing human keypoints beyond the bounding box

When estimating human pose with a partial image of a person, we, humans, do not confine the spatial range of our estimation to the given image and can readily localize keypoints outside of the image by referring to visual clues such as the body size. However, computational methods for human pose estimation do not consider those keypoints outside and focus only on the bounded area of a given image. In this paper, we propose a neural network and a data augmentation method to extend the range of human pose estimation beyond the bounding box. While our Position Puzzle Network expands the spatial range of keypoint localization by refining the position and the size of the target’s bounding box, Position Puzzle Augmentation enables the keypoint detector to estimate keypoints not only within, but also beyond the input image. We show that the proposed method enhances the baseline keypoint detectors by 39.5% and 30.5% on average in mAP and mAR, respectively, by enabling the localization of keypoints out of the bounding box using a cropped image dataset prepared for proper evaluation. Additionally, we verify that the proposed method does not degrade the performance under the original benchmarks and instead, improves the performance by alleviating false-positive errors.