Fingertip in the Eye: An Attention-Based Method for Real-Time Hand Tracking and Fingertip Detection in Egocentric Videos

The hand and fingertip tracking is the crucial part in the egocentric vision interaction, and it remains a challenging problem due to various factors like dynamic environment and hand deformation. We propose a convolutional neural network (CNN) based method for the real-time and accurate hand tracking and fingertip detection in RGB sequences captured by an egocentric mobile camera. Firstly, we build a large scale dataset, Ego-Finger, containing plenty of scenarios and human labeled ground truth. Secondly, we propose a two stage CNN pipeline, i.e., the human vision inspired Attention-based Hand Tracker (AHT) and the hand physical constrained Multi-Points Fingertip Detector (MFD). Comparing with state-of-the-art methods, the proposed method achieves very promising results in the real-time fashion.