Egocentric Recognition of Handled Objects: Benchmark and Analysis

Recognizing objects being manipulated in hands can provide essential information about a person's activities and have far-reaching impacts on the application of vision in everyday life. The egocentric viewpoint from a wearable camera has unique advantages in recognizing handled ob- jects, such as havinga close view andseeing objectsin their natural positions. We collect a comprehensive dataset and analyze the feasibilities and challenges of the egocentric recognition of handled objects. We use a lapel-worn camera and record uncompressed video streams as human subjects manipulate objects in daily activities. We use 42 day-to-day objects that vary in size, shape, color and textureness. 10 video sequences are shot for each object under different illuminations and back- grounds. We use this dataset and a SIFT-based recogni- tion system to analyze and quantitatively characterize the main challenges in egocentric object recognition, such as motion blur and hand occlusion, along with its unique con- straints, such as hand color, location prior and temporal consistency. SIFT-based recognition has an average recog- nitionrate of12%, andreaches20%throughenforcingtem- poral consistency. We use simulations to estimate the upper bound for SIFT-based recognition at 64%, the loss of ac- curacy due to background clutter at 20%, and that of hand occlusion at 13%. Our quantitative evaluations show that the egocentric recognition of handled objects is a challeng- ing but feasible problem with many unique characteristics and many opportunities for future research.