My research focuses on designing machines that measure, decode, and learn the deeper meaning of human behaviors by exploiting the in-situ measurements of wearable cameras. A simple usage of the wearable cameras does not solve behavioral understanding problems due to the first person biases such as camera placement, anthropometric configurations, and physical/social interactions. Therefore, representing human behaviors via first person perception is challenging, and I address this challenge through the following three ingredients: • Joint attention To understand social behaviors, e.g., social formations, in a form of joint attention, or social saliency (Figure 1(a)) [1, 2, 3, 4]. • Physical sensation To predict one’s behaviors by decoding first person sensation into physical quantities such as force, momentum, and energy (Figure 1(b)) [5, 6, 7]. • Social signal To recognize the meaning of human behaviors, e.g., social signals, by reconstructing their activities in 3D (Figure 1(c)) [8, 9, 10, 11, 12]. In my Ph.D. and postdoctoral research, I have demonstrated the validity of my representation through video editing [3], sport analytics [4], performance capture [12], and behavior prediction [1, 2, 6]. Research projects have been featured in major media including IEEE Spectrum, NBC News, Discovery News, and Wired, and I have co-organized a tutorial based on my thesis [13], “Group Behavior Analysis and Its Applications1 ” in conjunction with CVPR 2015.