Using vision based tracking to support real-time graphical instruction for students who have visual impairments

In this paper, we describe a real-time vision-based tracking system to help students who are blind or visually impaired (SBVI) to follow instructional discourse that employs graphical illustrations. The vision system employs a color model based tracking for both the instructor's pointing behavior and the SBVI's reading behavior, and maps the pointing positions into the same coordinates. Our Haptic Deictic System - HDS system also employs a haptic glove to provide SBVI access to the situated pointing behavior of instructor that is performed in conjunction with speech, and provides the instructor real-time visual feedback on the SBVI's reading actions. Thus, our system supports two-way situated multimodal communication. In this paper, we first introduce our system, and then we discuss studies that show the efficacy of our approach.