Smart Cap: A Deep Learning and IoT Based Assistant for the Visually Impaired

India is home to the largest number of visually impaired people in the world, about 40 million, which accounts for 20% of the world's blind population. Moreover, more than 90% of these people have little to no access to the necessary assistive technologies. The paper proposes `Smart Cap,' a first-person vision-based assistant, aimed at bringing the world as a narrative to these visually impaired people of India. The Smart Cap acts as a conversational agent bringing together the disciplines of Internet of Things and Deep Learning and provides features like face recognition, image captioning, text detection and recognition, and online newspaper reading. The hardware architecture consists of a Raspberry Pi, a USB webcam, a USB microphone, earphones, power source, and extension cables. The user can interact with the Smart Cap by giving specific commands, which trigger the corresponding feature module that returns an audio output. The face recognition module is based on the dlib's face recognition project. It is a two-step process of detecting a face in the image and identifying it. The image captioning task synthesizes an attention-based CNN-LSTM encoder-decoder model coupled with beam search for finding the best caption. Google's Vision API service is used for text detection and recognition. An additional feature of online newspaper reading is also provided, thus, keeping the blind person up to date with the daily news.