In the realm of human–computer interaction, convolutional neural networks (CNN) have recently gained increasing attention for solving hand gesture recognition problems. Numerous existing CNN-based architectures perform well with recognition accuracy but may result in high computational complexity and require excessive resources when deployed on embedded devices. In this paper, an efficient hardware accelerator based on Tiny YOLOv2 networks using binary weights and low-bit activations is proposed to address the above issue. The processing elements (PEs) are designed to leverage low-bit calculations and resource allocation is employed to improve hardware performance. Furthermore, the design space of an accelerator is explored by fine-tuning parameters such as parallelization factors and clock rates. Experiments conducted on the PASCAL VOC and Hindi Indian Sign Language (ISL) datasets demonstrate that the Tiny YOLOv2 model with 1-bit weights and 5-bit activations achieves impressive mean average precision (mAP) scores of 50.2