Aim Where You Look: Eye-Tracking-Based UAV Control Framework for Automatic Target Aiming

Unmanned aerial vehicles (UAVs) often require real-time and accurate control in military, rescue, and transportation applications. There is an urgent demand from operators for usability and interoperability. Human-machine collaboration enhancement and tiny machine learning aimed at edge computing hold the potential to greatly improve the operational efficiency of UAVs. In this paper, we propose a novel UAV control framework based on eye-tracking technology, i.e. Eye-Tracking-based UAV (ETUAV), which combines with lightweight object detection to assist the control of the UAV for attitude adjustment through the gaze information of the UAV operator. The eye-tracking-based UAV control method we propose involves real-time tracking of the operator’s gaze using our self-developed head-mounted eye tracker, combined with object detection to achieve automatic targeting. We also propose an incremental proportion integration differentiation (PID) control algorithm for adjusting the UAV’s attitude, which provides automatic and real-time UAV control. We develop a system prototype based on the DJI UAV and conduct performance benchmarks and comparisons. The experimental results indicate that the operational latency of our method is significantly less than manual operation and the built-in automatic tracking function in the DJI application. The “Aim Where You Look” UAV operation framework proposed in this article significantly streamlines the tasks for operators in time-critical scenarios.