Acquisition Of 3-D Trajectories With Labeling Support For Multi-Species Insects Under Unconstrained Flying Conditions

In the work presented here, a new technique based upon stereo vision is proposed to acquire three-dimensional time-resolved trajectories with labeling support for multi-species insects under unconstrained flying conditions. A low-cost, off-the-shelf depth camera is used for stereo vision which is equipped with two wide-angle globalshutter IR imagers to compute depth map and a separate narrow-angle color imager to have color and texture information. Two novel strategies are employed to tackle the challenges imposed by the small size of insects, fast movements, and natural daylight, etc.; (i) introduction of a simple but robust technique for detection and tracking of insects using only the valid depth map data and subsequently to acquire 3-D trajectories (ii) object localization scheme for the insects in acquired trajectories in allied color frames based upon the analysis of insects' flying kinematics and their 3-D location coordinates. The object localization of flying insects in allied color frames provides support to label the multi-spp. trajectories with respective types of the species. The proposed technique provides completely generalized solution to acquire 3-D trajectories of the multi-spp. insects in natural outdoor conditions and in the presented work it is applied to acquire the trajectories of honey bees (Apis mellifera) and invasive hornets (Vespa spp.) near beehives. These hornets are serious honey bee stressors and cause severe damage to the hive's foraging force. The trajectory patterns of honey bees together with Vespa spp. can be analyzed for early detection of Vespa spp. near beehives as well as to estimate stress levels on the honey bees. Acquired 3-D trajectory data are validated for trajectory measurements as well as for the labeling support information. A mean error upper bound of 5 mm is estimated in trajectory measurements whereas 100% interspp. and up to 94% intra-spp. labeling support accuracies are recorded.