Abstract Automatic re‐identification of animals has significant potential to address pressing ecological and conservation challenges through improved population monitoring, individual health assessment and detailed behavioural analyses. Although numerous computer‐vision‐based solutions have been proposed and many achieve high accuracy, most remain unsuitable for real‐time analysis and deployment on low‐power edge devices (e.g. drones, camera traps). Here, we address both aspects and introduce an open‐source tool for Real‐time Animal Pattern re‐Identification on edge Devices (RAPID). RAPID processes over 40–60 cropped bounding box images per second on a standard PC or laptop and more than 10 images on an inexpensive off‐the‐shelf edge device. The algorithm operates efficiently in data‐ and compute‐limited environments, relying solely on CPU, leaving GPU resources available for other tasks, all while maintaining or even surpassing state‐of‐the‐art accuracy. Furthermore, each prediction is accompanied by a data‐driven confidence score, facilitating reliable downstream use. Our approach leverages SIFT (scale‐invariant feature transform) descriptors, which continue to demonstrate competitive robustness and accuracy against recent traditional and deep‐learning‐based methods. To overcome SIFT's main limitation—its high‐dimensional feature vectors and the associated computational cost—we integrate recent advances in vector similarity search beyond constructing a database of feature vectors rather than database images, thereby accelerating the query processing. The resulting pipeline is carefully designed to be intentionally minimalistic yet highly effective, retaining only the key components essential for accurate and fast re‐identification. We evaluate RAPID on six datasets: four publicly available animal re‐identification benchmark datasets and two new identity‐labelled datasets we release alongside this paper, namely ZebraStereoID, which contains multiview video footage of zebras, and JaguarID, a small camera trap dataset consisting of day‐ and night‐time videos. Our evaluation demonstrates strong generalisability for species, camera systems and environmental conditions. Additionally, we introduce a RAPID‐based tool, FalseTagFinder, for cleaning benchmark dataset labels, providing corrected labels for the StripeSpotter dataset as an example. Data, code and video abstract available at https://keeper.mpdl.mpg.de/d/228be8688e6146bd8221/?p=%2F&mode=list and https://www.youtube.com/watch?v=O6NWzLEivr8.

Thermal soaring enables birds to perform cost-efficient flights. Although aerodynamic rules dictate the costs of flight, soaring species vary strongly in their morphologies and behavioral strategies. To quantify morphology-related differences in behavioral cross-country strategies, we analyzed a large dataset consisting of over a hundred individuals from 12 soaring species recorded with high-frequency tracking devices. We quantified their performance during thermalling and gliding flights and their overall cross-country behavior. Our results confirmed aerodynamic theory across the species; species with higher wing loading typically flew faster and consequently turned on a larger radius than lighter ones. Furthermore, the combination of circling radius and minimum sink speed determines the maximum benefits soaring birds obtain from thermals. Notably, we observed a spectrum of strategies regarding the adaptivity to thermal strength and uncovered a universal rule for cross-country strategies for all analyzed species which can provide inspiration for technical applications, like autopilot for robotic gliders.

Animal groups come in diverse forms-from fish schools swimming in unison to crickets chirping in synchrony. Although these behaviors may seem considerably different to one another, they share a common mathematical core, and can therefore be considered in a unified manner. We discuss the commonalities and differences by synthesizing existing literature from both fields, encompassing both theoretical and empirical advances. We emphasize the crucial role of mixing, induced by individual movements, as a main differentiating factor. Along the way, we propose promising future directions for achieving a more comprehensive understanding of self-organized collective behavior.

Automatic re-identification of animals has significant potential for addressing pressing challenges of our times via population monitoring, individual health assessment or detailed behavior analysis. Although numerous computer vision based solutions have been proposed and many achieve high accuracy, those still remain unsuitable for real-time analysis and deployment on edge devices. Here, we introduce RAPID, an open-source1 algorithm for re-identifying patterned animal images at a rate exceeding 40-60 queries per second on standard PC or Laptop and over 10 queries per second on an inexpensive off-the-shelf edge device. RAPID operates efficiently in computationally restricted environments, relying solely on CPU, leaving GPU resources available for other tasks, all while maintaining state-ofthe-art accuracy. Our approach leverages SIFT descriptors, which continue to demonstrate competitive robustness and accuracy against recent traditional and deep-learning based methods. To counterpoise SIFT’s primary limitations its high-dimensional feature vectors and the associated computational costs - we incorporate recent advancements in vector similarity search and construct a database of feature vectors rather than database images, further accelerating the search process. We evaluate our method, RAPID, on public re-identification datasets; additionally propose a RAPID-based tool, FalseTagFinder, for cleaning benchmark dataset labels and as a demonstration, provide corrections for the StripeSpotter dataset. ### Competing Interest Statement The authors have declared no competing interest. Ministry of Culture and Innovation of Hungary from the National Research, Development and Innovation Fund, Financed under the KDP-2023 Funding scheme, c2264929 project WildCap-Health funded through TUE.AI Agile Research Funds 2024

Revealing the evolved mechanisms that give rise to collective behavior is a central objective in the study of cellular and organismal systems. In addition, understanding the algorithmic basis of social interactions in a causal and quantitative way offers an important foundation for subsequently quantifying social deficits. Here, with virtual reality technology, we used virtual robot fish to reverse engineer the sensory-motor control of social response during schooling in a vertebrate model: juvenile zebrafish (Danio rerio). In addition to providing a highly controlled means to understand how zebrafish translate visual input into movement decisions, networking our systems allowed real fish to swim and interact together in the same virtual world. Thus, we were able to directly test models of social interactions in situ. A key feature of social response is shown to be single- and multitarget-oriented pursuit. This is based on an egocentric representation of the positional information of conspecifics and is highly robust to incomplete sensory input. We demonstrated, including with a Turing test and a scalability test for pursuit behavior, that all key features of this behavior are accounted for by individuals following a simple experimentally derived proportional derivative control law, which we termed "BioPD." Because target pursuit is key to effective control of autonomous vehicles, we evaluated-as a proof of principle-the potential use of this simple evolved control law for human-engineered systems. In doing so, we found close-to-optimal pursuit performance in autonomous vehicle (terrestrial, airborne, and watercraft) pursuit while requiring limited system-specific tuning or optimization.

Localized thermal convective updrafts in the atmosphere, commonly referred to as thermals, serve as a significant source of energy-efficient flight for birds, human pilots, and autonomous aircraft. Measuring the vertical airspeed distribution within these updrafts to estimate their strength and characteristics is a significant empirical challenge. In this study, we introduce a proof-of-concept distributed thermal measurement system that uses small, cost-effective multirotor drones equipped with standard sensors, eliminating the need for specialized airspeed instruments. These drones estimate updraft strength by analyzing performance parameters such as power consumption or rotor rotation speed. First, we conducted extensive investigations in a simulated environment that incorporated varying windy conditions to establish the relationship between the updraft speed (the vertical speed of the air) and the performance parameters of the drones. Following this, we conducted outdoor experiments involving up to 49 multirotor drones to demonstrate the effectiveness of the distributed measurement system in action. By advancing our understanding of thermal updrafts, this research contributes valuable information to analyze avian flight behavior and also facilitates the development of realistic simulation environments. These advancements can enhance the design of thermal-utilizing self-driving algorithms for unmanned aerial vehicles (UAVs), paving the way for more efficient and adaptive robotic systems.

Rodents serve as an important model for examining both individual and collective behavior. Dominance within rodent social structures can determine access to critical resources, such as food and mating opportunities. Yet, many aspects of the intricate interplay between individual behaviors and the resulting group social hierarchy, especially its evolution over time, remain unexplored. In this study, we utilized an automated tracking system that continuously monitored groups of male rats for over 250 days to enable an in-depth analysis of individual behavior and the overarching group dynamic. We describe the evolution of social structures within a group and additionally investigate how past behaviors influence the emergence of new social hierarchies when group composition and experimental area changes. Notably, we find that conventional individual and pairwise tests exhibit a weak correlation with group behavior, highlighting their limited accuracy in predicting behavioral outcomes in a collective context. These results emphasize the context-dependence of social behavior as an emergent property of interactions within a group and highlight the need to measure and quantify social behavior in more naturalistic environments.

Coordinated motion in animal groups has predominantly been studied with a focus on spatial interactions, such as how individuals position and orient themselves relative to one another. Temporal aspects have, by contrast, received much less attention. Here, by studying pairwise interactions in juvenile zebrafish (Danio rerio)-including using immersive volumetric virtual reality (VR) with which we can directly test models of social interactions in situ-we reveal that there exists a rhythmic out-of-phase (i.e., an alternating) temporal coordination dynamic. We find that reciprocal (bi-directional) feedback is both necessary and sufficient to explain this emergent coupling. Beyond a mechanistic understanding, we find, both from VR experiments and analysis of freely swimming pairs, that temporal coordination considerably improves spatial responsiveness, such as to changes in the direction of motion of a partner. Our findings highlight the synergistic role of spatial and temporal coupling in facilitating effective communication between individuals on the move. It is known that spatially localized interactions can give rise to self-organized collective motion. Here, by studying pairwise interactions in juvenile zebrafish, authors reveal the role of reciprocal temporal coupling and find that temporal coordination considerably improves spatial responsiveness, such as reacting to changes in the direction of motion of a partner.

Markerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.

Collective behaviour, social interactions and leadership in animal groups are often driven by individual differences. However, most studies focus on same-species groups, in which individual variation is relatively low. Multispecies groups, however, entail interactions among highly divergent phenotypes, ranging from simple exploitative actions to complex coordinated networks. Here we studied hunting groups of otherwise-solitary Octopus cyanea and multiple fish species, to unravel hidden mechanisms of leadership and associated dynamics in functional nature and complexity, when divergence is maximized. Using three-dimensional field-based tracking and field experiments, we found that these groups exhibit complex functional dynamics and composition-dependent properties. Social influence is hierarchically distributed over multiscale dimensions representing role specializations: fish (particularly goatfish) drive environmental exploration, deciding where, while the octopus decides if, and when, the group moves. Thus, 'classical leadership' can be insufficient to describe complex heterogeneous systems, in which leadership instead can be driven by both stimulating and inhibiting movement. Furthermore, group composition altered individual investment and collective action, triggering partner control mechanisms (that is, punching) and benefits for the de facto leader, the octopus. This seemingly non-social invertebrate flexibly adapts to heterospecific actions, showing hallmarks of social competence and cognition. These findings expand our current understanding of what leadership is and what sociality is. Using three-dimensional field-based tracking and field experiments, the authors find that octopus-fish collective hunting groups exhibit complex functional dynamics and composition-dependent properties, in which different members of the group lead on different decisions.

Recent advances in machine learning and computer vision are revolutionizing the field of animal behavior by enabling researchers to track the poses and locations of freely moving animals without any marker attachment. However, large datasets of annotated images of animals for markerless pose tracking, especially high-resolution images taken from multiple angles with accurate 3D annotations, are still scant. Here, we propose a method that uses a motion capture (mo-cap) system to obtain a large amount of annotated data on animal movement and posture (2D and 3D) in a semi-automatic manner. Our method is novel in that it extracts the 3D positions of morphological keypoints (e.g eyes, beak, tail) in reference to the positions of markers attached to the animals. Using this method, we obtained, and offer here, a new dataset - 3D-POP with approximately 300k annotated frames (4 million instances) in the form of videos having groups of one to ten freely moving birds from 4 different camera views in a 3.6m x 4.2m area. 3D-POP is the first dataset of flocking birds with accurate keypoint annotations in 2D and 3D along with bounding box and individual identities and will facilitate the development of solutions for problems of 2D to 3D markerless pose, trajectory tracking, and identification in birds.

The SMART-BARN (scalable multimodal arena for real-time tracking behavior of animals in large numbers) achieves fast, robust acquisition of movement, behavior, communication, and interactions of animals in groups, within a large (14.7 meters by 6.6 meters by 3.8 meters), three-dimensional environment using multiple information channels. Behavior is measured from a wide range of taxa (insects, birds, mammals, etc.) and body size (from moths to humans) simultaneously. This system integrates multiple, concurrent measurement techniques including submillimeter precision and high-speed (300 hertz) motion capture, acoustic recording and localization, automated behavioral recognition (computer vision), and remote computer-controlled interactive units (e.g., automated feeders and animal-borne devices). The data streams are available in real time allowing highly controlled and behavior-dependent closed-loop experiments, while producing comprehensive datasets for offline analysis. The diverse capabilities of SMART-BARN are demonstrated through three challenging avian case studies, while highlighting its broad applicability to the fine-scale analysis of collective animal behavior across species.

Studying animal societies needs detailed observation of many individuals, but technological advances offer new opportunities in this field. Here, we present a state-of-the-art drone observation of a multilevel herd of Przewalski's horses, consisting of harems (one-male, multifemale groups). We track, in high spatio-temporal resolution, the movements of 238 individually identified horses on drone videos, and combine movement analyses with demographic data from two decades of population monitoring. Analysis of collective movements reveals how the structure of the herd's social network is related to kinship and familiarity of individuals. The network centrality of harems is related to their age and how long the harem stallions have kept harems previously. Harems of genetically related stallions are closer to each other in the network, and female exchange is more frequent between closer harems. High movement similarity of females from different harems predicts becoming harem mates in the future. Our results show that only a few minutes of fine-scale movement tracking combined with high throughput data driven analysis can reveal the structure of a society, reconstruct past group dynamics and predict future ones.

Many animal behaviours exhibit complex temporal dynamics, suggesting there are multiple timescales at which they should be studied. However, researchers often focus on behaviours that occur over relatively restricted temporal scales, typically ones that are more accessible to human observation. The situation becomes even more complex when considering multiple animals interacting, where behavioural coupling can introduce new timescales of importance. Here, we present a technique to study the time-varying nature of social influence in mobile animal groups across multiple temporal scales. As case studies, we analyse golden shiner fish and homing pigeons, which move in different media. By analysing pairwise interactions among individuals, we show that predictive power of the factors affecting social influence depends on the timescale of analysis. Over short timescales the relative position of a neighbour best predicts its influence and the distribution of influence across group members is relatively linear, with a small slope. At longer timescales, however, both relative position and kinematics are found to predict influence, and nonlinearity in the influence distribution increases, with a small number of individuals being disproportionately influential. Our results demonstrate that different interpretations of social influence arise from analysing behaviour at different timescales, highlighting the importance of considering its multiscale nature. This article is part of a discussion meeting issue ‘Collective behaviour through time’.

Most tracking data encompasses humans, the availability of annotated tracking data for animals is limited, especially for multiple objects. To overcome this obstacle, we present I-MuPPET, a system to estimate and track 2D keypoints of multiple pigeons at interactive speed. We train a Keypoint R-CNN on single pigeons in a fully supervised manner and infer keypoints and bounding boxes of multiple pigeons with that neural network. We use a state of the art tracker to track the individual pigeons in video sequences. I-MuPPET is tested quantitatively on single pigeon motion capture data, and we achieve comparable accuracy to state of the art 2D animal pose estimation methods in terms of Root Mean Square Error (RMSE). Additionally, we test I-MuPPET to estimate and track poses of multiple pigeons in video sequences with up to four pigeons and obtain stable and accurate results with up to 17 fps. To establish a baseline for future research, we perform a detailed quantitative tracking evaluation, which yields encouraging results.

Abstract Using a large-scale motion-capture system and custom head-calibration methods, we reconstructed the head-centric view of freely behaving pigeons and examined how they orient their head when presented with various types of attention-getting objects at various relative locations. Pigeons predominantly employed their retinal specializations to view a visual target, namely their foveas projecting laterally (at an azimuth of ±75°) into the horizon, and their visually-sensitive “red areas” projecting broadly into the lower-frontal visual field. Pigeons used their foveas to view any distant object while they used their red areas to view a nearby food-related object on the ground (< 50 cm). Pigeons “fixated” a visual target with their foveas; the intervals between head saccades were longer when the visual target was viewed by birds’ foveas compared to when it was viewed by any other visual field region. Furthermore, pigeons used their right eye to observe food-related objects and their left eye to do so for threat-related or social stimuli. Despite the known difficulty in identifying where a bird is attending, we show that it is possible to estimate the visual attention of freely behaving individuals by tracking the projections of their retinal specializations in their visual field with cutting-edge methods.

Using a motion-capture system and custom head-calibration methods, we reconstructed the head-centric view of freely behaving pigeons and examined how they orient their head when presented with various types of attention-getting objects at various relative locations. Pigeons predominantly employed their retinal specializations to view a visual target, namely their foveas projecting laterally (at an azimuth of ± 75°) into the horizon, and their visually-sensitive "red areas" projecting broadly into the lower-frontal visual field. Pigeons used their foveas to view any distant object while they used their red areas to view a nearby object on the ground (< 50 cm). Pigeons "fixated" a visual target with their foveas; the intervals between head-saccades were longer when the visual target was viewed by birds' foveas compared to when it was viewed by any other region. Furthermore, pigeons showed a weak preference to use their right eye to examine small objects distinctive in detailed features and their left eye to view threat-related or social stimuli. Despite the known difficulty in identifying where a bird is attending, we show that it is possible to estimate the visual attention of freely-behaving birds by tracking the projections of their retinal specializations in their visual field with cutting-edge methods.

Choosing among spatially distributed options is a central challenge for animals, from deciding among alternative potential food sources or refuges to choosing with whom to associate. Using an integrated theoretical and experimental approach (employing immersive virtual reality), we consider the interplay between movement and vectorial integration during decision-making regarding two, or more, options in space. In computational models of this process, we reveal the occurrence of spontaneous and abrupt "critical" transitions (associated with specific geometrical relationships) whereby organisms spontaneously switch from averaging vectorial information among, to suddenly excluding one among, the remaining options. This bifurcation process repeats until only one option-the one ultimately selected-remains. Thus, we predict that the brain repeatedly breaks multichoice decisions into a series of binary decisions in space-time. Experiments with fruit flies, desert locusts, and larval zebrafish reveal that they exhibit these same bifurcations, demonstrating that across taxa and ecological contexts, there exist fundamental geometric principles that are essential to explain how, and why, animals move the way they do.