Abstract Zebrafish (Danio rerio) has emerged as a valuable vertebrate model organism for studies of social interactions. While previous multiple-animal research has focused on gross movement, here we present a machine vision workflow to capture and analyze fine-grained social interactions through the tracking of three anatomical landmarks (at the head, pectoral fins, and tail) as well as the identity of the fish in 3D. We release a dataset of N = 173 five-hour recordings of adult zebrafish dyads, including male/male and female/female wild-type pairs and disease-model mutants, sampling complex behaviors such as dominance contests and aggressive/submissive motifs. The recordings are of high temporal resolution (f s = 140 Hz) from a large imaging volume ~ 10 body lengths per linear dimension, and include both square and cylindrical arenas. This dataset offers a critical resource for biologists seeking to understand the neural basis of social behavior, for machine learning researchers working to improve posture tracking, and for the broader quantitative understanding of natural behavior.

Across biological subdisciplines, the last decade has seen an explosion of high-dimensional datasets. At the ICTS workshop ‘Unifying Theories in High-Dimensional Biophysics’, we discussed whether this high dimensionality poses a challenge or an opportunity for theoretically describing, understanding and predicting biological systems. We discussed methods, models and frameworks that can be used for this purpose. This Comment summarizes our discussions from the perspectives of individual participants.

Pose estimation methods and motion capture systems have opened doors to quantitative measurements of animal kinematics. While animal behavior experiments are expensive and complex, tracking errors sometimes make large portions of the experimental data unusable. Here our deep learning method, Deep Imputation for Skeleton data (DISK), uncovers dependencies between keypoints and their dynamics to impute missing tracking data without the help of any manual annotations. We demonstrate the utility and performance of DISK on seven animal skeletons including multi-animal setups. The imputed recordings allow us to detect more episodes of motion, such as steps, and obtain more statistically robust results when comparing these episodes between experimental conditions. In addition, by learning to impute the missing content, DISK learns meaningful representations of the data capturing, for example, underlying actions. This stand-alone imputation package, available at https://github.com/bozeklab/DISK.git/ , is applicable to outputs of tracking methods (marker-based or markerless) and allows for varied types of downstream analysis.

Social behaviors include some of the most interesting interactions in living systems yet their principled characterization remains unsolved. Here we suggest that at the core of social interactions is the notion of mutual prediction, which we analyze in the context of two male zebrafish engaged in a dominance contest. Using 3D velocity trajectories, we construct the mutual information between a two-animal past and one-animal future, and we quantify the redundant, unique, and synergistic components using partial information decomposition across time windows. We find that predictive information decomposition naturally aligns with important social concepts, such as mirroring and dominance. At contest end, we find asymmetries in self-unique and redundant information that reflect the emergent dominance relationship. Applied to mecp2 zebrafish mutants, an autism model, we find that predictive information is reduced overall, but especially for synergistic flows, which is indicative of difficulties in more complex social dynamics. Significance Statement Social interactions are rich and diverse, ranging from mirroring to complementary actions. A unifying framework for defining and analyzing such interaction types has long been needed. Here, based on modern information theory, we formulate how the past state of interacting organisms encodes the future state of an individual. This framework provides a natural decomposition of pairwise social dynamics into mirroring, independent action, directed influence, and joint action. Applied to dominance contests in zebrafish, these modes of interaction capture distinct phases of conflict, their assessment strategies, and the resulting dominance relationships. Moreover, our analysis reveals a specific disruption in the social behavior of mutant zebrafish linked to autism, shedding new light on impairments in communication and social learning. ### Competing Interest Statement The authors have declared no competing interest. OIST Graduate University

While two-body fighting behavior occurs throughout the animal kingdom to settle dominance disputes, important questions such as how the dynamics ultimately lead to a winner and loser are unresolved. Here we examine fighting behavior at high resolution in male zebrafish. We combine multiple cameras, a large volume containing a transparent interior cage to avoid reflection artifacts, with computer vision to track multiple body points across multiple organisms while maintaining individual identity in three dimensions. In the body point trajectories we find a spectrum of timescales which we use to build informative joint coordinates consisting of relative orientation and distance. We use the distribution of these coordinates to automatically identify fight epochs, and we demonstrate the postfight emergence of an abrupt asymmetry in relative orientations—a clear and quantitative signal of hierarchy formation. We identify short-time, multi-animal behaviors as clustered transitions between joint configurations, and show that fight epochs are spanned by a subset of these clusters, which we denote as maneuvers. The resulting space of maneuvers is rich but interpretable, including motifs such as “attacks” and “circling.” In the longer-time dynamics of maneuver frequencies we find differential and changing strategies, including that the eventual loser attacks more often towards the end of the contest. Our results suggest a reevaluation of relevant assessment models in zebrafish, while our approach is generally applicable to other animal systems. Published by the American Physical Society 2024

Animal locomotion is often subject to constraints arising from anatomical/physiological asymmetries. We use the nematode C.~elegans as a minimal model system to ask whether such constraints might shape locomotion patterns optimized during evolution. We focus on turning behaviours in two contrasting environmental contexts: (i) random exploration in the absence of strong stimuli and (ii) acute avoidance (escape) navigation upon encountering a strong aversive stimulus. We characterise the full repertoire of reorientation behaviours, including gradual reorientations and various posturally distinct sharp turns. During exploration, our measurements and theoretical modelling indicate that orientation fluctuations on short timescales are, on average, optimized to compensate the constraining gradual turn bias on long timescales. During escape, our data suggests that the reorientation is controlled not by an analog logic of continuous turn-amplitude modulations, but rather through the digital logic of selecting discrete turn types, leading to a symmetric escape performance despite strongly asymmetric turning biases.

How do we capture the breadth of behavior in animal movement, from rapid body twitches to aging? Using high-resolution videos of the nematode worm Caenorhabditis elegans, we show that a single dynamics connects posture-scale fluctuations with trajectory diffusion and longer-lived behavioral states. We take short posture sequences as an instantaneous behavioral measure, fixing the sequence length for maximal prediction. Within the space of posture sequences, we construct a fine-scale, maximum entropy partition so that transitions among microstates define a high-fidelity Markov model, which we also use as a means of principled coarse-graining. We translate these dynamics into movement using resistive force theory, capturing the statistical properties of foraging trajectories. Predictive across scales, we leverage the longest-lived eigenvectors of the inferred Markov chain to perform a top-down subdivision of the worm's foraging behavior, revealing both "runs-and-pirouettes" as well as previously uncharacterized finer-scale behaviors. We use our model to investigate the relevance of these fine-scale behaviors for foraging success, recovering a trade-off between local and global search strategies.

Isolating slower dynamics from fast fluctuations has proven remarkably powerful, but how do we proceed from partial observations of dynamical systems for which we lack underlying equations? Here, we construct maximally predictive states by concatenating measurements in time, partitioning the resulting sequences using maximum entropy, and choosing the sequence length to maximize short-time predictive information. Transitions between these states yield a simple approximation of the transfer operator, which we use to reveal timescale separation and long-lived collective modes through the operator spectrum. Applicable to both deterministic and stochastic processes, we illustrate our approach through partial observations of the Lorenz system and the stochastic dynamics of a particle in a double-well potential. We use our transfer operator approach to provide a new estimator of the Kolmogorov-Sinai entropy, which we demonstrate in discrete and continuous-time systems, as well as the movement behavior of the nematode worm C. elegans.

While two-body fighting behavior occurs throughout the animal kingdom to settle dominance disputes, important questions such as how the dynamics ultimately lead to a winner and loser are unresolved. Here we examine fighting behavior at high-resolution in male zebrafish. We combine multiple cameras, a large volume containing a transparent interior cage to avoid reflection artifacts, with computer vision to track multiple body points across multiple organisms while maintaining individual identity in 3D. In the body point trajectories we find a spectrum of timescales which we use to build informative joint coordinates consisting of relative orientation and distance. We use the distribution of these coordinates to automatically identify fight epochs, and we demonstrate the post-fight emergence of an abrupt asymmetry in relative orientations-a clear and quantitative signal of hierarchy formation. We identify short-time, multi-animal behaviors as clustered transitions between joint configurations, and show that fight epochs are spanned by a subset of these clusters, which we denote as maneuvers. The resulting space of maneuvers is rich but interpretable, including motifs such as “attacks” and “circling”. In the longer-time dynamics of maneuver frequencies we find differential and changing strategies, including that the eventual loser attacks more often towards the end of the contest. Our results suggest a reevaluation of relevant assessment models in zebrafish, while our approach is generally applicable to other animal systems.

How do we capture the breadth of behavior in animal movement, from rapid body twitches to aging? Using high-resolution videos of the nematode worm C. elegans , we show that a single dynamics connects posture-scale fluctuations with trajectory diffusion, and longer-lived behavioral states. We take short posture sequences as an instantaneous behavioral measure, fixing the sequence length for maximal prediction. Within the space of posture sequences we construct a fine-scale, maximum entropy partition so that transitions among microstates define a high-fidelity Markov model, which we also use as a means of principled coarse-graining. We translate these dynamics into movement using resistive force theory, capturing the statistical properties of foraging trajectories. Predictive across scales, we leverage the longest-lived eigenvectors of the inferred Markov chain to perform a top-down subdivision of the worm’s foraging behavior, revealing both “runs-and-pirouettes” as well as previously uncharacterized finer-scale behaviors. We use our model to investigate the relevance of these fine-scale behaviors for foraging success, recovering a trade-off between local and global search strategies. SIGNIFICANCE STATEMENT Complex phenotypes, such as an animal’s behavior, generally depend on an overwhelming number of processes that span a vast range of scales. While there is no reason that behavioral dynamics permit simple models, by subsuming inherent nonlinearities and memory into maximally-predictive microstates, we find one for C. elegans foraging. The resulting “Markov worm” is effectively indistinguishable from real worm motion across a range of timescales, and we can decompose our model dynamics both to recover and discover behavioral states. Employing a simple form of substrate interactions, we connect postures to trajectories, illuminating how worms explore the environment. In more complex organisms, our approach can also link behaviors across time, from rapid muscular control to neuromodulation.

While stress reactions can emerge long after the triggering event, it remains elusive how they emerge after a protracted, seemingly stress-free period during which stress incubates. Here, we study the behavioral development in mice isolated after observing an aggressive encounter inflicted upon their pair-housed partners. We developed a spatially resolved fine-scale behavioral analysis and applied it to standard behavioral tests. It reveals that the seemingly sudden behavioral changes developed gradually. These behavioral changes were not observed if the aggressive encounter happened to a stranger mouse, suggesting that social bonding is a prerequisite for stress incubation in this paradigm. This finding was corroborated by hemisphere-specific morphological changes in cortex regions centering at the anterior cingulate cortex, a cognitive and emotional center. Our non-invasive analytical methods to capture informative behavioral details may have applications beyond laboratory animals.

Datasets to reproduce the results from Costa et al. (2022) "Maximally predictive states: from partial observations to long timescales"

From cells in tissue, to bird flocks, to human crowds, living systems display a stunning variety of collective behaviors. Yet quantifying such phenomena first requires tracking a significant fraction of the group members in natural conditions, a substantial and ongoing challenge. We present a comprehensive, computational method for tracking an entire colony of the honey bee Apis mellifera using high-resolution video on a natural honeycomb background. We adapt a convolutional neural network (CNN) segmentation architecture to automatically identify bee and brood cell positions, body orientations and within-cell states. We achieve high accuracy (~10% body width error in position, ~10° error in orientation, and true positive rate > 90%) and demonstrate months-long monitoring of sociometric colony fluctuations. These fluctuations include ~24 h cycles in the counted detections, negative correlation between bee and brood, and nightly enhancement of bees inside comb cells. We combine detected positions with visual features of organism-centered images to track individuals over time and through challenging occluding events, recovering ~79% of bee trajectories from five observation hives over 5 min timespans. The trajectories reveal important individual behaviors, including waggle dances and crawling inside comb cells. Our results provide opportunities for the quantitative study of collective bee behavior and for advancing tracking techniques of crowded systems.

We present a comprehensive, computational method for tracking an entire colony of the honey bee Apis mellifera using high-resolution video on a natural honeycomb background. We adapt a convolutional neural network (CNN) segmentation architecture to automatically identify bee and brood cell positions, body orientations and within-cell states. We achieve high accuracy (~10% body width error in position, ~10° error in orientation, and true positive rate > 90%) and demonstrate months-long monitoring of sociometric colony fluctuations. We combine extracted positions with rich visual features of organism-centered images to track individuals over time and through challenging occluding events, recovering ~79% of bee trajectories from five observation hives over a span of 5 minutes. The resulting trajectories reveal important behaviors, including fast motion, comb-cell activity, and waggle dances. Our results provide new opportunities for the quantitative study of collective bee behavior and for advancing tracking techniques of crowded systems.

An important model system for understanding genes, neurons and behavior, the nematode worm C. elegans naturally moves through a variety of complex postures, for which estimation from video data is challenging. We introduce an open-source Python package, WormPose, for 2D pose estimation in C. elegans, including self-occluded, coiled shapes. We leverage advances in machine vision afforded from convolutional neural networks and introduce a synthetic yet realistic generative model for images of worm posture, thus avoiding the need for human-labeled training. WormPose is effective and adaptable for imaging conditions across worm tracking efforts. We quantify pose estimation using synthetic data as well as N2 and mutant worms in on-food conditions. We further demonstrate WormPose by analyzing long (∼ 8 hour), fast-sampled (∼ 30 Hz) recordings of on-food N2 worms to provide a posture-scale analysis of roaming/dwelling behaviors.

Honeybee swarms are a landmark example of collective behavior. To become a coherent swarm, bees locate their queen by tracking her pheromones. But how can distant individuals exploit these chemical signals, which decay rapidly in space and time? Here, we combine a behavioral assay with the machine vision detection of organism location and scenting (pheromone propagation via wing fanning) behavior to track the search and aggregation dynamics of the honeybee Apis mellifera L. We find that bees collectively create a scenting-mediated communication network by arranging in a specific spatial distribution where there is a characteristic distance between individuals and directional signaling away from the queen. To better understand such a flow-mediated directional communication strategy, we developed an agent-based model where bee agents obeying simple, local behavioral rules exist in a flow environment in which the chemical signals diffuse and decay. Our model serves as a guide to exploring how physical parameters affect the collective scenting behavior and shows that increased directional bias in scenting leads to a more efficient aggregation process that avoids local equilibrium configurations of isotropic (nondirectional and axisymmetric) communication, such as small bee clusters that persist throughout the simulation. Our results highlight an example of extended classical stigmergy: Rather than depositing static information in the environment, individual bees locally sense and globally manipulate the physical fields of chemical concentration and airflow.

Time-lapse microscopy is routinely used to follow cells within organoids, allowing direct study of division and differentiation patterns. There is an increasing interest in cell tracking in organoids, which makes it possible to study their growth and homeostasis at the single-cell level. As tracking these cells by hand is prohibitively time consuming, automation using a computer program is required. Unfortunately, organoids have a high cell density and fast cell movement, which makes automated cell tracking difficult. In this work, a semi-automated cell tracker has been developed. To detect the nuclei, we use a machine learning approach based on a convolutional neural network. To form cell trajectories, we link detections at different time points together using a min-cost flow solver. The tracker raises warnings for situations with likely errors. Rapid changes in nucleus volume and position are reported for manual review, as well as cases where nuclei divide, appear and disappear. When the warning system is adjusted such that virtually error-free lineage trees can be obtained, still less than 2% of all detected nuclei positions are marked for manual analysis. This provides an enormous speed boost over manual cell tracking, while still providing tracking data of the same quality as manual tracking.