We propose a framework for detecting action patterns from motion sequences and modeling the sensory-motor relationship of animals, using a generative recurrent neural network. The network has a discriminative part (classifying actions) and a generative part (predicting motion), whose recurrent cells are laterally connected, allowing higher levels of the network to represent high level behavioral phenomena. We test our framework on two types of tracking data, fruit fly behavior and online handwriting. Our results show that 1) taking advantage of unlabeled sequences, by predicting future motion, significantly improves action detection performance when training labels are scarce, 2) the network learns to represent high level phenomena such as writer identity and fly gender, without supervision, and 3) simulated motion trajectories, generated by treating motion prediction as input to the network, look realistic and may be used to qualitatively evaluate whether the model has learnt generative control rules.

We propose a framework for detecting action patterns from motion sequences and modeling the sensory-motor relationship of animals, using a generative recurrent neural network. The network has a discriminative part (classifying actions) and a generative part (predicting motion), whose recurrent cells are laterally connected, allowing higher levels of the network to represent high level phenomena. We test our framework on two types of data, fruit fly behavior and online handwriting. Our results show that 1) taking advantage of unlabeled sequences, by predicting future motion, significantly improves action detection performance when training labels are scarce, 2) the network learns to represent high level phenomena such as writer identity and fly gender, without supervision, and 3) simulated motion trajectories, generated by treating motion prediction as input to the network, look realistic and may be used to qualitatively evaluate whether the model has learnt generative control rules.

We describe a system that tracks pairs of fruit flies and automatically detects and classifies their actions. We compare experimentally the value of a frame-level feature representation with the more elaborate notion of ‘bout features’ that capture the structure within actions. Similarly, we compare a simple sliding window classifier architecture with a more sophisticated structured output architecture, and find that window based detectors outperform the much slower structured counterparts, and approach human performance. In addition we test our top performing detector on the CRIM13 mouse dataset, finding that it matches the performance of the best published method. Our Fly-vs-Fly dataset contains 22 hours of video showing pairs of fruit flies engaging in 10 social interactions in three different contexts; it is fully annotated by experts, and published with articulated pose trajectory features.

How animals use sensory information to weigh the risks vs. benefits of behavioral decisions remains poorly understood. Inter-male aggression is triggered when animals perceive both the presence of an appetitive resource, such as food or females, and of competing conspecific males. How such signals are detected and integrated to control the decision to fight is not clear. For instance, it is unclear whether food increases aggression directly, or as a secondary consequence of increased social interactions caused by attraction to food. Here we use the vinegar fly, Drosophila melanogaster, to investigate the manner by which food influences aggression. We show that food promotes aggression in flies, and that it does so independently of any effect on frequency of contact between males, increase in locomotor activity or general enhancement of social interactions. Importantly, the level of aggression depends on the absolute amount of food, rather than on its surface area or concentration. When food resources exceed a certain level, aggression is diminished, suggestive of reduced competition. Finally, we show that detection of sugar via Gr5a+ gustatory receptor neurons (GRNs) is necessary for food-promoted aggression. These data demonstrate that food exerts a specific effect to promote aggression in male flies, and that this effect is mediated, at least in part, by sweet-sensing GRNs.

Males of most species are more aggressive than females, but the neural mechanisms underlying this dimorphism are not clear. Here, we identify a neuron and a gene that control the higher level of aggression characteristic of Drosophila melanogaster males. Males, but not females, contain a small cluster of FruM(+) neurons that express the neuropeptide tachykinin (Tk). Activation and silencing of these neurons increased and decreased, respectively, intermale aggression without affecting male-female courtship behavior. Mutations in both Tk and a candidate receptor, Takr86C, suppressed the effect of neuronal activation, whereas overexpression of Tk potentiated it. Tk neuron activation overcame reduced aggressiveness caused by eliminating a variety of sensory or contextual cues, suggesting that it promotes aggressive arousal or motivation. Tachykinin/Substance P has been implicated in aggression in mammals, including humans. Thus, the higher aggressiveness of Drosophila males reflects the sexually dimorphic expression of a neuropeptide that controls agonistic behaviors across phylogeny.

Automatically classifying behavior of humans and animals from video is one of the most interesting and challenging fields of computer vision, [3, 1, 6]. Most of the successful human behavior recognition works use as features for classification information extracted from a direct representation of the scene (visual features), as opposed to indirect representations such as silhouettes, body parts, pose or object positions, which can be very sensitive to viewpoint variation and occlusions in real-world videos [10]. In contrast, indirect representation of the scene is widely used in the case of animals [1, 6, 4]. Animal enclosures allow for a more controlled filming, which reduces viewpoint variations and occlusions, facilitating the extraction of indirect representations of objects in the scene. Moreover, animal bodies are less expressive than humans, therefore causing direct visual features to work worse on animals [2]. One of the most widely used features for animal behavior recognition is the position of animals in time (result of either manual annotations or an object detection+tracking algorithm) [1, 6, 4, 2]. From the positions, usually several trajectory features are computed, such as distance between animals, their direction of movement, velocity, acceleration, etc. These trajectory features are used, together with the behavior labels, to train a supervised classifier that learns the discriminative features across behaviors. In scenarios where videos are previously segmented 1, such as in KTH [11] or Hollywood2 [8] human datasets, a classic supervised classifier such as SVM [14] or AdaBoost [5] is often used. In more realistic scenarios, however, the task is to fully segment a continuous video into behavior intervals (behavior category, starting frame, ending frame). Most works on animal behavior recognition fall into this category [1, 6, 4, 2], while recent effort has also been made in the human action recognition community to move in this direction, e.g. Virat dataset [9]. In this scenario, more intricate classifiers are needed. In [6] authors use a two layer SVMHMM, while in [2] authors extend Auto-context [13] to video. These classifiers are able to detect behavior classes and at the same time learn behavior transitions, successfully segmenting long, continuous videos into smooth behavior intervals. In this work, we describe a novel extension of Structured SVM and benchmark it against the Auto-context method to measure its robustness and versatility. The rest of the paper is as follows: Section 2 briefly presents the two methods to be compared, Section 3 presents results on two different datasets (mice and flies) and discusses the results.

We present a multisensory method for estimating the transformation of a mobile phone between two images taken from its camera. Pose estimation is a necessary step for applications such as 3D reconstruction and panorama construction, but detecting and matching robust features can be computationally expensive. In this paper we propose a method for combining the inertial sensors (accelerometers and gyroscopes) of a mobile phone with its camera to provide a fast and accurate pose estimation. We use the inertial based pose to warp two images into the same perspective frame. We then employ an adaptive FAST feature detector and image patches, normalized with respect to illumination, as feature descriptors. After the warping the images are approximately aligned with each other so the search for matching key-points also becomes faster and in certain cases more reliable. Our results show that by incorporating the inertial sensors we can considerably speed up the process of detecting and matching key-points between two images, which is the most time consuming step of the pose estimation.

ABSTRACT Digital logic circuit design is an inherently computable process that can greatly benefit from real-time feedback and evaluation. This paper presents CircuitBoard, an application for designing and testing hand drawn,digital logic circuits using a sketch-based interface, such as on a Tablet PC. Our system aims to provide these tools to a paper-like interface, an environment,shown,to be natural and,conducive,to design,conceptualization. We,outline,our,low-level strategies for stroke segmentation,and,logic element recognition, as well as our high-level approaches to circuit evaluation and the motivations,behind our system's user interface design. Author Keywords Sketch recognition, circuit design, user interfaces. ACM Classification Keywords