Motion segmentation and motion estimation are important topics in computer vision. Tensor Voting is a process that addresses both issues simultaneously; but running time is a challenge. We propose a novel approach which can yield both the motion segmentation and the motion estimation in the presence of discontinuities. This method is a combination of a non-iterative boosted-speed voting process in sparse space in a first stage, and a Graph-Cuts framework for boundary refinement in a second stage. Here, we concentrate on the motion segmentation problem. After initially choosing a sparse space by sampling the original image, we represent each of these pixels as 4-D tensor points and apply the voting framework to enforce local smoothness of motion. Afterwards, the boundary refinement is obtained by using the Graph-Cuts image segmentation. Our results attained in different types of motion show that the method outperforms other Tensor Voting approaches in speed, and the results are comparable with other methodologies in motion segmentation.
We present a robust framework for tracking people in crowded outdoor environments monitored by multiple cameras with a goal of real-time performance. Since no single algorithm is perfect for the task of object tracking in all cases, we instead take an alternate approach. Our algorithm dynamically switches between several available trackers on-the-fly by evaluating the current state/context of the scene. Autonomous agents that make the switching decisions are assigned to each object in the scene. Initialization of new agents and the handoff between various tracking algorithms are completely automated. The collaboration between different trackers is shown to improve performance compared to the individual methods in terms of both computation and reliability. The tracker switching framework is evaluated on a multi-camera dataset and both qualitative and quantitative results are presented.
This paper describes a technique to estimate human face pose from color video sequence using dynamic Bayesian network(DBN). As face and facial features trackers usually track eyes, pupils, mouth corners and skin region(face), our proposed method utilizes merely three of these features - pupils, mouth center and skin region - to compute the evidence for DBN inference. No additional image processing algorithm is required, thus, it is simple and operates in real-time. The evidence, which are called horizontal ratio and vertical ratio in this paper, are determined using model-based technique and designed significantly to simultaneously solve two problems in tracking task; scaling factor and noise influence. Results reveal that the proposed method can be realized in real-time on a 2.2 GHz Celeron CPU machine with very satisfactory pose estimation results.
Surveillance and tracking systems often observe the same scene over extended time periods. When object motion is constrained by the scene (for instance, cars on roads, or pedestrians on sidewalks), it is advantageous to characterize and use scene-specific and location-specific priors to aid the tracking algorithm. This paper develops and demonstrates a method for creating priors for tracking that are conditioned on the current location of the object in the scene. These priors can be naturally incorporated in a number of tracking algorithms to make tracking more efficient and more accurate. We present a novel method to sample from these priors and show performance improvements (in both efficiency and accuracy) for two different tracking algorithms in two different problem domains.
In the literature, stereo matching is used for building pixel correspondences for stereo image pairs. Such correspondences can serve as fundamentals for applications such as 3D scene reconstruction. In some applications, however, stereo vision is adopted for object localization so that only object correspondences are required. However, existing pixel based stereo matching approaches are computationally inefficient for these applications. In this paper, we address the problem of object correspondence construction in stereo camera systems by using a fast and accurate algorithm adopting reverse stereo triangulation. This algorithm is based on a belief that any incorrect object pair will demonstrate inconsistency in its spatial location calculated from reverse stereo triangulation, so that correct object pairs can be identified accurately from all possible object pairs. We present experimental results from a dual camera human face capturing system in which more than 99% genuine object correspondences can be accurately identified, while 100% of falsely detected objects are eliminated. Besides, our proposed method can handle no less than 100 object pairs within 1 ms in a P4 1.5 GHz desktop PC.
We propose a method for enhancing the stability of tracking people by incorporating long-term observations of human actions in a scene. Basic human actions, such as walking or standing still, are frequently observed at particular locations in an observation scene. By observing human actions for a long period of time, we can identify regions that are more likely to be occupied by a person. These regions have a high probability of a person existing compared with others. The key idea of our approach is to incorporate this probability as a bias in generating samples under the framework of a particle filter for tracking people. We call this bias the environmental existence map (EEM). The EEM is iteratively updated at every frame by using the tracking results from our tracker, which leads to more stable tracking of people. Our experimental results demonstrate the effectiveness of our method.
We present a top-down approach to simultaneously track and recognize articulated full-body human motion using learned action models that is robust to variations in style, lighting, background,occlusion and viewpoint. To this end, we introduce the hierarchical variable transition hidden Markov model (HVT-HMM) that is a three-layered extension of the variable transition hidden Markov model (VTHMM). The top-most layer of the HVT-HMM represents the composite actions and contains a single Markov chain, the middle layer represents the primitive actions which are modeled using a VTHMM whose state transition probability varies with time and the bottom-most layer represents the body pose transitions using a HMM. We represent the pose using a 23D body model and present efficient learning and decoding algorithms for HVT-HMM. Further, in classical Viterbi decoding the entire sequence must be seen before the state at any instant can be recognized and hence can potentially have large latency for long video sequences. In order to address this we use a variable window approach to decoding with very low latency. We demonstrate our methods first in a domain for recognizing two-handed gestures and then in a domain with actions involving articulated motion of the entire body. Our approach shows 90-100% action recognition in both domains and runs at real-time (ap 30 fps) with very low average latency (ap 2 frames).
Spatial-temporal local motion features have shown promising results in complex human action classification. Most of the previous works [6],[16],[21] treat these spatial- temporal features as a bag of video words, omitting any long range, global information in either the spatial or temporal domain. Other ways of learning temporal signature of motion tend to impose a fixed trajectory of the features or parts of human body returned by tracking algorithms. This leaves little flexibility for the algorithm to learn the optimal temporal pattern describing these motions. In this paper, we propose the usage of spatial-temporal correlograms to encode flexible long range temporal information into the spatial-temporal motion features. This results into a much richer description of human actions. We then apply an unsupervised generative model to learn different classes of human actions from these ST-correlograms. KTH dataset, one of the most challenging and popular human action dataset, is used for experimental evaluation. Our algorithm achieves the highest classification accuracy reported for this dataset under an unsupervised learning scheme.
This paper presents a detection based object tracking method that forms object trajectories by associating detection responses. Discriminative classifiers of objects of a known class are learned and applied to the video sequence frame by frame. The output of the detection module is a "soft decision", which consists of a set of detection responses of different confidence levels. Responses of different confidence levels are generated by classifiers with different complexities. The cheap classifiers are applied to the whole image first, while the expensive classifiers are only applied to the region accepted as object by the cheap classifiers. Object trajectories are initialized from the responses of higher confidence; hypothesized objects are tracked by associating with all the responses in the order of their confidence levels. The proposed approach is applied to the problems of human tracking in indoor meeting videos and outdoor surveillance videos. The system is evaluated on two public video corpora and compared with some previous methods.
Recent research has advocated the use of a covariance matrix of image features for tracking objects instead of the conventional histogram object representation models used in popular algorithms. In this paper we extend the covariance tracker and propose efficient algorithms with an emphasis on both improving the tracking accuracy and reducing the execution time. The algorithms are compared to a baseline covariance tracker and the popular histogram-based mean shift tracker. Quantitative evaluations on a publicly available dataset demonstrate the efficacy of the presented methods. Our algorithms obtain significant speedups factors up to 330 while reducing the tracking errors by 86-90% relative to the baseline approach.
Recent works in action recognition have begun to treat actions as space-time volumes. This allows actions to be converted into 3-D shapes, thus converting the problem into that of volumetric matching. However, the special nature of the temporal dimension and the lack of intuitive volumetric features makes the problem both challenging and interesting. In a data-driven and bottom-up approach, we propose a dictionary of mid-level features called Space- Time Shapelets. This dictionary tries to characterize the space of local space-time shapes, or equivalently local motion patterns formed by the actions. Representing an action as a bag of these space-time patterns allows us to reduce the combinatorial space of these volumes, become robust to partial occlusions and errors in extracting spatial support. The proposed method is computationally efficient and achieves competitive results on a standard dataset.
Tracking and recognition of objects in video sequences suffer from difficulties in learning appropriate object models. Often a high degree of supervision is required, including manual annotation of many training images. We aim at unsupervised learning of object models and present a novel way to build models based on motion information extracted from video sequences. We require a coarse delineation of moving objects and subsequent segmentation of these motion areas into regions as preprocessing steps and analyze the resulting regions with respect to their stable detection over many frames. These 'temporally stable regions' are then used to build graphs of reliably detected object parts which form our model. Our approach combines the feature- based analysis of feature vectors for each region with the structural analysis of the graphical object models. Our experiments demonstrate the capabilities of this novel method to build object models for people and to robustly track them, but the method is in general applicable to learn object models for any object category, provided that the object moves and is observed by a stationary camera.
We present a novel method for detecting unusual modes of behavior in video surveillance data, suitable for supporting home-based care of elderly patients. Our approach is based on detecting unusual patterns of inactivity. We first learn a spatial map of normal inactivity for an observed scene, expressed as a two-dimensional mixture of Gaussians. The map components are used to construct a Hidden Markov Model representing normal patterns of behavior. A threshold model is also inferred, and unusual behavior detected by comparing the model likelihoods. Our learning procedures are unsupervised, and yield a highly transparent model of scene activity. We present an evaluation of our approach, and show that it is effective in detecting unusual behavior across a range of parameter settings.
Segmentation of video sequences requires the segmentations of consecutive frames to be consistent with each other. We propose to use a three dimensional Conditional Random Fields (CRF) to address this problem. A triple of consecutive image frames are treated as a small 3D volume to be segmented. Our spatial-temporal CRF model combines both local discriminative features and the conditional homogeneity of labeling variables in both the spatial and the temporal domain. After training the model parameters with a small set of training data, the optimal labeling is obtained through a probabilistic inference by Sum-product loopy belief propagation. We achieve accurate segmentation results on the standard video sequences, which demonstrates the promising capability of the proposed approach.
Reconstructing a 3D scene from a moving camera is one of the most important issues in the field of computer vision. In this scenario, not all points are known in all images (e.g. due to occlusion), thus generating missing data. The state of the art handles the missing points in this context by enforcing rank constraints on the point track matrix. However, quite frequently, close up views tend to capture planar surfaces producing degenerate data. If one single frame is degenerate, the whole sequence will produce high errors on the shape reconstruction, even though the observation matrix verifies the rank 4 constraint. In this paper, we propose to solve the structure from motion problem with degenerate data, introducing a new factorization algorithm that imposes the full scaled orthographic model in one single optimization procedure. By imposing all model constraints, a unique (correct) 3D shape is estimated regardless of the data degeneracies. Experiments show that remarkably good reconstructions are obtained with an approximate models such as orthography.
Due to increased interest in visual surveillance, various multiple object tracking methods have been recently proposed and applied to pedestrian tracking. However in presence of intensive inter-object occlusion and sensor gaps, most of these methods result in tracking failures. We present a two-stage multi-object tracking approach to robustly track pedestrians in such complex scenarios. We first generate high confidence partial track segments (tracklets) using a robust pedestrian detector and then associate the tracklets in a global optimization framework. Unlike the existing two-stage tracking methods, our method uses the unasso- ciated low confidence detections (residuals) between the tracklets, which improves the tracking performance. We evaluate our method on the CAVIAR dataset and show that our method performs better than state-of-the-art methods.
We present a novel approach to learning motion behavior in video, and detecting abnormal behavior, using hierarchical clustering of hidden Markov models (HMMs). A continuous stream of track data is used for online and on-demand creation and training of HMMs, where tracks may be of highly variable length and scenes may be very complex with an unknown number of motion patterns. We show how these HMMs can be used for on-line clustering of tracks that represent normal behavior and for detection of deviant tracks. The track clustering algorithm uses a hierarchical agglomerative HMM clustering technique that jointly determines all the HMM parameters (including the number of states) via an expectation maximization (EM) algorithm and the Akaike information criteria. Results are demonstrated on a highly complex scene containing dozens of routes, significant occlusions and hundreds of moving objects.
The paper describes a methodology for the recognition of high-level group activities. Our system recognizes group activities including group actions, group-persons interactions, group-group (i.e. inter-group) interactions, intra-group interactions, and their combinations described using a common representation scheme. Our approach is to represent various types of complex group activities with a programming language-like representation, and then to recognize represented activities based on the recognition of activities of individual group members. A hierarchical recognition algorithm is designed for the recognition of high-level group activities. The system was tested to recognize activities such as 'two groups fighting', 'a group of thieves stealing an object from another group', and 'a group of policemen arresting a group of criminals (or a criminal)'. Videos downloaded from YouTube as well as videos that we have taken are tested. Experimental results shows that our system recognizes complicated group activities, and it does it more reliably and accurately compared to previous approaches.
This paper describes a novel tracking performance evaluation metric based on the successful detection of events, rather than low-level image processing criteria. A general event metric is defined to measure whether the agents and actions in the scene given by the ground truth were correctly tracked by comparing two event lists using dynamic programming. This metric is suitable to evaluate and compare different tracking approaches where the underlying algorithm may be completely different. Furthermore, we introduce an automatic extraction of those semantically high level events from different types of low level tracking data and human annotated ground truth. A case study with two different trackers on public datasets shows the effectiveness of this evaluation scheme.
In surveillance applications, it is common to have multiple cameras observing targets exhibiting motion on a ground plane. Tracking and estimation of the location of a target on the plane becomes an important inference problem. In this paper, we study the problem of combining estimates of location obtained from multiple cameras. We model the relation between the uncertainty in the location estimation to the position and location of the camera with respect to the plane (which is encoded by a 2D projective transformation). This is addressed by a theoretical study of the properties of a random variable under a projective transformation and analysis of the geometric setting when the moments of the transformed random variable exist. In this context, we prove that ground plane tracking near the horizon line is often inaccurate. Using suitable approximations to compute the moments, a minimum variance estimator is designed to fuse the multi-camera location estimates. Finally, we present experimental results that illustrate the importance of such modeling in location estimation and tracking.
