An approach to estimation for hybrid systems is presented that utilizes uncertain perceptional information about the system's mode to improve tracking of its mode and continuous states. This results in significant improvements in situations where previously reported methods of estimation for hybrid systems perform poorly due to poor distinguishability of the modes. Even in applications where the modes are more easily distinguished, the approach presented can improve performance by decreasing the mode estimation delay of the estimator. Here, state tracking is achieved using a new form of Rao-Blackwellized particle filter called the mode-observed Gaussian Particle Filter. This new filter extends existing hybrid estimation algorithms to admit uncertain but discrete mode-related observations in addition to the information available from more traditional sensors. The framework for estimation using both traditional and perceptional information is applicable to any stochastic hybrid system with mode-related perceptional observations available. Furthermore, the computational efficiency of the Rao-Blackwellized particle filter is maintained, making this filter suitable for real-time implementation. An application that motivates this research is an automatic underwater robotic observation system that follows and films individual deep ocean animals. In that context, to improve tracking of agile animals, a mode-observed Gaussian Particle Filter is applied to supplement measurements of relative position and water-relative velocities of the specimen with perception of what the animal is doing (via visual cues). In addition to improving position and velocity tracking, this filter improves estimation of animal behavior modes to facilitate potentially the automated collection of behavioral data. Performance of the new filter is demonstrated using data from simulation as well as field data collected when tracking actual specimens in the ocean.

The Multiple Kill Vehicle (MKV) system, which is being developed by the US Missile Defense Agency (MDA), is a midcourse payload that includes a carrier vehicle and a number of small kill vehicles. During the mission, the carrier vehicle dispenses the kill vehicles to address a complex threat environment and directs each kill vehicle toward the intercept point for its assigned threat object. As part of the long range carrier vehicle sensor development strategy, MDA and project leaders have developed a pathfinder sensor and are in the process of developing two subsequent demonstration sensors to provide proof of concept and to demonstrate technology. To increase the probability of successful development of the sensor system, detailed calibration measurements have been included as part of the sensor development. A detailed sensor calibration can provide a thorough understanding of sensor operation and performance, verifying that the sensor can meet the mission requirements. This approach to instrument knowledge will help ensure the program success and reduce cost and schedule risks. The Space Dynamics Laboratory at Utah State University (SDL) completed a calibration test campaign for the pathfinder sensor in April 2008. Similar calibration efforts are planned in 2009 for the two demonstration sensors. This paper provides an overview of calibration benefits, requirements, approach, facility, measurements, and preliminary results of the pathfinder calibration.

The Multiple Kill Vehicle (MKV) system, which is being developed by the US Missile Defense Agency (MDA), is a midcourse payload that includes a carrier vehicle and a number of small kill vehicles. During the mission, the carrier vehicle dispenses the kill vehicles to address a complex threat environment and directs each kill vehicle toward the intercept point for its assigned threat object. As part of the long range carrier vehicle sensor development strategy, MDA and project leaders have developed a pathfinder sensor and are in the process of developing two subsequent demonstration sensors to provide proof of concept and to demonstrate technology. To increase the probability of successful development of the sensor system, detailed calibration measurements have been included as part of the sensor development. A detailed sensor calibration can provide a thorough understanding of sensor operation and performance, verifying that the sensor can meet the mission requirements. This approach to instrument knowledge will help ensure the program success and reduce cost and schedule risks. The Space Dynamics Laboratory at Utah State University (SDL) completed a calibration test campaign for the pathfinder sensor in April 2008. Similar calibration efforts are planned in 2009 for the two demonstration sensors. This paper provides an overview of calibration benefits, requirements, approach, facility, measurements, and preliminary results of the pathfinder calibration.

A vision-based automatic tracking system for ocean animals in the midwater has been demonstrated in Monterey Bay, CA. Currently, the input to this system is a measurement of relative position of a target with respect to the tracking vehicle, from which relative velocities are estimated by differentiation. In this paper, the estimation of target velocities is extended to use knowledge of the modal nature of the motions of the tracked target and to incorporate the discrete output of an online classifier that categorizes the visually observable body motions of the animal. First, by using a multiple model estimator, a more expressive hybrid dynamical model is imposed on the target. Then, the estimator is augmented to input the discrete classification from the secondary vision algorithm by recasting the process and sensor models as a dynamic Bayesian network (DBN). By leveraging the information in the body motion classifications, the estimator is able to detect mode changes before the resulting changes in velocity are apparent and a significant improvement in velocity estimation is realized. This, in turn, generates the potential for improved closed-loop tracking performance.

This paper presents an approach for shared control of a Remotely Operated Vehicle (ROV) to assist the vehicle's pilots in servicing of moored underwater platforms. By providing precise automatic control of the ROV with respect to the mooring, a shared control system is established such that the ROV hovers automatically with respect to the mooring and the pilot is free to focus only on the manipulation tasks. The positioning system uses a single calibrated camera to measure bearings to several fiduciary markers on a test mooring of known position in the mooring's coordinates. An Unscented Kalman Filter (UKF) fuses these bearing measurements with vehicle sensors such as compass, inclinometers and rate gyros to estimate the relative position and orientation of the mooring with respect to the ROV. The approach leverages technology that has been presented previously for vision-based automatic tracking and observation of deep ocean animals. Results from simulation and field trials of this positioning system using the ROV Ventana in Monterey Bay are presented

A vision-based automatic tracking and observation system installed on an ROV has successfully tracked midwater ocean animals such as jellyfish in Monterey Bay, California. This system uses stereo vision to localize the tracking vehicle with respect to the target of interest and closes control loops to maintain the target in the views of the cameras. Reliance on the vision sensor imposes a constraint on the control system performance to keep the target in the fields of view of the cameras at all times. The constraint can be expressed as maximum allowable pointing and positioning errors, which are inversely proportional to the stando distance to the specimen. For the system to track small specimens at short range, the constraint of keeping the target within the vision cones becomes very dicult to maintain continuously and the out-of-frame events that result are unrecoverable for the current technology. To expand the operational envelope of the system to include observation of smaller specimens at short stando distances, a new approach is demonstrated that complements vision with sensors typically found on underwater vehicles. This approach softens the constraint of the viewing cones of the vision system by allowing tracking to continue during brief out-of-frame events. A non-linear multi-rate estimator implemented with a Sigma Point Kalman Filter (SPKF) fuses vision with water-relative velocities from a Doppler Velocity Log (DVL) and other vehicle measurements. With this estimator, the target’s position relative to the vehicle is propagated during periods of time when the specimen cannot be seen. The design of an estimator for this problem requires consideration of issues such as assumptions about the motion dynamics of the target, limited knowledge of the vehicle’s dynamic model and robustness to unmodeled disturbances. Simulated tracking results and data from field experiments are presented.

A vision-based automatic tracking and observation system installed on an ROV has successfully tracked midwater ocean animals (such as jellyfish) in Monterey Bay, California. This system uses stereo vision to localize the tracking vehicle with respect to the target of interest and closes control loops to maintain the target in the views of the cameras. The vision and control algorithms have been reported previously. This paper documents the partial redesign of the system in response to issues identified during extensive field testing of the system. The sensing system has been redesigned with the specific objectives of improving initialization, achieving better robustness to disturbances, and enabling tracking of smaller specimens. All of these improvements are achieved by redesigning the stereo camera system to increase significantly the volume that is viewable by both cameras. Also, an optional heading control loop now augments the control architecture to prevent sustained clocking of the vehicle due to disturbances in the tracking control system’s null space. With these design improvements, the automatic tracking and observation system has been fielded as a pilot aid on the ROV Ventana, and has proven capable of tracking specimens that vary widely in size, appearance and behavior.

A vision-based automatic tracking system for gelatinous animals has been developed and demonstrated under a program of joint research between the Stanford University Aerospace Robotics Lab and the Monterey Bay Aquarium Research Institute (MBARI). In field tests using MBARI's ROV Ventana in the Monterey Bay, this system has demonstrated fully autonomous closed-loop control of Ventana to track a jellyfish for periods up to 1.5 hours. In these tests, conventional PID and Sliding Mode Control laws have both been used that rely primarily on the measurement of relative position errors derived from the vision-based system. This tracking system has been designed for both ROV and AUV deployments. One difference between the logic embedded in this system and the way human pilots operate is that human pilots typically exploit their a priori knowledge of how a jellyfish moves in formulating their control commands. That is, they do not rely solely on lead information determined through differentiation. Presented here is a first step for incorporating this additional knowledge-based lead information into the automatic control system. The ultimate goal is determine if this can be used to improve the overall performance and robustness of the tracking task. A key step towards quantification of motion behavior of gelatinous animals is a reliable capability to detect motion mode changes. This paper focuses on recognition of mode changes by applying techniques in real-time computer vision and supervised machine learning in the form of a support vector machine (SVM). Methods are presented to distinguish between active and resting modes, and to detect and measure rhythmic patterns in the body motions of these animals.

A vision-based automatic tracking system mounted on the ROV Ventana has successfully tracked jellyfish in Monterey Bay, California as part of a joint project between Stanford University and the Monterey Bay Aquarium Research Institute (MBARI). To enhance performance, improved lead information about the target's motion is desired. Human pilots derive lead information about that motion through the perception of motion modes of these animals and the propulsion associated with them. Although this kind of information is of a different nature than what is typically available to an automatic control system, performance gains could be achieved by incorporation of such information. Jellyfish and related gelatinous animals exhibit very distinct modes of motion that are visually recognizable to human observers. For an autonomous animal-tracking system to interpret the motion of its target, and to generate lead information useful for control, it must first be able to identify the mode motion of the animal under observation. This paper explores techniques in computer vision to detect and recognize the key motion modes and mode change events typical of these animals. Methods are presented to distinguish between active and resting modes, and to detect and monitor rhythmic patterns in the body motions of these animals.