Robot manipulators require knowledge about their environment in order to perform their desired actions. In several robotic tasks, vision sensors play a critical role by providing the necessary quantity and quality of information regarding the robot's environment. For example, “visual servoing” algorithms may control a robot manipulator in order to track moving objects that are being imaged by a camera. Current visual servoing systems often lack the ability to detect automatically objects that appear within the camera's field of view. In this research, we present a robust “figureiground” framework for visually detecting objects of interest. An important contribution of this research is a collection of optimization schemes that allow the detection framework to operate within the real-time limits of visual servoing systems. The most significant of these schemes involves the use of “spontaneous” and “continuous” domains. The number and location of continuous domains are. allowed to change over time, adjusting to the dynamic conditions of the detection process. We have developed actual servoing systems in order to test the framework's feasibility and to demonstrate its usefulness for visually controlling a robot manipulator.

This paper presents a framework for the automatic visual detection of moving objects in robotic servoing tasks. The paper describes a "figure/ground" scheme which is able to perform detection without making many assumptions that would limit the generality of the approach. We describe optimizations which allow for real-time execution of the frame-differencing that is the basis of our framework. Experimentation has demonstrated that the results of the visual detection can provide information helpful in focusing attention on a specific subset of objects. Furthermore, we show how the detection technique can be integrated with existing methods for visual tracking. We mention some details that have been addressed in order to apply these theories to an actual robotic system, including the use of an optimal controller Our paper explains how this framework has been implemented in our experimental robotic system MRVT, and it describes several results obtained from this experimentation.

The system proposed in this paper uses active deformable models to track pedestrians moving in dynamic real-world scenes. First, figure pixels are separated from a fixed or slowly evolving ground image. Then, an initial segmentation process identifies interesting pixel blobs for tracking. The output of the segmentation process is used to choose the starting position of the control points of the active deformable model. Once tracking has begun, the control points are updated at frame rates by minimizing an energy function involving the relative position of model points, image data, and the characteristics of figure pixels.

Traditionally, the robotic visual servoing/tracking problem has received attention from researchers for its interesting control and computer vision issues. However many visual servoing tasks also require the ability to automatically detect moving objects. Until recently, very few efforts have been reported in the area of automatic detection of servoing targets. This paper presents a robust detection scheme for use in robotic visual servoing experiments. It detects and tracks moving objects through the use of a "figure/ground" approach. Experimentation has shown the feasibility of this approach under general conditions. This paper provides a description of the authors system implementation in an experimental robotic system, along with a collection of results. The paper also contains a discussion of problems and issues for future work.