The problems posed by the representation and recognition of the movements of 3-D shapes are analysed. A representation is proposed for the movements of shapes that lie within the scope of the Marr & Nishihara (1978) 3-D model representation of static shapes. The basic problem is how to segment a stream of movement into pieces, each of which can be described separately. The representation proposed here is based upon segmenting a movement at moments when a component axis, e. g. an arm, starts to move relative to its local coordinate frame (here the torso). For example, walking is divided into a segment of the stationary states between each swing of the arms and legs, and the actual motions between the stationary points (relative to the torso, not the ground). This representation is called the state─motion─state (SMS) moving shape representation, and several examples of its application are given.

Marr's demonstrations that retinal receptive field geometry could be derived by Fourier transformation of spatial frequency sensitivity data, that edges and contours could be detected by finding zero crossings in the light gradient by taking the Laplacian or second directional derivative, that excitatory and inhibitory receptive fields could be constructed from DOG functions (the difference of two Gaussians), and that the visual system used a two-dimensional convolution integral with a Gaussian prefilter as an operator for bandwidth optimation on the retinal light distribution, were more powerful than anything that had been seen up to that time. It was as if vision research suddenly acquired its own Principia Mathematica, or perhaps General Relativity Theory, in terms of the new explanatory power Marr's theories provided. Truly an extraordinary book from an extraordinary thinker in the area of perception, vision, and the brain.

The construction of directionally selective units, and their use in the processing of visual motion, are considered. The zero crossings of ∇ 2 G(x, y) ∗ I(x, y) are located, as in Marr & Hildreth (1980). That is, the image is filtered through centre-surround receptive fields, and the zero values in the output are found. In addition, the time derivative ∂[∇ 2 G(x, y) ∗ l(x, y) ]/∂ t is measured at the zero crossings, and serves to constrain the local direction of motion to within 180°. The direction of motion can be determined in a second stage, for example by combining the local constraints. The second part of the paper suggests a specific model of the information processing by the X and Y cells of the retina and lateral geniculate nucleus, and certain classes of cortical simple cells. A number of psychophysical and neurophysiological predictions are derived from the theory.

We suggest from the psychophysical data on two-point and line acuity that the smallest foveal channel in human vision must have an excitatory center with a diameter of around 1' 20". Taking into account the optics of the eye and the finite size of the receptors, we show that this would correspond well to the properties of a retinal ganglion cell receiving excitatory input from a single cone.

A theory of edge detection is presented. The analysis proceeds in two parts. (1) Intensity changes, which occur in a natural image over a wide range of scales, are detected separately at different scales. An appropriate filter for this purpose at a given scale is found to be the second derivative of a Gaussian, and it is shown that, provided some simple conditions are satisfied, these primary filters need not be orientation-dependent. Thus, intensity changes at a given scale are best detected by finding the zero values of delta 2G(x,y)*I(x,y) for image I, where G(x,y) is a two-dimensional Gaussian distribution and delta 2 is the Laplacian. The intensity changes thus discovered in each of the channels are then represented by oriented primitives called zero-crossing segments, and evidence is given that this representation is complete. (2) Intensity changes in images arise from surface discontinuities or from reflectance or illumination boundaries, and these all have the property that they are spatially. Because of this, the zero-crossing segments from the different channels are not independent, and rules are deduced for combining them into a description of the image. This description is called the raw primal sketch. The theory explains several basic psychophysical findings, and the operation of forming oriented zero-crossing segments from the output of centre-surround delta 2G filters acting on the image forms the basis for a physiological model of simple cells (see Marr & Ullman 1979).

For human vision to be explained by a computational theory, the first question is plain: What are the problems the brain solves when we see? It is argued that vision is the construction of efficient symbolic descriptions from images of the world. An important aspect of vision is therefore the choice of representations for the different kinds of information in a visual scene. An overall framework is suggested for extracting shape information from images, in which the analysis proceeds through three representations; (1) the primal sketch, which makes explicit the intensity changes and local two-dimensional geometry of an image, (2) the 2 1/2-D sketch, which is a viewer-centred representation of the depth, orientation and discontinuities of the visible surfaces, and (3) the 3-D model representation, which allows an object-centred description of the three-dimensional structure and organization of a viewed shape. The critical act in formulating computational theories for processes capable of constructing these representations is the discovery of valid constraints on the way the world behaves, that provide sufficient additional information to allow recovery of the desired characteristic. Finally, once a computational theory for a process has been formulated, algorithms for implementing it may be designed, and their performance compared with that of the human visual processor.

Under appropriate conditions zero-crossings of a bandpass signal are very rich in information. The authors examine here the relevance of this result to the early stages of visual information processing, where zero-crossings in the output of independent spatial-frequency-tuned channels may contain sufficient information for much of the subsequent processing.

An algorithm is proposed for solving the stereoscopic matching problem. The algorithm consists of five steps: (1) Each image is filtered at different orientations with bar masks of four sizes that increase with eccentricity; the equivalent filters are one or two octaves wide. (2) Zero-crossings in the filtered images, which roughly correspond to edges, are localized. Positions of the ends of lines and edges are also found. (3) For each mask orientation and size, matching takes place between pairs of zero-crossings or terminationss of the same sign in the two images, for a range of disparities up to about the width of the mask's central region. (4) Wide masks can control vergence movements, thus causing small masks to come into correspondence. (5) When a correspondence is achieved, it is stored in a dynamic buffer, called the 2 1/2-D sketch. It is shown that this proposal provides a theoretical framework for most existing psychophysical and neurophysiological data about stereopsis. Several critical experimental predictions are also made, for instance about the size of Panum's area under various conditions. The results of such experiments would tell us whether, for example, cooperativity is necessary for the matching process.

For human vision to be explained by a computational theory, the first question is plain: What are the problems the brain solves when we see?

The human visual process can be studied by examining the computational problems associated with deriving useful information from retinal images. In this paper, we apply this approach to the problem of representing three-dimensional shapes for the purpose of recognition. 1. Three criteria, accessibility, scope and uniqueness , and stability and sensitivity , are presented for judging the usefulness of a representation for shape recognition. 2. Three aspects of a representation’s design are considered, (i) the representation’s coordinate system, (ii) its primitives, which are the primary units of shape information used in the representation, and (iii) the organization the representation imposes on the information in its descriptions. 3. In terms of these design issues and the criteria presented, a shape representation for recognition should: (i) use an object-centred coordinate system, (ii) include volumetric primitives of varied sizes, and (iii) have a modular organization. A representation based on a shape’s natural axes (for example the axes identified by a stick figure) follows directly from these choices. 4. The basic process for deriving a shape description in this representation must involve: (i) a means for identifying the natural axes of a shape in its image and (ii) a mechanism for transforming viewer-centred axis specifications to specifications in an object-centred coordinate system. 5. Shape recognition involves: (i) a collection of stored shape descriptions, and (ii) various indexes into the collection that allow a newly derived description to be associated with an appropriate stored description. The most important of these indexes allows shape recognition to proceed conservatively from the general to the specific based on the specificity of the information available from the image. 6. New constraints supplied by a conservative recognition process can be used to extract more information from the image. A relaxation process for carrying out this constraint analysis is described.

Marr and Poggio (1976) recently described a cooperative algorithm that solves the correspondence problem for stereopsis. This article uses a probabilistic technique to analyze the convergence of that algorithm, and derives the conditions governing the stability of the solution state. The actual results of applying the algorithm to random-dot stereograms are compared with the probabilistic analysis. A satisfactory mathematical analysis of the asymptotic behaviour of the algorithm is possible for a suitable choice of the parameter values and loading rules, and again the actual performance of the algorithm under these conditions is compared with the theoretical predictions. Finally, some problems raised by the analysis of this type of “cooperative” algorithm are briefly discussed.

: Vision is the construction of efficient symbolic descriptions from images of the world. An important aspect of vision is the choice of representations for the different kinds of information in a visual scene. In the early stages of the analysis of an image, the representations used depend more on what it is possible to compute from an image than on what is ultimately desirable, but later representations can be more sensitive to the specific needs of recognition. This essay surveys recent work in vision at M.I.T. from a perspective in which the representational problems assume a primary importance. An overall framework is suggested for visual information processing, in which the analysis proceeds through three representations; (1) the primal sketch, which makes explicit the intensity changes and local two-dimensional geometry of an image, (2) the 2 1/2-D sketch, which is a viewer-centered representation of the depth, orientation and discontinuities of the visible surfaces, and (3) the 3-D model representation, which allows an object-centered description of the three-dimensional structure and organization of a viewed shape. Recent results concerning processes for constructing and maintaining these representations are summarized and discussed. (Author)

The goal of Artificial Intelligence is to identify and solve tractable information processing problems. In so doing, two types of theory arise. Here, they are labelled Types 1 and 2, and their characteristics are outlined. This discussion creates a more than usually rigorous perspective of the subject, from which past work and future prospects are briefly reviewed.

The extraction of stereo-disparity information from two images depends upon establishing a correspondence between them. In this article we analyze the nature of the correspondence computation and derive a cooperative algorithm that implements it. We show that this algorithm successfully extracts information from random-dot stereograms, and its implications for the psychophysics and neurophysiology of the visual system are briefly discussed.

The vision problem begins with a large, gray-level intensity array and culminates in a description that depends on that array and on the purpose for which it is being viewed. The problem of interest is what has to go on in between. In this article, we shall restrict our attention to single-frame, monochromatic, monocular images without specularities, reflections, translucency, transparency or light sources and shall study some of the problems that arise in understanding early and intermediate levels of visual information processing.

An introduction is given to a theory of early visual information processing. The theory has been implemented, and examples are given of images at various stages of analysis. It is argued that the first step of consequence is to compute a primitive but rich description of the grey-level changes present in an image. The description is expressed in a vocabulary of kinds of intensity change (EDGE, SHADING-EDGE, EXTENDED-EDGE, LINE, BLOB etc.). Modifying parameters are bound to the elements in the description, specifying their POSITION, ORIENTATION, TERMINATION points, CONTRAST, SIZE and FUZZINESS. This description is obtained from the intensity array by fixed techniques, and it is called the primal sketch. For most images, the primal sketch is large and unwieldy. The second important step in visual information processing is to group its contents in a way that is appropriate for later recognition. From our ability to interpret drawings with little semantic content, one may infer the presence in our perceptual equipment of symbolic processes that can define "place-tokens" in an image in various ways, and can group them according to certain rules. Homomorphic techniques fail to account for many of these grouping phenomena, whose explanations require mechanisms of construction rather than mechanisms of detection. The necessary grouping of elements in the primal sketch may be achieved by a mechanism that has available the processes inferred from above, together with the ability to select items by first order discriminations acting on the elements' parameters. Only occasionally do these mechanisms use downward-flowing information about the contents of the particular image being processed. It is argued that "non-attentive" vision is in practice implemented by these grouping operations and first order discriminations acting on the primal sketch. The class of computations so obtained differs slightly from the class of second order operations on the intensity array. The extraction of a form from the primal sketch using these techniques amounts to the separation of figure from ground. It is concluded that most of the separation can be carried out by using techniques that do not depend upon the particular image in question. Therefore, figure-ground separation can normally precede the description of the shape of the extracted form. Up to this point, higher-level knowledge and purpose are brought to bear on only a few of the decisions taken during the processing. This relegates the widespread use of downward-flowing information to a later stage than is found in current machine-vision programs, and implies that such knowledge should influence the control of, rather than interfering with, the actual data-processing that is taking place lower down.