A motion sequence may be represented as a single pattern in x-y-t space; a velocity of motion corresponds to a three-dimensional orientation in this space. Motion sinformation can be extracted by a system that responds to the oriented spatiotemporal energy. We discuss a class of models for human motion mechanisms in which the first stage consists of linear filters that are oriented in space-time and tuned in spatial frequency. The outputs of quadrature pairs of such filters are squared and summed to give a measure of motion energy. These responses are then fed into an opponent stage. Energy models can be built from elements that are consistent with known physiology and psychophysics, and they permit a qualitative understanding of a variety of motion phenomena.

The structure-from-motion problem has been extensively studied in the field of computer vision. Yet, the bulk of the existing work assumes that the scene contains only a single moving object. The more realistic case where an unknown number of objects move in the scene has received little attention, especially for its theoretical treatment. In this paper we present a new method for separating and recovering the motion and shape of multiple independently moving objects in a sequence of images. The method does not require prior knowledge of the number of objects, nor is dependent on any grouping of features into an object at the image level. For this purpose, we introduce a mathematical construct of object shapes, called the shape interaction matrix, which is invariant to both the object motions and the selection of coordinate systems. This invariant structure is computable solely from the observed trajectories of image features without grouping them into individual objects. Once the matr...

We present a biologically-motivated system for the recognition of actions from video sequences. The approach builds on recent work on object recognition based on hierarchical feedforward architectures [25, 16, 20] and extends a neurobiological model of motion processing in the visual cortex [10]. The system consists of a hierarchy of spatio-temporal feature detectors of increasing complexity: an input sequence is first analyzed by an array of motiondirection sensitive units which, through a hierarchy of processing stages, lead to position-invariant spatio-temporal feature detectors. We experiment with different types of motion-direction sensitive units as well as different system architectures. As in [16], we find that sparse features in intermediate stages outperform dense ones and that using a simple feature selection approach leads to an efficient system that performs better with far fewer features. We test the approach on different publicly available action datasets, in all cases achieving the highest results reported to date. 1.

This thesis describes some new approaches to the representation and analysis of visual motion, as perceived by a biological or machine visual system. We begin by discussing the computation of image motion fields, the projection of motion in the three-dimensional world onto the two-dimensional image plane. This computation is notoriously difficult, and there are a wide variety of approaches that have been developed for use in image processing, machine vision, and biological modeling. We show that a large number of the basic techniques are quite similar in nature, differing primarily in conceptual motivation, and that they each fail to handle a set of situations that occur commonly in natural scenery. The central theme of the thesis is that the failure of these algorithms is due primarily to the use of vector fields as a representation for visual motion. We argue that the translational vector field representation is inherently impoverished and error-prone. Furthermore, there is evidence that a ...

Electrophysiological studies indicate that neurons in the Middle Temporal (MT) area of the primate brain are selective for the velocity of visual stimuli. This paper describes a computational model of MT physiology, in which local image velocities are represented via the distribution of MT neuronal responses. The computation is performed in two stages, corresponding to neurons in cortical areas V1 and MT. Each stage computes a weighted linear sum of inputs, followed by rectification and divisive normalization. V1 receptive field weights are designed for orientation and direction selectivity. MT receptive field weights are designed for velocity (both speed and direction) selectivity. The paper includes computational simulations accounting for a wide range of physiological data, and describes experiments that could be used to further test and refine the model.

Many psychophysical and physiological experiments indicate that visual motion analysis and stereoscopic depth perception are processed together in the brain. However, little computational effort has been devoted to combining these two visual modalities into a common framework based on physiological mechanisms. We present such an integrated model in this paper. We have previously developed a physiologically realistic model for binocular disparity computation (Qian, 1994). Here we demonstrate that under some general and physiological assumptions, our stereo vision model can be combined naturally with motion energy models to achieve motionstereo integration. The integrated model may be used to explain a wide range of experimental observations regarding motion-stereo interaction. As an example, we show that the model can provide a unified account of the classical Pulfrich effect (Morgan and Thompson, 1975) and the generalized Pulfrich phenomena to dynamic noise patterns (Tyler, 1974; Falk,...

This paper presents a model for the perception of transparently combined moving images. We advocate a framework consisting of a local motion mechanism which can operate in the presence of transparency, and a global mechanism that integrates information across space. We present a new method for the local motion testing mechanism, using "donut" velocity selective mechanisms formed from the weighted combination of spatio-temporal energy units. This method has the advantage over traditional methods that it does not fail when there are multiple motions in the sequence. The global layer selection mechanism attempts to account for the local velocity distributions with a small set of global functions. Using donut mechanisms permits a simplified layer selection optimization, in which inhibition between layers is determined by the product of their predicted velocity distributions. With this scheme, we demonstrate the decomposition of image sequences containing additively combined multiple moving...

Lu and Sperling [Vision Res. 35, 2697 (1995)] proposed that human visual motion perception is served by three separate motion systems: a first-order system that responds to moving luminance patterns, a second-order system that responds to moving modulations of feature types—stimuli in which the expected luminance is the same everywhere but an area of higher contrast or of flicker moves, and a third-order system that computes the motion of marked locations in a ‘‘salience map,’ ’ that is, a neural representation of visual space in which the locations of important visual features (‘‘figure’’) are marked and ‘‘ground’ ’ is unmarked. Subsequently, there have been some strongly confirmatory reports: different gain-control mechanisms for first- and second-order motion, selective impairment of first- versus second- and/or third-order motion by different brain injuries, and the classification of new third-order motions, e.g., isoluminant chromatic motion. Various procedures have successfully discriminated between second- and third-order motion (when first-order motion is excluded): dual tasks, second-order reversed phi, motion competition, and selective adaptation. Meanwhile, eight apparent contradictions to the three-systems theory have been proposed. A review and reanalysis here of the new evidence, pro and con, resolves the challenges and yields a more clearly defined and significantly strengthened theory. © 2001 Optical Society of America OCIS codes: 330.4150.

Adaptation to a moving visual pattern induces shifts in the perceived motion of subsequently viewed moving patterns. Explanations of such effects are typically based on adaptation-induced sensitivity changes in spatio-temporal frequency tuned mechanisms (STFMs). An alternative hypothesis is that adaptation occurs in mechanisms that independently encode direction and speed (DSMs). Yet a third possibility is that adaptation occurrs in mechanisms that encode 2D pattern velocity (VMs). We performed a series of psychophysical experiments to examine predictions made by each of the three hypotheses. The results indicate that: (1) adaptation-induced shifts are relatively independent of spatial pattern of both adapting and test stimuli; (2) the shift in perceived direction of motion of a plaid stimulus after adaptation to a grating indicates a shift in the motion of the plaid pattern, and not a shift in the motion of the plaid components; and (3) the 2D pattern of shift in perceived velocity ra...

Optical flow estimation is a critical mechanism for autonomous mobile robots as it provides a range of useful information. As real-time processing is mandatory in this case, an efficient solution is the use of specific VLSI analog circuits. This paper presents a simple and regular architecture based on analog circuits which implements the entire processing line from photoreceptor to accurate and reliable optical flow estimation. The algorithm we propose, is an energy-based method using a novel wideband velocitytuned filter whichproves to be an efficient alternativetothe well known Gabor filters. Our approach shows that a high level of accuracy can be obtained from a small number of loosely tuned filters. It exhibits similar or improved performance to that of other existing algorithms, but with a much lower complexity.