This paper describes the Semi-Global Matching (SGM) stereo method. It uses a pixelwise, Mutual Information based matching cost for compensating radiometric differences of input images. Pixelwise matching is supported by a smoothness constraint that is usually expressed as a global cost function. SGM performs a fast approximation by pathwise optimizations from all directions. The discussion also addresses occlusion detection, subpixel refinement and multi-baseline matching. Additionally, postprocessing steps for removing outliers, recovering from specific problems of structured environments and the interpolation of gaps are presented. Finally, strategies for processing almost arbitrarily large images and fusion of disparity images using orthographic projection are proposed. A comparison on standard stereo images shows that SGM is among the currently top-ranked algorithms and is best, if subpixel accuracy is considered. The complexity is linear to the number of pixels and disparity range, which results in a runtime of just 1-2s on typical test images. An in depth evaluation of the Mutual Information based matching cost demonstrates a tolerance against a wide range of radiometric transformations. Finally, examples of reconstructions from huge aerial frame and pushbroom images demonstrate that the presented ideas are working well on practical problems.

In this paper, we present a belief propagation based global algorithm that generates high quality results while maintaining real-time performance. To our knowledge, it is the first BP based global method that runs at real-time speed. Our efficiency performance gains mainly from the parallelism of graphics hardware,which leads to a 45 times speedup compared to the CPU implementation. To qualify the accurancy of our approach, the experimental results are evaluated on the Middlebury data sets, showing that our approach is among the best (ranked first in the new evaluation system) for all real-time approaches. In addition, since the running time of general BP is linear to the number of iterations, adopting a large number of iterations is not feasible for practical applications. Hence a novel approach is proposed to adaptively update pixel cost. Unlike general BP methods, the running time of our proposed algorithm dramatically converges.

Matching cost aggregation is one of the oldest and still popular methods for stereo correspondence. While effec-tive and efficient, cost aggregation methods typically aggregate the matching cost by summing/averaging over a user-specified, local support region. This is obviously on-ly locally-optimal, and the computational complexity of the full-kernel implementation usually depends on the region size. In this paper, the cost aggregation problem is re-examined and a non-local solution is proposed. The match-ing cost values are aggregated adaptively based on pixel similarity on a tree structure derived from the stereo im-age pair to preserve depth edges. The nodes of this tree are all the image pixels, and the edges are all the edges between the nearest neighboring pixels. The similarity between any two pixels is decided by their shortest distance on the tree. The proposed method is non-local as every n-ode receives supports from all other nodes on the tree. As can be expected, the proposed non-local solution outper-forms all local cost aggregation methods on the standard (Middlebury) benchmark. Besides, it has great advantage in extremely low computational complexity: only a total of 2 addition/subtraction operations and 3 multiplication oper-ations are required for each pixel at each disparity level. It is very close to the complexity of unnormalized box filtering using integral image which requires 6 addition/subtraction operations. Unnormalized box filter is the fastest local cost aggregation method but blurs across depth edges. The pro-posed method was tested on a MacBook Air laptop comput-er with a 1.8 GHz Intel Core i7 CPU and 4 GB memory. The average runtime on the Middlebury data sets is about 90 milliseconds, and is only about 1.25 × slower than unnor-malized box filter. A non-local disparity refinement method is also proposed based on the non-local cost aggregation method.

Stereo matching, tree-based dynamic programming, fast stereo method. This work describes a fast method for computing dense stereo correspondences that is capable of generating results close to the state-of-the-art. We propose running a separate disparity computation process in each image pixel. The idea is to root a tree graph on the pixel whose disparity needs to be reconstructed. The tree thereby forms an individual approximation of the standard four-connected grid for this specific pixel. An exact optimum of a predefined energy function on the applied tree structure is determined via dynamic programming (DP), and the root pixel is assigned to the disparity of optimal costs. We present two simple tree structures that allow for the efficient calculation of all trees ’ optima with only four scanline-based DP passes. These simple trees are designed to capture all pixels of the reference frame and incorporate horizontal and vertical smoothness edges in order to weaken the scanline streaking problem inherent in DP-based approaches. We evaluate our results using the Middlebury test set. Our algorithm currently ranks at the eighth position of approximately 30 algorithms in the Middlebury database. More importantly, it is the currently best-performing method that does not use image segmentation and is significantly faster than most competing algorithms. Our method needs less than a second to determine the disparity map for typical stereo pairs. 1

Optimization is a powerful paradigm for expressing and solving problems in a wide range of areas, and has been successfully applied to many vision problems. Discrete optimization techniques are especially interesting, since by carefully exploiting problem structure they often provide non-trivial guarantees concerning solution quality. In this paper we briefly review dynamic programming and graph algorithms, and discuss representative examples of how these discrete optimization techniques have been applied to some classical vision problems. We focus on the low-level vision problem of stereo; the mid-level problem of interactive object segmentation; and the high-level problem of model-based recognition.

This paper presents methods for recovering accurate binocular disparity estimates in the vicinity of 3-D surface discontinuities. Of particular concern are methods that impact coarse-to-fine, local block-based matching as it forms the basis of the fastest and the most resource efficient stereo computation procedures. Several advances are put forth. First, a novel coarse-to-fine refinement that adapts match window support across scale to ameliorate corruption of disparity estimates near boundaries is presented; a detailed analysis of coarse-to-fine 3-D boundary processing is given as well. Second, a novel formulation of half-occlusion cues within the coarse-to-fine block matching framework is described; the relation of the proposed solution to previous methods is extensively discussed. Third, the use of colour or intensity segmentation for better recovery of 3-D boundaries is investigated; a formulation specific to a coarse-to-fine local block-based matching is given. Empirical results show that incorporation of these advances in the standard coarse-to-fine, block matching framework reduces disparity errors by a factor of two, while performing

In this paper, we propose a new stereo matching method using the population-based Markov Chain Monte Carlo (Pop-MCMC), which belongs to the sampling-based methods. Since the previous MCMC methods produce only one sample at a time, only local moves are available. In contrast, the proposed Pop-MCMC uses multiple chains in parallel and produces multiple samples at a time. It thereby enables global moves by exchanging information between samples, which in turn, leads to faster mixing rate. In the view of optimization, it means that we can reach a lower energy state rapidly. In order to apply Pop-MCMC to the stereo matching problem, we design two effective 2-D mutation and crossover moves among multiple chains to explore a high dimensional state space efficiently. The experimental results on real stereo images demonstrate that the proposed algorithm gives much faster convergence rate than conventional sampling-based methods including SA (Simulated Annealing) and SWC (Swendsen-Wang Cuts). And it also gives consistently lower energy solutions than BP (Belief Propagation) in our experiments. In addition, we also analyze the effect of each move in Pop-MCMC and examine the effect of parameters such as temperature and the number of the chains.

ABSTRACT This paper presents experimental methods and results for 3D environment reconstruction from monocular video augmented with inertial data. One application targets sparsely furnished room interiors, using high quality handheld video with a normal field of view, and linear accelerations and angular velocities from an attached inertial measurement unit. A second application targets natural terrain with manmade structures, using heavily compressed aerial video with a narrow field of view, and position and orientation data from the aircraft navigation system. In both applications, the translational and rotational offsets between the camera and inertial reference frames are initially unknown, and only a small fraction of the scene is visible in any one video frame. We start by estimating sparse structure and motion fro m 2D feature tracks using a Kalman filter and/or repeated, partial bundle adjustments requiring bounded time per video frame. The first application additionally incorporates a weak assumption of bounding perpendicular planes to minimize a tendency of the motion estimation to drift, while the second application requires tight integration of the navigational data to alleviate the poor conditioning caused by the narrow field of view. This is followed by dense structure recovery via graph-cut-based multi-view stereo, meshing, and optional mesh simplification. Finally, input images are texture-mapped onto the 3D surface for rendering. We show sample results from multiple, novel viewpoints.

Abstract. Dynamic Programming (DP) is a popular and efficient method for calculating disparity maps from stereo images. It allows for meeting real-time constraints even on low-cost hardware. Therefore, it is frequently used in real-world applications, although more accurate algorithms exist. We present a refined DP stereo processing algorithm which is based on a standard implementation. However it is more flexible and shows increased performance. In particular, we introduce the idea of multi-path backtracking to exploit the information gained from DP more effectively. We show how to automatically tune all parameters of our approach offline by an evolutionary algorithm. The performance was assessed on benchmark data. The number of incorrect disparities was reduced by 40 % compared to the DP reference implementation while the overall complexity increased only slightly. 1

Abstract—Illumination changes cause serious problems in many computer vision applications. We present a new method for ad-dressing robust depth estimation from a stereo pair under varying illumination conditions. First, a spatially varying multiplicative model is developed to account for brightness changes induced between left and right views. The depth estimation problem, based on this model, is then formulated as a constrained optimization problem in which an appropriate convex objective function is minimized under various convex constraints modelling prior knowledge and observed information. The resulting multicon-strained optimization problem is finally solved via a parallel block iterative algorithm which offers great flexibility in the incorpora-tion of several constraints. Experimental results on both synthetic and real stereo pairs demonstrate the good performance of our method to efficiently recover depth and illumination variation fields, simultaneously. Index Terms—Convex constraints, convex optimization, depth estimation, illumination variation, regularization, set theoretic es-timation, stereo vision, total variation, variational methods. I.