Work on photometric stereo has shown how to recover the shape and reflectance properties of an object using multiple images taken with a fixed viewpoint and variable lighting conditions. This work has primarily relied on the presence of a single point source of light in each image. In this paper we show how to perform photometric stereo assuming that all lights in a scene are isotropic and distant from the object but otherwise unconstrained. Lighting in each image may be an unknown and arbitrary combination of diffuse, point and extended sources. Our work is based on recent results showing that for Lambertian objects, general lighting conditions can be represented using low order spherical harmonics. Using this representation we can recover shape by performing a simple optimization in a low-dimensional space. We also analyze the shape ambiguities that arise in such a representation. 1.

The problem of low-rank matrix factorization in the presence of missing data has seen significant attention in recent computer vision research. The approach that dominates the literature is EM-like alternation of closed-form solutions for the two factors of the matrix. An obvious alternative is nonlinear optimization of both factors simultaneously, a strategy which has seen little published research. This paper provides a comprehensive comparison of the two strategies by evaluating previously published factorization algorithms as well as some second order methods not previously presented for this problem. We conclude that, although alternation approaches can be very quick, their propensity to glacial convergence in narrow valleys of the cost function means that averagecase performance is worse than second-order strategies. Further, we demonstrate the importance of two main observations: one, that schemes based on closed-form solutions alone are not suitable and that non-linear optimization strategies are faster, more accurate and provide more flexible frameworks for continued progress; and two, that basic objective functions are not adequate and that regularization priors must be incorporated, a process that is easier with nonlinear methods. 1.

this paper. This method is described in detail in [15]. We can briefly describe the method as formulating the least squares problem as a bilinear optimization, and then iteratively holding one set of variables constant while the others are optimized, so that each optimization is linear. We use their method in our experiments, because it has good convergence properties and is easy to implement. For the problem they consider, Shum et al. state that a random starting point is sufficient to produce a good final solution. However, their experiments on this point cannot be used to draw conclusions for the problem of determining 3-D structure from a sequence of 2-D images. 3 A Novel Algorithm

Abstract. Appearance-based modeling of objects and scenes using PCA has been successfully applied in many recognition tasks. Robust methods which have made the recognition stage less susceptible to outliers, occlusions, and varying illumination have further enlarged the domain of applicability. However, much less research has been done in achieving robustness in the learning stage. In this paper, we propose a novel robust PCA method for obtaining a consistent subspace representation in the presence of outlying pixels in the training images. The method is based on the EM algorithm for estimation of principal subspaces in the presence of missing data. By treating the outlying points as missing pixels, we arrive at a robust PCA representation. We demonstrate experimentally that the proposed method is efficient. In addition, we apply the method to a set of panoramic images to build a representation that enables surveillance and view-based mobile robot localization. 1

This paper considers the problem of factorizing a matrix with missing components into a product of two smaller matrices, also known as principal component analysis with missing data (PCAMD). The Wiberg algorithm is a numerical algorithm developed for the problem in the community of applied mathematics. We argue that the algorithm has not been correctly understood in the computer vision community. Although there are many studies in our com-munity, almost every one of which refers to the Wiberg study, as far as we know, there is no literature in which the performance of the Wiberg algorithm is investigated or the detail of the algorithm is presented. In this paper, we present derivation of the algorithm along with a problem in its implementation that needs to be carefully considered, and then examine its performance. The experimental results demonstrate that the Wiberg algorithm shows a consid-erably good performance, which should contradict the conventional view in our community, namely that minimization-based algorithms tend to fail to converge to a global minimum rel-atively frequently. The performance of the Wiberg algorithm is such that even starting with random initial values, it converges in most cases to a correct solution, even when the matrix has many missing components and the data are contaminated with very strong noise. Our con-clusion is that the Wiberg algorithm can also be used as a standard algorithm for the problems of computer vision. 3 1

The known factorization algorithm for the maximum likelihood affine reconstruction requires that all the feature points used must be visible in all views. We derive here a closed-form-expression for the 3D coordinates of the feature points and translation vectors given the inhomogeneous affine projection matrices, but no single feature point is required to be visible in all views. The expression represents a closed-form maximum likelihood affine triangulation under missing data and unknown translations and it implies an expectation maximization algorithm for the maximum likelihood affine reconstruction where all the measured data may be used. The solution also has applications in bundle adjustment and identifying degenerate motion.

Abstract. Several computer vision applications require estimating a rank deficient matrix from noisy observations of its entries. When the observation matrix has no missing data, the LS solution of such problem is known to be given by the SVD. However, in practice, when several entries of the matrix are not observed, the problem has no closed form solution. In this paper, we study two iterative algorithms for minimizing the non-linear LS cost function obtained when estimating rank deficient matrices from partial observations. In the first algorithm, the iterations are the well known Expectation and Maximization (EM) steps that have succeeded in several estimation problems with missing data. The second algorithm, which we call Row-Column (RC), estimates, in alternate steps, the row and column spaces of the solution matrix. Our conclusions are that RC performs better than EM in what respects to the robustness to the initialization and to the convergence speed. We also demonstrate the algorithms when inferring 3D structure from video sequences. 1

Traditional photometric stereo algorithms employ a Lambertian reflectance model with a varying albedo field and involve the appearance of only one object. In this paper, we generalize photometric stereo algorithms to handle all appearances of all objects in a class, in particular the human face class, by making use of the linear Lambertian property. A linear Lambertian object is one which is linearly spanned by a set of basis objects and has a Lambertian surface. The linear property leads to a rank constraint and, consequently, a factorization of an observation matrix that consists of exemplar images of different objects (e.g., faces of different subjects) under different, unknown illuminations. Integrability and symmetry constraints are used to fully recover the subspace bases using a novel linearized algorithm that takes the varying albedo field into account. The effectiveness of the linear Lambertian property is further investigated by using it for the problem of illumination-invariant face recognition using just one image. Attached shadows are incorporated in the model by a careful treatment of the inherent non-linearity in Lambert’s law. This enables us to extend our algorithm to perform face recognition in the presence of multiple illumination sources. Experimental results using standard data sets are presented.

In this paper we address the problem of recovering structure and motion from a large number of intrinsically calibrated perspective cameras. We describe a method that combines (1) weak-perspective reconstruction in the presence of noisy and missing data and (2) an algorithm that updates weakperspective reconstruction to perspective reconstruction by incrementally estimating the projective depths. The method also solves for the reversal ambiguity associated with affine factorization techniques. The method has been successfully applied to the problem of calibrating the external parameters (position and orientation) of several multiple-camera setups. Results obtained with synthetic and experimental data compare favourably with results obtained with nonlinear minimization such as bundle adjustment. 1.

this paper we address the problem of building a class of robust factorization algorithms that solve for the shape and motion parameters with both affine (weak perspective) and perspective camera models. We introduce a Gaussian/uniform mixture model and its associated EM algorithm. This allows us to address parameter estimation within a data clustering approach. We propose a robust technique that works with any affine factorization method and makes it resilient to outliers. In addition, we show how such a framework can be further embedded into an iterative perspective factorization scheme. We carry out a large number of experiments to validate our algorithms and to compare them with existing ones. We also compare our approach with factorization methods that use M-estimators.