In spite of over two decades of intense research, illumination and pose invariance remain prohibitively challenging aspects of face recognition for most practical applications. The objective of this work is to recognize faces using video sequences both for training and recognition input, in a realistic, unconstrained setup in which lighting, pose and user motion pattern have a wide variability and face images are of low resolution.

In this paper, we present a new method to change the illumination condition of a face image, with unknown face geometry and albedo information. This problem is particularly difficult when there is only one single image of the subject available and it was taken under a harsh lighting condition. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using spherical harmonic representation. However, the approximation error can be large under harsh lighting conditions [2] thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion based framework that uses a Markov Random Field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to harsh lighting conditions, but insensitive to partial occlusions as well. The performance of our framework is demonstrated through various experimental results, including the improvement to the face recognition rate under harsh lighting conditions. 1.

Classification with only one labeled example per class is a challenging problem in machine learning and pattern recognition. While there have been some attempts to address this problem in the context of specific applications, very little work has been done so far on the problem under more general object classification settings. In this paper, we propose a graph-based approach to the problem. Based on a robust path-based similarity measure proposed recently, we construct a weighted graph using the robust path-based similarities as edge weights. A kernel matrix, called graph Laplacian kernel, is then defined based on the graph Laplacian. With the kernel matrix, in principle any kernel-based classifier can be used for classification. In particular, we demonstrate the use of a kernel nearest neighbor classifier on some synthetic data and real-world image sets, showing that our method can successfully solve some difficult classification tasks with only very few labeled examples. 1.

We present a closed-form solution to the problem of recovering the 3D shape of a non-rigid potentially stretchable surface from 3D-to-2D correspondences. In other words, we can reconstruct a surface from a single image without a priori knowledge of its deformations in that image. State-of-the-art solutions to non-rigid 3D shape recovery rely on the fact that distances between neighboring surface points must be preserved and are therefore limited to inelastic surfaces. Here, we show that replacing the inextensibility constraints by shading ones removes this limitation while still allowing 3D reconstruction in closed-form. We demonstrate our method and compare it to an earlier one using both synthetic and real data. 1.

Pose and illumination variations remain a persistent challenge in face recognition. In this paper, we present a framework for face recognition from video sequences that is robust to large changes in facial pose and lighting conditions. Our method is based on a recently obtained theoretical result that can integrate the effects of motion, lighting and shape in generating an image using a perspective camera. This result can be used to estimate the pose and structure of the face and the illumination conditions for each frame in a video sequence in the presence of multiple point and extended light sources. The pose and illumination estimates in the probe and gallery sequences can then be compared for recognition applications. If similar parameters exist in both the probe and gallery, the similarity between the set of images can be directly computed. If the lighting and pose parameters in the probe and gallery are different, we will synthesize the images using the face model estimated from the training data corresponding to the conditions in the probe sequences. The method can handle situations where the pose and lighting conditions in the training and testing data are very different. We will show results on a video-based face recognition dataset that we have collected. 2

Abstract. Face recognition under varying pose is a challenging problem, especially when illumination variations are also present. Under Lambertian model, spherical harmonics representation has proved to be effective in modelling illumination variations for a given pose. In this paper, we extend the spherical harmonics representation to encode pose information. More specifically, we show that 2D harmonic basis images at different poses are related by close-form linear combinations. This enables an analytic method for generating new basis images at a different pose which are typically required to handle illumination variations at that particular pose. Furthermore, the orthonormality of the linear combinations is utilized to propose an efficient method for robust face recognition where only one set of front-view basis images per subject is stored. In the method, we directly project a rotated testing image onto the space of front-view basis images after establishing the image correspondence. Very good recognition results have been demonstrated using this method. 1

Human faces are neither exactly Lambertian nor entirely convex and hence most models in literature which make the Lambertian assumption, fall short when dealing with specularities and cast shadows. In this paper, we present a novel anti-symmetric tensor spline (a spline for tensorvalued functions) based method for the estimation of the Apparent BRDF (ABRDF) field for human faces that seamlessly accounts for specularities and cast shadows. Furthermore, unlike other methods, it does not require any 3D information to build the model and can work with as few as 9 images. In order to validate the accuracy of our antisymmetric tensor spline model, we present a novel approximation of the ABRDF using a continuous mixture of singlelobed spherical functions. We demonstrate the effectiveness of our anti-symmetric tensor-spline model in comparison to other popular models in the literature, by presenting extensive results for face relighting and face recognition using the Extended Yale B database. 1.

Pose and illumination variations remain a persistent problem in face recognition algorithms. In this paper we present a method for accurately estimating the pose and illumination conditions, and use it for registration and tracking in video-based face recognition algorithms. This is achieved by using a joint motion, illumination and shape model that is bilinear in the motion and illumination variables. The motion is represented in terms of translation and rotation of the object centroid, and the illumination is represented using a spherical harmonics linear basis. We start by estimating a rough pose by projecting the image onto the spherical harmonics basis functions. This pose estimate is used to initialize the registration algorithm, which works by minimizing a square error criterion in the bilinear model. Thereafter, 3D tracking proceeds by alternately estimating motion and illumination parameters. The method does not assume any model for the variation of the illumination conditions- lighting can change slowly or drastically and can originate from combination of point and extended sources. We demonstrate the effectiveness of our methods on several real-world video sequences under severe changes of lighting conditions.

Desktop video conferencing often suffers from bad lighting, which may be caused by harsh shadowing, saturated regions, etc. The primary reason for this is the lack of control over lighting in the user’s environment. A hardware-based solution to this problem would be to place lights near the video camera, but these would be distracting to the user. We use a set of infrared lights placed around the computer monitor to gather a sequence of frames which is used to infer surface normals of the scene. These are used in combination with a visible spectrum image to create an improved relighting result. Index Terms — face relighting, image based rendering, video conferencing 1.

The objective of this work is to recognize faces using video sequences both for training and novel input, in a realistic, unconstrained setup in which lighting, pose and user motion pattern have a wide variability and face images are of low resolution. There are three major areas of novelty: (i) illumination generalization is achieved by combining coarse histogram correction with fine illumination manifold-based normalization; (ii) pose robustness is achieved by decomposing each appearance manifold into semantic Gaussian pose clusters, comparing the corresponding clusters and fusing the results using an RBF network; (iii) we describe a fully automatic recognition system based on the proposed method and an extensive empirical evaluation on 600 head motion video sequences with extreme illumination, pose and motion pattern variation. On this challenging data set our system consistently demonstrated a very high recognition rate (95% on average).