Taking a portrait picture is frequently an uncomfortable task for the photographed person, and the resulting shots are in many cases unnatural and unsatisfactory. A common solution is to take a large number of pictures and then select the favorite ones. Instead of a manual selection, an automatic selection of good portraits from a captured video sequence can increase the chances to obtain a pleasing portrait picture without much effort. In this paper, we present an automatic solution for classification of face images from video sequences. This, in turn, can not only ease the task of taking good portrait pictures, but it can also make the acquisition of portraits more comfortable. 1

We propose a framework for automatic enhancement of group photographs by facial expression analysis. We are motivated by the observation that group photographs are seldom perfect. Subjects may have inadvertently closed their eyes, may be looking away, or may not be smiling at that moment. Given a set of photographs of the same group of people, our algorithm uses facial analysis to determine a goodness score for each face instance in those photos. This scoring function is based on classifiers for facial expressions such as smiles and eye-closure, trained over a large set of annotated photos. Given these scores, a best composite for the set is synthesized by (a) selecting the photo with the best overall score, and (b) replacing any low-scoring faces in that photo with high-scoring faces of the same person from other photos, using alignment and seamless composition.

Frame Number Figure 1: A plot of the ratings assigned by humans in our psychology study (mean shown in dark gray, per-frame standard deviation shown in light gray), and the ratings assigned by our predictive model (cyan) across the frames of a short video sequence. Both series of ratings have been normalized by their mean and standard deviation. We also show several automatically-selected video frames at peaks (green, top) and valleys (red, bottom) of our predicted rating. In this paper, we train a computer to select still frames from video that work well as candid portraits. Because of the subjective nature of this task, we conduct a human subjects study to collect ratings of video frames across multiple videos. Then, we compute a number of features and train a model to predict the average rating of a video frame. We evaluate our model with cross-validation, and show that it is better able to select quality still frames than previous techniques, such as simply omitting frames that contain blinking or motion blur, or selecting only smiles. We also evaluate our technique qualitatively on videos that were not part of our validation set, and were taken outdoors and under different lighting conditions. Links: DL PDF WEB VIDEO DATA 1