We present a technique for capturing an actor’s live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-multiplexed basis lighting conditions, and to record these conditions with a highspeed video camera so that many conditions are recorded in the span of the desired output frame interval. We investigate several lighting bases for representing the sphere of incident illumination using a set of discrete LED light sources, and we estimate and compensate for subject motion using optical flow and image warping based on a set of tracking frames inserted into the lighting basis. To composite the illuminated performance into a new background, we include a time-multiplexed matte within the basis. We also show that the acquired data enables time-varying surface normals, albedo, and ambient occlusion to be estimated, which can be used to transform the actor’s reflectance to produce both subtle and stylistic effects.

This paper draws from art history and perception to place computer depiction in the broader context of picture production. It highlights the often underestimated complexity of the interactions between features in the picture and features of the represented scene. Depiction is not always a unidirectional projection from a 3D scene to a 2D picture, but involves much feedback and influence from the picture space to the object space. Depiction can be seen as a pre-existing 3D reality projected onto 2D, but also as a 2D pictorial representation that is superficially compatible with an hypothetic 3D scene. We show that depiction is essentially an optimization problem, producing the best picture given goals and constraints. We introduce a classification of basic depiction techniques based on four kinds of issue. The spatial system deals with the mapping of spatial properties between 3D and 2D (including, but not restricted to, perspective projection). The primitive system deals with the dimensionality and mappings between picture primitives and scene primitives. Attributes deal with the assignment of visual properties such as colors, texture, or thickness. Finally, marks are the physical implementations of the picture (e.g. brush strokes, mosaic cells). A distinction is introduced between interaction and picturegeneration methods, and techniques are then organized depending on the dimensionality of the inputs and outputs.

We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance and incident radiance at a surface point. For reflectance, we develop a BRDF tensor representation that can be factorized into adjustable terms for lighting, viewing, and BRDF parameters. For incident radiance, there exists a non-linear relationship between indirect lighting and BRDFs in a scene, which makes linear light transport frameworks such as PRT unsuitable. To overcome this problem, we introduce precomputed transfer tensors (PTTs) which decompose indirect lighting into precomputable components that are each a function of BRDFs in the scene, and can be rapidly combined at run time to correctly determine incident radiance. We additionally describe a method for efficient handling of high-frequency specular reflections by separating them from the BRDF tensor representation and processing them using precomputed visibility information. With relighting based on PTTs, interactive performance with indirect lighting is demonstrated in applications to BRDF animation and material tuning.

Lighting is a fundamental aspect of computer cinematography that involves the placement and configuration of lights to establish mood and enhance storytelling. This process is labor intensive as artists repeatedly adjust the parameters of a large set of complex lights to achieve a desired effect. Typical lighting controls affect the final image indirectly, requiring a large number of trials to obtain a suitable result. We present an interactive system wherein an artist paints desired lighting effects directly into the scene, and the computer solves for parameters that achieve the desired look. The artist can paint color, light shape, shadows, highlights, and reflections using a suite of tools designed for painting light. Our system matches these effects using a nonlinear optimizer made robust by a combination of initial estimates, system design, and user-guided optimization. In contrast, previous work on painting light has not permitted the lights to move, allowing for linear optimization but preventing its use in computer cinematography. To demonstrate our approach we lit several scenes, mainly using a direct illumination renderer designed for computer animation, but also including two other rendering styles. We show that painting interfaces can quickly produce high quality lighting setups, easing the lighting artist’s workflow.

In this paper, we present a lighting design framework for neardiffuse real objects, starting from a set of prerecorded photographs of an object under various lighting conditions. Light sources are placed at fixed positions around an object, for which the intensities are to be determined. Using existing digital imaging software, the lighting designer paints the desired illumination distribution on a photograph of an object. This painted-on illumination is used to determine the light intensities which produce a shading of the real object matching the desired illumination distribution. Certain areas in the painted image can be favored among others by weighting their importance in the optimization algorithm.

The goal of the system presented here is to allow a technical director to interactively adjust the lights in a frame. Since the time to render a cinematic quality frame can often span several hours, simply re-rendering the frame

We present an automated approach for high-quality preview of feature-film rendering during lighting design. Similar to previous work, we use a deep-framebuffer shaded on the GPU to achieve interactive performance. Our first contribution is to generate the deep-framebuffer and corresponding shaders automatically through data-flow analysis and compilation of the original scene. Cache compression reduces automatically-generated deep-framebuffers to reasonable size for complex production scenes and shaders. We also propose a new structure, the indirect framebuffer, that decouples shading samples from final pixels and allows a deep-framebuffer to handle antialiasing, motion blur and transparency efficiently. Progressive refinement enables fast feedback at coarser resolution. We demonstrate our approach in real-world production.

We present an automated approach for high-quality preview of feature-film rendering during lighting design. Similar to previous work, we use a deep-framebuffer shaded on the GPU to achieve interactive performance. Our first contribution is to generate the deep-framebuffer and corresponding shaders automatically through data-flow analysis and compilation of the original scene. Cache compression reduces automaticallygenerated deep-framebuffers to reasonable size for complex production scenes and shaders. We also propose a new structure, the indirect framebuffer, that decouples shading samples from final pixels and allows a deep-framebuffer to handle antialiasing, motion blur and transparency efficiently. Progressive refinement enables fast feedback at coarser resolution. We demonstrate our approach in real-world production.

Figure 1: Images rendered by Lpics relighting engine versus software renderer. Times reported are the time a lighting artist must wait for feedback after moving one light. In computer cinematography, the process of lighting design involves placing and configuring lights to define the visual appearance of environments and to enhance story elements. This process is labor intensive and time consuming, primarily because lighting artists receive poor feedback from existing tools: interactive previews have very poor quality, while final-quality images often take hours to render. This paper presents an interactive cinematic lighting system used in the production of computer-animated feature films containing environments of very high complexity, in which surface and light appearances are described using procedural RenderMan shaders. Our system provides lighting artists with high-quality previews at interactive framerates with only small approximations compared to the final rendered images. This is accomplished by combining numerical estimation of surface response, image-space caching, deferred shading, and the computational power of modern graphics hardware. Our system has been successfully used in the production of two feature-length animated films, dramatically accelerating lighting tasks. In our experience interactivity fundamentally changes an artist’s workflow, improving both productivity and artistic expressiveness.

image based lighting, graphics, information theory This paper presents a principled method for choosing informative lighting directions for physical objects. An ensemble of images of an object or scene is captured, each with a known, predetermined lighting direction. Diffuse reflection functions are then estimated for each pixel across such an ensemble. Once these are estimated, the object or scene can be interactively relit as it would appear illuminated from an arbitrary lighting direction. We present two approaches for evaluating images as a function of lighting direction. The first uses image compressibility evaluated across a grid of samples in lighting space. The second uses image variance and prediction error variance, which are monotonically related to compressibility for Gaussian distributions. The advantage of the variance approach is that both image variance and prediction error variance can be analytically derived from the scene reflection functions, and evaluated at the rate of a few nanoseconds per lighting direction.