Google Street View has provided millions of users with the ability to visually locate businesses around the world using 360° panoramic imagery. Due to the bulky, custom hardware required to precisely geo-locate the imagery, such as laser scanners and high precision GPS devices, Street View experiences have been limited to large areas that can provide cost-effective collections. This has prevented users from discovering places such as the interiors of small businesses.

Importance sampling provides a practical, production-proven method for integrating diffuse and glossy surface reflections with arbitrary image-based environment or area lighting constructs. Here, functions are evaluated at random points across a domain to produce an estimate of an integral. When using a large number of sample points, the method produces a very accurate result of the integral and provides a strong basis for simulating complex problems such as light transport. Frequently, using the necessary number of samples to reach the exact result is too computationally expensive and fewer samples are evaluated at the cost of visual noise, or variance, within the image. Importance sampling offers a means to reduce the variance by skewing the samples toward regions of the illumination integral that provide the most energy. For instance, the direction of specular reflection or a bright light source within an environment more likely represent the final value of the integral than a random sample. The variance can be reduced more efficiently by combining multiple components of the illumination integral, such as the lighting and material function, to determine where to sample, which is the principle of Multiple Importance Sampling (MIS). As an alternative to the noise in importance sampling, Filtered Importance Sampling (FIS) can provide fast integration, where the lighting environment look-ups are prefiltered to give a smoother result with a significantly smaller number of samples. Importance sampling, MIS and FIS have various practical implications. In this quarter-day course, we cover the necessary background for using Monte Carlo-based techniques for direct lighting and explain how various visual effects companies use these shading methods in their production pipelines.

In Mixed Reality (MR) applications, immersion of virtual objects in captured video contributes to the perceived unification of two worlds, one real, one synthetic. Since virtual actors and surround may appear both closer and farther than real objects, compositing must consider spatial relationships in the resulting world. Chroma keying, often called blue screening or green screening, is one common solution to this problem. This method is under‐constrained and most commonly addressed through a combination of environment preparation and commercial products. In interactive MR domains that impose restrictions on the video camera hardware, such as in experiences using video see‐through (VST) head‐mounted displays (HMD), chroma keying becomes even more difficult due to the relatively low camera quality, the use of m ltiple camera sources (one per eye), and the required processing speed. Dealing with these constraints requires a fast and affordable solution. In our approach, we precondition the chroma key by using principal component analysis (PCA) to obtain usable alpha mattes from video streams in real‐time on commodity graphics processing units (GPUs). In addition, we demonstrate how our method compares to off‐line commercial keying tools and how it performs with respect to signal noise within the video stream. Copyright © 2009 John Wiley & Sons, Ltd.

We propose an analysis of numerical integration based on sampling theory, whereby the integration error caused by aliasing is suppressed by pre-filtering. We derive a pre-filter for evaluating the illumination integral yielding filtered importance sampling, a simple GPU-based rendering algorithm for image-based lighting. Furthermore, we extend the algorithm with real-time visibility computation. Free from any pre-computation, the algorithm supports fully dynamic scenes and, above all, is simple to implement.

In the computer graphics production environment, artists often must tweak specific lighting and material parameters to match a mind's eye vision of the appearance of a 3D scene. However, the interaction between a material and a lighting environment is often too complex to cognitively predict without visualization. Therefore, artists operate in a design cycle, where they tweak the parameters, wait for a visualization, and repeat, seeking to obtain a desired look. We propose the use of appearance-driven material design. Here, artists directly design the appearance of reflected light for a specific view, surface point, and time. In this thesis, we discuss several methods for appearance-driven design with homogeneous materials, spatially-varying materials, and appearance-matching materials, where each uses a unique modeling and optimization paradigm. Moreover, we present a novel treatment of the illumination integral using sampling theory that can utilize the computational power of the graphics processing unit (GPU) to provide real-time visualization of the appearance of various materials illuminated by complex environment lighting. As a system, the modeling, optimization and rendering steps all operate on arbitrary geometry and in detailed lighting environments, while still providing instant feedback to the designer. Thus, our approach allows materials to play an active role in the process of set design and story-telling, a capability that was, until now, difficult to achieve due to the unavailability of interactive tools appropriate for artists.

We present a novel approach that allows users to intuitively and interactively manipulate High Dynamic Range (HDR) images using commonly available Low Dynamic Range (LDR) displays. This solves the problem of how to draw with contrasts that are much larger than the monitor can display. Whereas commercial HDR-enabled drawing programs manipulate tone mapped representations of HDR images, we provide an intuitive brush interface that supports interaction with the unmapped HDR imagery. Our approach introduces two new brush constructs to a typical virtual painting interface, such as Adobe Photoshop. First, we present a brush that locally adjusts the display of the HDR image to a dynamic range specified within a real-time, interactive, local histogram of the region around the cursor. This affords precise, quantitative control of the HDR contrast values produced by the brush. Second, we demonstrate a brush that uses the perception of glare as the underlying basis for determining the contrasts painted onto the HDR image, giving artistic control over the HDR contrasts. By maintaining an HDR image, the result is available for further manipulation and processing by algorithms, such as those used in image-based rendering, for which an LDR representation is inadequate. Finally, we use the Graphics Processing Unit to provide real-time visual feedback for the effects of each image manipulation.

Figure 1: (a) Rendering using deterministic importance sampling without filtering produces aliasing in the estimate of the illumination integral, which translates into image artifacts. (b) Randomizing the directions for each image pixel trades alias for visible noise. (c) Our filtered importance sampling suppresses the alias at the cost of slightly blurred highlights. (d) Reference solution. Images (a), (b), and (c) are rendered with 40 importance samples per pixel. (e) Illustration of the size of the filter used on the environment map `i.

Colorization is a computer-assisted process of adding color to a monochrome image or movie. The process typically involves segmenting images into regions and tracking these regions across image sequences. Neither of these tasks can be performed reliably in practice; consequently, colorization requires considerable user intervention and remains a tedious, time-consuming, and expensive task. In this paper we present a simple colorization method that requires neither precise image segmentation, nor accurate region tracking. Our method is based on a simple premise: neighboring pixels in space-time that have similar intensities should have similar colors. We formalize this premise using a quadratic cost function and obtain an optimization problem that can be solved efficiently using standard techniques. In our approach an artist only needs to annotate the image with a few color scribbles, and the indicated colors are automatically propagated in both space and time to produce a fully colorized image or sequence. We demonstrate that high quality colorizations of stills and movie clips may be obtained from a relatively modest amount of user input. CR Categories: I.4.9 [Image Processing and Computer Vision]: Applications;

A majority of the artists in the graphics community use one of the mainstream packages for modeling and designing a scene for 3D rendering. These packages - such as Alias' Maya, Discreet's 3ds Max, or Newtek's Lightwave -provide slider bars and numerical inputs that let the artist design the material's attributes, such as smoothness and metallic qualities. While advanced users develop a mental mapping from numerical input to material appearance, novice users might not have a natural feel for a material's numeric parameterization. We present a novel and intuitive approach for material design through direct control of the material's bidirectional reflectance distribution function (BRDF) via a series of brush strokes. Using our system, BRDF-Shop, the artist can paint highlights onto a spherical canvas and model a physically correct BRDF. Our principal hypothesis behind this approach is that artists understand materials through the shape and position of the highlights. We propose that these brush strokes; in combination with a real-time display, allow the artist to create a BRDF with intrinsic knowledge of how the highlight will appear on a given object.

We present a tutorial on adaptive, or variable, block sizes with both preand post-filtering for improved lossy image compression. We explain the inherit benefits of adaptive block size encoding in comparison to the JPEG and JPEG2000 standard. Additionally, we look at the possibility of using perceptually based importance operators, used in High Dynamic Range imaging and importance sampling, as a method for determining block size. Finally, we implement and compare different adaptive block size techniques after DCT and arithmetic encoding.

We present an interface for quick and intuitive development of arbitrary, but physically correct, Bi-directional Reflectance Distribution Functions, or BRDFs. Our interface, referred to as BRDF-Shop, provides artists the ability to create a BRDF through positioning and manipulating highlights on a spherical canvas. We develop a novel mapping between painted highlights and specular lobes of an extended Ward BRDF model. The implementation of BRDFShop utilizes programmable graphics hardware to provide a realtime visualization of the material on a complex object in environment lighting. CR Categories: I.3.4 [Graphics Utilities]: Graphics editors; I.3.6 [Methodologies and Techniques]: Interactive Techniques; I.3.7 [Three-Dimensional Graphics and Realism]: Color, shading, shadowing, and texture

In this paper, we propose three novel approaches to performing vector multiplication on arbitrarily sparse matrices. We developed the proposed methods specifically for implementation on the Graphics Processing Unit with an emphasis on increasing speed for very large sparse matrices. The novel method, Boxed-TJDS, BoxedCRS and Hybrid Boxed-TJDS, were designed using the preexisting sparse matrix storage formats CRS and TJDS. We perform an empirical analysis of the the novel approaches against an SSE implementation of the mathematical operation using matrices of varying sparsity and size. Our results show that the three novel methods represent a range of possibilities for sparse matrix vector multiplication. While the Hybrid and BoxedTJDS methods perform moderately slower than and comparable to the SSE implementation, respectively, the Boxed-CRS approaches performs considerably faster than SSE as the matrix size increases in non-zero values. ∗{colbert,gaelen,eurip}@cs.ucf.edu

Augmented reality involves mixing captured video with rendered elements in real-time. For augmented reality to be effective in training and simulation applications, the computer generated c omponents need to blend in well with the captured video. Straightforward compositing is not sufficient, since the chromatic content of video and rendered data may be very different such that it is immediately obvious which parts of the composited image were rendered and which were captured. We propose a simple and effective method to color-correct the computer generated imagery. The method relies on the computation of simple statistics such as mean and variance, but does so in an appropriately chosen color space - which is key to the effectiveness of our approach. By shifting and scaling the pixel data in the rendered stream to take on the mean and variance of the captured video stream, the rendered elements blend in very well. Our implementation currently reads, color-corrects and composites video and rendered streams at a rate of more than 22 frames per second for a 720x480 pixel format. Without color correction, our implementation generates around 30 frames per second, indicating that our approach comes at a reasonably small computational cost.