Explainable neural models have gained a lot of attention in recent years. However, conventional encoder–decoder models do not capture information regarding the importance of the involved latent variables and rely on a heuristic a-priori specification of the dimensionality of the latent space or its selection based on multiple trainings. In this paper, we focus on the efficient structuring of the latent space of encoder–decoder approaches for explainable data reconstruction and compression. For this purpose, we leverage the concept of Shapley values to determine the contribution of the latent variables on the model's output and rank them according to decreasing importance. As a result, a truncation of the latent dimensions to those that contribute the most to the overall reconstruction allows a trade-off between model compactness (i.e. dimensionality of the latent space) and representational power (i.e. reconstruction quality). In contrast to other recent autoencoder variants that incorporate a PCA-based ordering of the latent variables, our approach does not require time-consuming training processes and does not introduce additional weights. This makes our approach particularly valuable for compact representation and compression. We validate our approach at the examples of representing and compressing images as well as high-dimensional reflectance data.

Noise-like artifacts are common in measured or fitted data across various domains, e.g. photography, geometric reconstructions in terms of point clouds or meshes, as well as reflectance measurements and the respective fitting of commonly used reflectance models to them. State-of-the-art denoising approaches focus on specific noise characteristics usually observed in photography. However, these approaches do not perform well if data is corrupted with location-dependent noise. A typical example is the acquisition of heterogeneous materials, which leads to different noise levels due to different behavior of the components either during acquisition or during reconstruction. We address this problem by first automatically determining location-dependent noise levels in the input data and demonstrate that state-of-the-art denoising algorithms can usually benefit from this guidance with only minor modifications to their loss function or employed regularization mechanisms. To generate this information for guidance, we analyze patchwise variances and subsequently derive per-pixel importance values. We demonstrate the benefits of such locally-guided denoising at the examples of the Deep Image Prior method and the Self2Self method.

Bird feathers exhibit fascinating reflectance, which is governed by fiber-like structures. Unlike hair and fur, the feather geometric structures follow intricate hierarchical patterns that span many orders of magnitude in scale. At the smallest scales, fiber elements have strongly non-cylindrical cross-sections and are often complemented by regular nanostructures, causing rich structural color. Therefore, past attempts to render feathers using fiber- or texture-based appearance models missed characteristic aspects of the visual appearance. We introduce a new feather modeling and rendering framework, which abstracts the microscopic geometry and reflectance into a microfacet-like BSDF. The R, TRT and T lobes, also known from hair and fur, here account for specular reflection off the cortex, diffuse reflection off the medulla, and transmission due to barbule spacing, respectively. Our BSDF, which does not require precomputation or storage, can be efficiently importance-sampled and readily integrated into rendering pipelines that represent feather geometry down to the barb level. We verify our approach using a BSDF-capturing setup for small biological structures, as well as against calibrated photographs of rock dove neck feathers.

Image-based appearance measurements are fundamentally limited in spatial resolution by the acquisition hardware. Due to the ever-increasing resolution of displaying hardware, high-resolution representations of digital material appearance are desireable for authentic renderings. In the present paper, we demonstrate that high-resolution bidirectional texture functions (BTFs) for materials can be obtained from low-resolution measurements using single-image convolutional neural network (CNN) architectures for image super-resolution. In particular, we show that this approach works for high-dynamic-range data and produces consistent BTFs, even though it operates on an image-by-image basis. Moreover, the CNN can be trained on down-sampled measured data, therefore no high-resolution ground-truth data, which would be difficult to obtain, is necessary. We train and test our method's performance on a large-scale BTF database and evaluate against the current state-of-the-art in BTF super-resolution, finding superior performance.

Appearance modeling is a difficult problem that still receives considerable attention from the graphics and vision communities. Though recent years have brought a growing number of high-quality material databases that have sparked new research, there is a general lack of evaluation benchmarks for performance assessment and fair comparisons between competing works. We therefore release a new dataset and pose a public challenge that will enable standardized evaluations. For this we measured 56 fabric samples with a commercial appearance scanner. We publish the resulting calibrated HDR images, along with baseline SVBRDF fits. The challenge is to recreate, under known light and view sampling, the appearance of a subset of unseen images. User submissions will be automatically evaluated and ranked by a set of standard image metrics. CCS Concepts • Computing methodologies → Reflectance modeling; Appearance and texture representations;

There is a general shortage of standardized comparisons in the field of appearance modeling. We therefore introduce a benchmark for assessing the quality of reflectance models on a dataset of high quality material measurements obtained with a commercial appearance scanner. The dataset currently consists of 56 fabric materials which are measured as radiometrically calibrated HDR images together with a precise surface geometry. We pose a public challenge to attract further participation and spark new research. Participants evaluate their models on provided directional light and view sampling to recreate the appearance of a set of unseen images per material. The results are automatically evaluated under various image metrics and ranked in a public leaderboard. Our benchmark provides standardized testing and thus enables fair comparisons between related works. We also release baseline SVBRDF material fits.

Using Pantora’s anisotropic textile preset, the overall acquisition is performed by rotating the material at five 45◦ steps, while capturing images with all four cameras. For each captured image, individual LED point lights (color filtered or not) are switched on. The entire process results in 348 (4 cameras × 3 rotation steps × 29 LEDs) point lit panchromatic images. 100 of those point lit images have additional color information through band filtered illumination available. Finally, the device captures images for each turntable rotation, viewed by each camera, while moving the linear light source. Depending on a glossiness preset, the linear light source declination angle step size is 4◦ (#steps = 14) for low, 2◦ (#steps = 28) for medium and 0.5◦ (#steps = 165) for high gloss materials. This results in (#steps×5×4) ∈ {280,560,3300} additional images for low, medium or high gloss materials respectively.

The analysis and modeling of cloth has received a lot of attention in recent years. While recent approaches are focused on woven cloth, we present a novel practical approach for the inference of more complex knitwear structures as well as the respective knitting instructions from only a single image without attached annotations. Knitwear is produced by repeating instances of the same pattern, consisting of grid-like arrangements of a small set of basic stitch types. Our framework addresses the identification and localization of the occurring stitch types, which is challenging due to huge appearance variations. The resulting coarsely localized stitch types are used to infer the underlying grid structure as well as for the extraction of the knitting instruction of pattern repeats, taking into account principles of Gestalt theory. Finally, the derived instructions allow the reproduction of the knitting structures, either as renderings or by actual knitting, as demonstrated in several examples.

The use of spatially varying reflectance models (SVBRDF) is the state of the art in physically based rendering and the ultimate goal is to acquire them from real world samples. Recently several promising deep learning approaches have emerged that create such models from a few uncalibrated photos, after being trained on synthetic SVBRDF datasets. While the achieved results are already very impressive, the reconstruction accuracy that is achieved by these approaches is still far from that of specialized devices. On the other hand, fitting SVBRDF parameter maps to the gibabytes of calibrated HDR images per material acquired by state of the art high quality material scanners takes on the order of several hours for realistic spatial resolutions. In this paper, we present a first deep learning approach that is capable of producing SVBRDF parameter maps more than two orders of magnitude faster than state of the art approaches, while still providing results of equal quality and generalizing to new materials unseen during the training. This is made possible by training our network on a large‐scale database of material scans that we have gathered with a commercially available SVBRDF scanner. In particular, we train a convolutional neural network to map calibrated input images to the 13 parameter maps of an anisotropic Ward BRDF, modified to account for Fresnel reflections, and evaluate the results by comparing the measured images against re‐renderings from our SVBRDF predictions. The novel approach is extensively validated on real world data taken from our material database, which we make publicly available under https://cg.cs.uni‐bonn.de/svbrdfs/ .

In the scope of this supplementary material, we provide additional results and information about the evaluation of our pipeline. First, we compare our approach to the sole use of the HemiCNN in order to assess the gain in quality of the reconstructed materials induced by our improvements. Afterwards, we provide information about the synthetic test scene setup and show additional visual as well as quantitative results for a large number of synthesized scenes using various different combinations of material and geometry. In the following quantitative error plots, we average L1 errors over the pixels of each frame. We mask out pixels for which our reconstruction does not provide any estimates to avoid distorting the averaged errors by gaps in the reconstruction.

We present a compact and efficient representation of spectra for accurate rendering using more than three dimensions. While tristimulus color spaces are sufficient for color display, a spectral renderer has to simulate light transport per wavelength. Consequently, emission spectra and surface albedos need to be known at each wavelength. It is practical to store dense samples for emission spectra but for albedo textures, the memory requirements of this approach are unreasonable. Prior works that approximate dense spectra from tristimulus data introduce strong errors under illuminants with sharp peaks and in indirect illumination. We represent spectra by an arbitrary number of Fourier coefficients. However, we do not use a common truncated Fourier series because its ringing could lead to albedos below zero or above one. Instead, we present a novel approach for reconstruction of bounded densities based on the theory of moments. The core of our technique is our bounded maximum entropy spectral estimate. It uses an efficient closed form to compute a smooth signal between zero and one that matches the given Fourier coefficients exactly. Still, a ground truth that localizes all of its mass around a few wavelengths can be reconstructed adequately. Therefore, our representation covers the full gamut of valid reflectances. The resulting textures are compact because each coefficient can be stored in 10 bits. For compatibility with existing tristimulus assets, we implement a mapping from tristimulus color spaces to three Fourier coefficients. Using three coefficients, our technique gives state of the art results without some of the drawbacks of related work. With four to eight coefficients, our representation is superior to all existing representations. Our focus is on offline rendering but we also demonstrate that the technique is fast enough for real-time rendering.

Real-time reflectance reconstruction under uncontrolled illumination conditions is well-known to be a challenging task due to the complex interplay of scene geometry, surface reflectance and illumination. Nonetheless, recent works succeed in recovering both unknown reflectance and illumination in an uncontrolled setting. However, they are either limited regarding the scene complexity (single objects / homogeneous materials) or are not suitable for real-time applications. Our proposed method enables the recovery of heterogeneous surface reflectance (multiple objects and spatially varying materials) in complex scenes at real-time frame rates. We achieve this goal in the following way: First, we perform a 3D scene reconstruction from an input RGB-D stream in real-time. We then use a deep learning based method to estimate Ward BRDF parameters from observations gathered from individual segmented scene objects. Subsequently we refine these reflectance parameters to allow for spatial variations across the object surfaces. We evaluate our method on synthetic scenes and successfully apply it to real-world data.

In a recent journal paper, we introduced a technique to represent reflectance spectra by an arbitrary number of Fourier coefficients. As a special case, we converted tristimulus data to three Fourier coefficients. After summarizing this work, we introduce the Fourier sRGB color space. It is defined in terms of Fourier coefficients but designed to behave similar to sRGB. Textures stored in Fourier sRGB support efficient spectral rendering but can be compressed with techniques designed for sRGB textures. Compression errors are similar to sRGB.

Analytical SVBRDF representations are widely used to represent spatially varying material appearance depending on view and light configurations. State-of-the-art industry-grade SVBRDF acquisition devices allow the acquisition within several minutes. For many materials with a surface reflectance behavior exhibiting complex effects of light exchange such as inter-reflections, self-occlusions or local subsurface scattering, SVBRDFs cannot accurately capture material appearance. We therefore propose a method to transform SVBRDF acquisition devices to full BTF acquisition devices. To this end, we use data-driven linear models obtained from a database of BTFs captured with a traditional BTF acquisition device in order to reconstruct high-resolution BTFs from the SVBRDF acquisition devices' sparse measurements. We deal with the high degree of sparsity using Tikhonov regularization. In our evaluation, we validate our approach on several materials and show that BTF-like material appearance can be generated from SVBRDF measurements in the range of several minutes.

When relighting digitized objects, strong color deviations can arise depending on the illumination conditions if the object's reflectance is only captured in RGB. To guarantee color-correct simulations, it is therefore of great importance to perform appearance capture with a finer spectral sampling than the three broad band RGB channels. Capturing both shape and multi-spectral reflectance at a high quality is a challenging task and - to the best of our knowledge - has not yet been performed at the quality and speed of our approach. We acquire surface geometry and multi-spectral spatially varying reflectance of objects of up to a few centimeters height with the TAC7 device, which is available commercially as of lately. We demonstrate the improvements in color-accuracy and the overall quality of the appearance capture by relighting our accurately digitized objects under varying illumination conditions.

In recent years there has been an increasing interest in multispectral imaging hardware. Among many other applications is the color-correct reproduction of materials. In this paper, we aim at circumventing the limitations of most devices, namely extensive acquisition times for acceptable signal-to-noise-ratios. For this purpose we propose a novel approach to spectral imaging that combines high-quality RGB data and spatial filtering of extremely noisy and sparsely measured spectral information. The capability of handling noisy spectral data allows a dramatic reduction of overall exposure times. The speed-up we achieve allows for spectral imaging at practical acquisition times. We use the RGB images for constraining the reconstruction of dense spectral information from the filtered noisy spectral data. A further important contribution is the extension of a commonly used radiometric calibration method for determining the camera response in the lowest, noise-dominated range of pixel values. We apply our approach both to capturing single high-quality spectral images, as well as to the acquisition of image-based multispectral surface reflectance. Our results demonstrate that we are able to lower the acquisition times for such multispectral reflectance from several days to the few hours necessary for an RGB-based measurement.

There have been several approaches to model and capture time-varying materials. Modeling approaches provide good results but are sometimes hard to apply because underlying processes are not yet understood or very complex. In this paper, we present a data-driven approach to record aging effects of metal and car paint with the help of Bidirectional Texture Functions. BTFs precisely capture spatially varying reflectance properties of a given material. However, once captured, one cannot change the appearance of a material that ages. Instead, we measure at several distinct aging steps and combine this information into a time-varying BTF which allows the user to interpolate between different stages of the aging process.