The transformation of industrial environments is progressing at a fast pace as more and more autonomous systems are installed and operated. Save and explainable AI algorithms are thus essential, especially for collaborative interactive systems that operate in human spaces. We propose the “Semantic Encoder”, a 2D-vision based CNN model trained on a purely synthetic dataset, to address the explainability aspect by extracting semantic descriptions of real objects based on their visual appearances. We can use the extracted semantic information to simply describe depicted samples or to differentiate between normal and anomalous samples, with the possibility to explain what caused the anomaly detection. The semantic description can be further used to sort samples by classifying them or to find a sample with specific semantic properties. We evaluate the Semantic Encoder with respect to its informative power by comparing the computed semantic features with features extracted by a VGG-16 model and classical image processing methods. The results are quantified based on the Generalized Discriminative Value (GDV). We also investigate how accurately anomalous samples are detected by computing ROC and PR curves. We use the semantic parameters to understand what causes good and inaccurate anomaly detection decisions. In addition, we evaluate the quality of the classification based sorting by examining confusion matrices and classification accuracy.

We applied convolutional versions of a "standard" autoencoder (CAE), a variational autoencoder (VAE) and an adversarial autoencoder (AAE) to two different publicly available datasets and compared their anomaly detection performances. We used the MNIST dataset [14] as a simple anomaly detection scenario. The CIFAR10 dataset [13] was used to examine the autoencoders in a more complex anomaly detection task. The anomaly detection performance of our different autoencoder types is compared in a qualitative and quantitative manner. The time needed for training the models is measured to capture their complexity. The simplest model demanding the simplest training, the CAE, computes results which are nearly as accurate and for some cases even better than results achieved by the VAE and AAE. We show that all three autoencoder types computed convincing anomaly detection results for the more simple-structured MNIST scenario. However, none of the autoencoder types proved to capture a good representation of the relevant features of the more complex CIFAR10 dataset, leading to moderately good anomaly detection performances.

We investigated the anomaly detection behaviour of three convolutional autoencoder types a “standard” convolutional autoencoder (CAE), a variational convolutional autoencoder (VAE) and an adversarial convolutional autoencoder (AAE) by applying them to different visual anomaly detection scenarios. First, we utilized our three autoencoder types to detect anomalous regions in two synthetically generated datasets. To investigate the convolutional autoencoders’ defect detection performances “in the industrial wild”, we applied the models on quality inspection images of non-defective and defective material regions. We compared the performances of all three autoencoder types based on their ability to detect anomalies and captured the training complexity by measuring the time needed for training them. Although the CAE is the simplest model, the trained model performed nearly as well as the more sophisticated autoencoder types, which depend on more complex training processes. For data that lacks regularity or shows purely stochastic patterns, all our autoencoders failed to compute meaningful results.

During the homogenization heat treatment of 6xxx series aluminum alloys, nanoscale precipitates—commonly named dispersoids—are formed that influence material properties during further processing by extrusion, forging, or rolling, as well as final product quality. Obtaining dispersoid size distributions is commonly accomplished by manually counting and measuring the diameter of the particles in metallographic sections investigated by means of electron microscopy. An automatization of this process, while desired, is difficult due to varying backgrounds, brightness and contrast levels, noise, dispersoid morphologies as well as scratches and interference from other types of intermetallic phases. In order to segment dispersoids in a wide range of 6xxx series aluminum alloys, a neural network is trained on the basis of electron micrographs of different alloy samples that include various possible separation artifacts and is compared to several benchmark models. The neural network evaluated in this work shows promising results, consistent over all analyzed samples, with a maximum error of roughly 20% while the benchmark models show errors of up to 85%.

We investigated optimal implementation strategies for industrial inspection systems aiming to detect cracks on ground steel billets' surfaces by combining state-of-the-art AI-based methods and classical computational imaging techniques. In 2D texture images, the interesting patterns of surface irregularities are often surrounded by visual clutter, which is to be ignored, e.g., grinding patterns. Even neural networks struggle to reliably distinguish between actual surface disruptions and irrelevant background patterns. Consequently, the image acquisition procedure already has to be optimised to the specific application. In our case, we use photometric stereo (PS) imaging to generate 3D surface models of steel billets using multiple illumination units. However, we demonstrate that the neural networks, especially in high-speed scenarios, still suffer from recognition deficiencies when using raw photometric stereo camera data, and are unable to generalise to new billets and image acquisition conditions. Only the additional application of adequate state-of-the-art image processing algorithms guarantees the best results in both aspects. The neural networks benefit when appropriate image acquisition methods together with image processing algorithms emphasise relevant surface structures and reduce overall pattern variation. Our proposed combined strategy shows a 9.25% better detection rate on validation data and is 14.7% better on test data, displaying the best generalisation.

We conducted a case study on a subset of the MUSCLE CIS image benchmark of modern coins with the goal to assess the potential of deep embedding learning for generating representative CNN feature vectors of coin images, which are clustered class by class. In the course of training our models (CNN), we applied algorithmic rotational augmentation to the coin images to enforce rotational invariance. While augmentation is a usual procedure for regularizing deep learning models towards more geometric invariance, exactly that procedure revealed an interesting yet precarious pitfall in deep embedding learning: its susceptibility to interpolation errors. That interpolation bias results in distorted and ambiguous representation clusters of coin classes in the feature space, jeopardizing classification capabilities.

The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution. To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the resulting generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar results to the perception-distortion tradeoff (Blau & Michaeli (2018)). Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE.

For calculating 3D information with stereo matching, usually correspondence analysis yields a so-called depth hypotheses cost stack, which contains information about similarities of the visible structures at all positions of the analyzed stereo images. Often those cost values comprise a large amount of noise and/or ambiguities, so that regularization is required. The Conditional Random Field (CRF) regularizer from Shekhovtsov et al. [Sh16] is a very good algorithm among various methods. Due to the usual iterative nature of those regularizers, they often do not meet the strict speed and memory requirements posed in many real-world applications. In this paper, we propose to substitute Shekhovtsov’s CRF algorithm with an especially designed U-shaped 3D Convolutional Neural Network (3D-CRF-CNN), which is taught proper regularization by the CRF algorithm as a teacher. Our experiments have shown, that such a 3D-CRF-CNN is not only able to mimic the CRF’s regularizing behavior, but - if properly setup - also comprises remarkable generalization capabilities compared to a state-of-the-art 2D-CNN that is trained on a slightly different, yet equivalent, task. The advantages of such a CNN regularizer are its predictable computational performance and its relatively simple architectural structure, which allows for easy development, speed up, and deployment. We demonstrate the feasibility of the concept of training a 3D-CRF-CNN to take over CRF’s regularizing functionality on the basis of available test data and show that it pays off to invest special effort into tailoring an according CNN architecture.

In industrial inspection settings, it is common that data is either hard or expensive to acquire. Generative modeling offers a way to reduce those costs by filling out scarce training data sets automatically. Generative Adversarial Networks (GANs) have shown incredible results in the field of artificial image data generation, but until recently were not ready for industrial applications, because of unclear performance metrics and instabilities. However, with the introduction of Wasserstein GAN, which comprises an interpretable loss metric and general stability, it is promising to try using those algorithms for industrial classification tasks. Therefore, we present a case study on a single digit image classification task of banknote serial numbers, where we simulate use cases with missing data. For those selected situations, different data generation algorithms were implemented incorporating GANs in various ways to augment scarce training data sets. As a measure of plausibility of thos e artificially generated data, we used the classification performance of a CNN trained on them. We analyzed the gains in classification accuracy when augmenting the training samples with GAN images and compare them to results with either more classically generated, rendered artificial data and near perfect training data situations, respectively.

Generative Adversarial Networks (GANs) have been used to model the underlying probability distribution of sample based datasets. GANs are notoriuos for training difficulties and their dependence on arbitrary hyperparameters. One recent improvement in GAN literature is to use the Wasserstein distance as loss function leading to Wasserstein Generative Adversarial Networks (WGANs). Using this as a basis, we show various ways in which the Wasserstein distance is estimated for the task of generative modelling. Additionally, the secrets in training such models are shown and summarized at the end of this work. Where applicable, we extend current works to different algorithms, different cost functions, and different regularization schemes to improve generative models.

We present a system to extract surface orientation and albedos from a single shot image using three differently colored illumination sources. Photometric stereo allows one to extract local surface information such as normals or gradients. Traditionally, the local orientations and albedos are computed using serveral acquisitions of the same viewing angle and under varying illumination directions. In applications with moving objects, where the acquisition-as well as processing speed are essential, such setups are poorly suited. We propose a single shot decomposition using three differently colored light sources under defined illumination directions. To allow for a fast and regularized inference, we built a split U-shaped convolutional neural network, which takes a single shot input and estimates both the surface orientation and albedo simultaneously.

In industrial quality inspection, it is often the case that a lot of data of desired product appearance can be provided at training time, while very little erroneous examples are available. Thus, in order to train an inspection system, the target appearance has to be learned independently from the availability of defect samples. Defects have to be identified as anomalies w.r.t. the trained data distributions in the online inspection phase. In deep learning, autoencoders are a well known choice to realize anomaly detection scenarios, where significantly larger reconstruction errors of objects’ images indicate defects. However, as the latent code contains enough information to reliably reconstruct good example images, the question arises if a decision about the validity of an input image can already be drawn in that latent space during online inspection. This would speed up the system by more than a factor of 2 by sparing the processing of the autoencoder’s decoder part. Variational Autoencoders (VAE) are a modern variant of the classical autoencoder architecture, which could facilitate this purpose, because of its imposed regularization term, that forces the latent codes to be standard normally distributed.

In this paper, we present a novel method for multi-frame superresolution (SR). Our main goal is to improve the spatial resolution of a multi-line scan camera for an industrial inspection task. High resolution output images are reconstructed using our proposed SR algorithm for multi-channel data, which is based on the trainable reaction-diffusion model. As this is a supervised learning approach, we simulate ground truth data for a real imaging scenario. We show that learning a regularizer for the SR problem improves the reconstruction results compared to an iterative reconstruction algorithm using TV or TGV regularization. We test the learned regularizer, trained on simulated data, on images acquired with the real camera setup and achieve excellent results.

Minutiae extraction is an important step for robust fingerprint identification. However, existing minutia extraction algorithms rely on time consuming and fragile image enhancement steps in order to work robustly. We propose a new approach, combining enhancement and extraction into a Convolutional Neural Network (CNN). This network is trained from scratch using synthetic fingerprints. To bridge the gap between synthetic and real fingerprints, refinements are used. Here, an approach based on Generative Adversarial Networks (GANs) is used to generate fingerprints suited for training such a network and improving its matching score on real fingerprints.

Holograms are security features applied to security documents like banknotes, passports, and ID cards in order to protect them from counterfeiting. Checking the authenticity of holograms is an important but difficult task, as holograms comprise different appearances for varying observation and/or illumination directions. Multi-view and photometric image acquisition and analysis procedures have been proposed to capture that variable appearance. We have developed a portable ring-light illumination module used to acquire photometric image stacks of holograms with mobile devices. By the application of Convolutional Neural Networks (CNN), we developed a vector representation that captures the essential appearance properties of hologram types in only a few values extracted from the photometric hologram stack. We present results based on Euro banknote holograms of genuine and counterfeited Euro banknotes. When compared to a model-based hologram descriptor, we show that our new learned CNN representation enables hologram authentication on the basis of our mobile acquisition method more reliably.

We present our work in progress in the direction of generating realistic, challenging and diverse test data in order to evaluate a banknote checking system. Test image generation can be divided into, firstly, the design and composition of highly diverse patterns to form the digital design of a banknote, secondly, the application of different variations which are caused by ink, paper and physically printing the note and, thirdly, the simulation of the subsequent image acquisition, which includes variations due to banknote transport, illumination, camera optics and electronics. These simulations are based on and compared to scans of demo banknotes. We present initial results in simulation of banknote paper, printing and image acquisition. We demonstrate the approach in an initial investigation on synthetic defect generation and its impact on banknote checking results. Algorithmic validation of checking routines is also demonstrated. First conclusions are drawn and further work is discussed.

This chapter provides an overview of acquisition methods, specific challenges, and solutions for image-based metal surface inspection. We discuss illumination and recording methods for a wide range of tasks such as inspection of rail surfaces, weld quality assessment, inspection of holograms on metallic foils, and reconstruction of coins. Algorithms exploiting the properties of the acquired image data are presented in detail. Image acquisition, tracking, and segmentation of weld beads in harsh environments are presented. Reference and non-reference weld quality assessments are discussed. Detection of surface disruptions as well as surface classification for images of rails from a mobile acquisition platform is demonstrated. Multi-view approaches are explored for the inspection of optically variable security elements on metallic foil and light-field data processing is applied to the inspection of coins.

Diffractive Optically Variable Image Devices (DOVIDs), sometimes loosely referred to as holograms, are popular security features for protecting banknotes, ID cards, or other security documents. Inspection, authentication, as well as forensic analysis of these security features are still demanding tasks requiring special hardware tools and expert knowledge. Existing equipment for such analyses is based either on a microscopic analysis of the grating structure or a point-wise projection and recording of the diffraction patterns. We investigated approaches for an examination of DOVID security features based on sampling the Bidirectional Reflectance Distribution Function (BRDF) of DOVIDs using photometric stereo- and light-field-based methods. Our approach is demonstrated on the practical task of automated discrimination between genuine and counterfeited DOVIDs on banknotes. For this purpose, we propose a tailored feature descriptor which is robust against several expected sources of inaccuracy but still specific enough for the given task. The suggested approach is analyzed from both theoretical as well as practical viewpoints and w.r.t. analysis based on photometric stereo and light fields. We show that especially the photometric method provides a reliable and robust tool for revealing DOVID behavior and authenticity.

Diffractive optically variable image devices (DOVIDs) are popular security features used to protect security documents such as banknotes, ID cards, passports, etc. Nevertheless, checking authenticity of these security features on both user as well as forensic level still remains a challenging task, requiring sophisticated hardware tools and expert knowledge. Based on a photometric acquisition setup comprised of 32 illumination sources from different directions and a recently proposed descriptor capturing the illumination dependent behavior, we investigate the information content, illumination pattern shape and clustering properties of the descriptor. We studied shape and discriminative power of reduced illumination configurations for the task of discrimination applied to DOVIDs using a sample of Euro banknotes.

Diffractive Optically Variable Image Devices (DOVIDs) are popular security features used to protect security documents such as banknotes, ID cards, passports, etc. Checking authenticity of these security features on both user as well as forensic level remains a challenging task, requiring sophisticated hardware tools and expert knowledge. Recently, we proposed a technique exploiting a large-scale photometric behavior of DOVIDs in order to discriminate denominations and detect counterfeits. Here we investigate invariance properties of the proposed method and demonstrate its robustness against various common perturbations, which may have negative impact on the acquisition quality in practice. Presented results show a great potential of this approach primarily for security and forensic purposes, but also for other applications, where automated inspection of DOVIDs is of interest.