Accurate waste classification is essential for effective recycling. For household waste, changing recycling policies and the introduction of new materials in consumer products continually reshape sorting categories across regions and over time. These changes challenge existing sensor-based sorting systems trained for a specific classification task, and restoring accuracy with fully supervised retraining is costly because it requires new labeled data and full model updates. We investigate whether foundation models can deliver accurate image-based waste classification with no or few labeled examples. We analyze existing methods such as multimodal large language models (MLLMs), vision-language models (VLMs), Vision Transformers (ViTs), and a baseline CNN across four datasets, including a new food vs. non-food packaging dataset, with varying numbers of labeled examples. Further, we propose adaptations to the existing approaches, such as chain-of-thought prompting for MLLMs, and ensemble prompting and a Tip-Adapter for VLMs. We show that MLLMs perform well on zero-shot classification and larger models like GPT-4o further improve with few examples, but at infeasible computational cost for industry-scale inference. For a comparably faster way of zero-shot classification, we show that VLMs yield an accuracy of 90.4% on TrashNet, by contrast, a CNN typically needs a few hundred labeled images to achieve similar performance. Using the training-free Tip-Adapter with only 10 labeled example images per class lifts macro-F1 by 8.1 points over the zero-shot VLM baseline. Overall, we propose a guideline for language-driven, training-free methods for waste classification.

Sensor-based sorting systems enable the physical separation of a material stream into two fractions. The sorting decision is based on the image data evaluation of the sensors used and is carried out using actuators. Various process parameters must be set depending on the properties of the material stream, the dimensioning of the system, and the required sorting accuracy. However, continuous verification and re-adjustment are necessary due to changing requirements and material stream compositions. In this paper, we introduce an approach for optimizing, recurrently monitoring and adjusting the process parameters of a sensor-based sorting system. Based on Bayesian Optimization, Gaussian process regression models are used as surrogate models to achieve specific requirements for system behavior with the uncertainties contained therein. This method minimizes the number of necessary experiments while simultaneously considering two possible optimization targets based on the requirements for both material output streams. In addition, uncertainties are considered during determining sorting accuracies in the model calculation. We evaluated the method with three example process parameters.

This paper investigates how the noise characteristics of synthetically generated camera images correspond to those of a real camera. We determine the photon transfer curve from a set of rendered images of a static scene. Furthermore, we present a method to identify the regions with high temporal noise, i.e., rendering noise, in synthetically generated data from a single rendered image. Finally, we present a strategy on how a parameterization of the rendering can be achieved that minimizes the noise while also minimizing the rendering runtime.

This article introduces a novel approach for automated visual inspection through inverse rendering of a digital twin, enhancing the detection of material defects. Our method generates reference images via ray tracing, simulating the actual appearance of products while accommodating inherent variations in manufacturing. By employing inverse rendering and automatic differentiation, we adjust the digital twin's parameters to align the simulated images with those captured by the inspection system. This optimization process effectively minimizes discrepancies, allowing us to compute difference images that primarily highlight material defects. Our experiments validate the approach, demonstrating significant improvements in signal-to-noise ratios and defect visibility compared to conventional methods. The potential applications of our approach are particularly notable in the context of Industry 4.0, especially in lot size 1 scenarios, where high product variability complicates traditional inspection processes. By leveraging the availability of CAD models, our method offers a practical solution for adaptive inspection frameworks that can efficiently handle diverse product designs.

Sensor-based sorting describes a family of systems that enable the removal of individual objects from a material stream. The technology is widely used in various industries such as agriculture, food, mining, and recycling. Examples of sorting tasks include the removal of fungus-infested grains, the enrichment of copper content in copper mining or the sorting of plastic waste according to the type of plastic. Sorting decisions are made based on information acquired by one or more sensors. A particular strength of the technology is the flexibility in sorting decisions, which is achieved by using various sensors and programming the data analysis. However, a comprehensive understanding of the process is necessary for the development of new sorting systems that can address previously unresolved tasks. This survey is aimed at innovative researchers and practitioners who are unfamiliar with sensor-based sorting or have only encountered certain aspects of the overall process. The references provided serve as starting points for further exploration of specific topics.

Sensor-based monitoring of construction and demolition waste (CDW) streams plays an important role in recycling (RC). Extracted knowledge about the composition of a material stream helps identifying RC paths, optimizing processing plants and form the basis for sorting. To enable economical use, it is necessary to ensure robust detection of individual objects even with high material throughput. Conventional algorithms struggle with resulting high occupancy densities and object overlap, making deep learning object detection methods more promising. In this study, different deep learning architectures for object detection (Region-based CNN/Region-based Convolutional Neural Network (Faster R-CNN), You only look once (YOLOv3), Single Shot MultiBox Detector (SSD)) are investigated with respect to their suitability for CDW characterization. A mixture of brick and sand-lime brick is considered as an exemplary waste stream. Particular attention is paid to detection performance with increasing occupancy density and particle overlap. A method for the generation of synthetic training images is presented, which avoids time-consuming manual labelling. By testing the models trained on synthetic data on real images, the success of the method is demonstrated. Requirements for synthetic training data composition, potential improvements and simplifications of different architecture approaches are discussed based on the characteristic of the detection task. In addition, the required inference time of the presented models is investigated to ensure their suitability for use under real-time conditions.

The optical bulk material sorting is a key technology on our way toward a circular economy and efficient recycling. However, controlling the sorting accuracy has so far been severely limited, as the achievable accuracy of conventional sorters is strongly determined by the mass flow and the mixing ratio of the incoming particle stream. To enable closed-loop control, in the previous work, we introduced a modification to the sorter design, in which controlled fractions of the already sorted mass flows are returned to the inlet of the sorter. In this article, we now propose two open-loop and two closed-loop feedback (CLF) stochastic model predictive controllers (MPCs) for the control of sorting systems with recirculation operating under dynamically changing conditions. In addition, we propose to integrate a desired minimum accuracy as a chance constraint into our controllers' stochastic formulation. Our evaluations using a coupled discrete element-computational fluid dynamics (DEM-CFD) simulation show that our controllers considerably improve on the system without recirculation and outperform the previously known controllers. Furthermore, we found that they are able to maintain a predefined minimum quality even in highly dynamic scenarios, making the approach highly valuable for tasks where achieving a certain quality at any point in time is crucial.

Optical sorters separate particles of different classes by first detecting them while they are transported, e.g., on a conveyor belt, and subsequently bursting out particles of undesired classes using compressed air nozzles. Currently, the most promising results are achieved by predictive tracking, a multitarget tracking approach based on extracted midpoints from area-scan camera images that analyzes the particles’ motion and activates the nozzles accordingly. However, predictive tracking requires expert knowledge for setup and preceding object detection. Moreover, particle shapes are only considered implicitly, and the need to solve an association problem rises the computational complexity of the algorithm. In this paper, we present GridSort, an image-based approach that forecasts the scene at the nozzle array using a convolutional long short-term memory neural network and subsequently extracts nozzle activations, thus circumventing the aforementioned weaknesses. We show how GridSort can be trained in an unsupervised fashion and evaluate it using a coupled discrete element–computational fluid dynamics simulation of an optical sorter. We compare our method with predictive tracking in terms of sorting accuracy and demonstrate that it is an easy-to-apply alternative while achieving state-of-the-art results.

Sensor-based sorting offers cutting-edge solutions for separating granular materials. The line-scanning sensors currently in use in such systems only produce a single observation of each object and no data on its movement. According to recent studies, using an area-scan camera has the potential to reduce both characterization and separation error in a sorting process. A predictive tracking approach based on Kalman filters makes it possible to estimate the followed paths and parametrize a unique motion model for each object using a multiobject tracking system. While earlier studies concentrated on physically-motivated motion models, it has been demonstrated that novel machine learning techniques produce predictions that are more accurate. In this paper, we describe the creation of a predictive tracking system based on neural networks. The new algorithm is applied to an experimental sorting system and to a numerical model of the sorter. Although the new approach does not yet fully reach the achieved sorting quality of the existing approaches, it allows the use of the general method without requiring expert knowledge or a fundamental understanding of the parameterization of the particle motion model.

Polycyclic aromatic hydrocarbons (PAH) containing tar-mixtures pose a challenge for recycling road rubble, as the tar containing elements have to be extracted and decontaminated for recycling. In this preliminary study, tar, bitumen and minerals are discriminated using a combination of color (RGB) and Hyperspectral Short Wave Infrared (SWIR) cameras. Further, the use of an autoencoder for detecting minerals embedded inside tar- and bitumen mixtures is proposed. Features are extracted from the spectra of the SWIR camera and the texture of the RGB images. For classification, linear discriminant analysis combined with a k-nearest neighbor classification is used. First results show a reliable detection of minerals and positive signs for separability of tar and bitumen. This work is a foundation for developing a sensor-based sorting system for physical separation of tar contaminated samples in road rubble.

A DEM-CFD (discrete element method - computational fluid dynamics) model of an optical belt sorter was extensively compared with experiments of a laboratory-scale sorter to assess the model's accuracy. Brick and sand-lime brick were considered as materials. First, the transport characteristics on the conveyor belt, involving mass flow, lateral particle distribution and proximity, were compared. Second, sorting results were benchmarked for varying mixture proportions at differing mass flows. It was found that the numerical model is able to reproduce the experimental results with high accuracy.

For many technical applications, obtaining sensory information about objects, a scene or the environment is crucial.These include, for example, determining product quality in quality assurance and sensor-based sorting, sensing the environment for robotics and automated vehicles, and many other tasks in measurement and automation technology.In all of these applications, machine vision systems have key advantages over other sensor principles and over the inspection by humans: The actual observation process-image acquisition-is contact-free, the data have a high information content due to their multi-dimensional nature, and a variety of image acquisition methods can be used to capture very different properties of the scene with high informative value.What is outstanding about machine vision, however, is that it emulates the most important human sense-the visual sense-so that many image processing procedures can be understood relatively easily by humans.On the other hand, technical image acquisition is not bound to the limitations of the human sense of sight (e. g., spectral sensitivity, temporal response, temporal and spatial resolution, reproducibility, objectivity, fatigue).Cameras and the images they capture also play an increasing role in daily life, which is immediately apparent from the omnipresence of smartphones with (now often multiple) cameras.This is accompanied by a high level of maturity in sensor technology and image data processing, which in turn benefits the technical applications of image processing.In machine vision systems, components of various disciplines, including optics, lighting technology, sensor

Optical sorting is a key technology for the circular economy and is widely applied in the food, mineral, and recycling industries. Despite its widespread use, one typically resorts to expensive means of adjusting the accuracy, e.g., by reducing the mass flow or changing mechanical or software parameters, which typically requires manual tuning in a lengthy, iterative process. To circumvent these drawbacks, we propose a new layout for optical sorters along with a controller that allows re-feeding of controlled fractions of the sorted mass flows. To this end, we build a dynamic model of the sorter, analyze its static behavior, and show how material recirculation affects the sorting accuracy. Furthermore, we build a model predictive controller (MPC) employing the model and evaluate the closed-loop sorting system using a coupled discrete element–computational fluid dynamics (DEM–CFD) simulation, demonstrating improved accuracy.

With the rising trend of consumers being offered by start-up companies portable devices and applications for checking quality of purchased products, it appears of paramount importance to assess the reliability of miniaturized sensors embedded in such devices. Here, eight sensors were assessed for food fraud applications in skimmed milk powder. The performance was evaluated with dry- and wet-blended powders mimicking adulterated materials by addition of either ammonium sulfate, semicarbazide, or cornstarch in the range 0.5–10% of profit. The quality of the spectra was assessed for an adequate identification of the outliers prior to a deep assessment of performance for both non-targeted (soft independent modelling of class analogy, SIMCA) and targeted analyses (partial least square regression with orthogonal signal correction, OPLS). Here, we show that the sensors have generally difficulties in detecting adulterants at ca. 5% supplementation, and often fail in achieving adequate specificity and detection capability. This is a concern as they may mislead future users, particularly consumers, if they are intended to be developed for handheld devices available publicly in smartphone-based applications.

The visual quality inspection of test objects having a complex geometry is a challenging task for automated artificial vision systems. Even for systems where the illumination and image acquisition setups are specifically tailored with respect to the properties of the test object, captured images often show unwanted signal components, e.g., surface reflections, which complicate the detection of present material defects. One way to mitigate this problem is to have an expert define image regions by hand which are excluded from the automated defect detection. Besides being a time-consuming procedure, this also results in the system being blind at the respective regions. Another approach is based on acquiring image value statistics (e.g., mean value and standard deviation) for every pixel of an image series captured from a set of defect-free test objects. This information can then be exploited during the inspection process by comparing image values with respect to the previously calculated statistics. Pixels whose image values lie outside the distribution for the defect-free case might indicate a material defect. Unfortunately, the calculated statistics are invalidated as soon as further preprocessing steps like smoothing or edge detection are applied. The statistics would have to be recalculated by applying the respective preprocessing steps to the images of the defect-free test objects. To resolve this drawback, this contribution presents a novel approach capable of adequately updating the calculated statistics with respect to the chain of required image processing steps. This is achieved by interpreting the statistics as uncertainties and by propagating them through the single processing steps via Gaussian uncertainty propagation. The required gradients are obtained via automated differentiation of the image processing steps. The effectiveness of the proposed approach is demonstrated by means of empirical experiments.

Traditionally, there is a trade-off between the numerical aperture and field of view for a microscope objective. Diffractive lens arrays (DLAs) with overlapping apertures are used to overcome such a problem. A spot array with an NA up to 0.83 and a pitch of 75 µm is produced by the proposed DLA at a wavelength of 488 nm. By measurement of the fluorescence beads, the DLA-based confocal setup shows the capability of high-resolution measurement over an area of 3 m m × 3 m m with a 2.5 × 0.07 N A objective. Further, the proposed fluorescence microscope is insensitive to optical aberrations, which has been demonstrated by imaging with a simple doublet lens.

In this investigation, a DEM-CFD model of an optical belt sorter is modified to become adaptive to varying belt speeds. For that, the positions and orientations of the nozzle bar and collecting containers are rearranged. Also, the duration of nozzle activation and optimal position of particle ejection are adjusted. For the derivation of optimal velocity-dependent parameters, a two-dimensional model is derived and optimized as a pre-processing step. The derived parameters are applied to the three-dimensional DEM-CFD model. Two optically distinguish-able types of demolition waste materials are considered. All conveyor belt velocities are investigated with instantaneously and lagged activated nozzles, which represent fast and realistic triggered nozzle activations. The application of optimized sorting setups shows promising sorting results for a broad range of conveyor belt ve-locities. The obtained results are discussed in terms of their feasibility in being applied to real optical belt sorters.

Multispectral cameras are gaining popularity in the field of smart farming and plant phenotyping. They are more cost-effective than hyperspectral cameras and frame-based imaging allows easier operation and processing, while at the same time the limited spectral resolution still allows the retrieval of relevant information about the plant status. Typically, multispectral cameras are available with equidistant channels spanning a defined spectral range. We propose a design approach based on Bayesian optimization to define problem-specific spectral channels for multispectral cameras in the plant phenotyping domain. Compared to established wavelength selection algorithms, our approach considers physical constraints of optical filters such as feasible filter function shape and width. The filter functions are optimized and tested to predict plant pigment concentration and Equivalent Water Thickness of simulated spectra generated with the PROSPECT-D leaf radiative transfer model. Problem-specific multispectral camera design could potentially enhance prediction performance of automated plant status monitoring.

In optical sorting of bulk material, the composition of particles may frequently change. State-of-the-art sorting approaches rely on tuning physical models of the particle motion. The aim of this work is to increase the prediction accuracy in complex fast-changing sorting scenarios with data-driven approaches. In this article, we propose two neural network (NN) experts for accurate prediction of a priori known particle types. To handle the large variety of particle types that can occur in real-world sorting scenarios, we introduce a simple but effective mixture of experts’ approach that combines NNs with hand-crafted motion models. Our new method not only improves the prediction accuracy for bulk material consisting of many particle classes, but also proves to be very adaptive and robust to new particle types.