While transformers have surpassed convolutional neural networks (CNNs) in various computer vision tasks, microelectronics defect detection still largely relies on CNNs. We hypothesize that this gap is due to the fact that a) transformers have an increased need for data and b) (labelled) image generation procedures for microelectronics are costly, and data is therefore sparse. Whereas in other domains, pre-training on large natural image datasets can mitigate this problem, in microelectronics transfer learning is hindered due to the dissimilarity of domain data and natural images. We address this challenge through self pre-training, where models are pre-trained directly on the target dataset, rather than another dataset. We propose a resource-efficient vision transformer (ViT) pre-training framework for defect detection in microelectronics based on masked autoencoders (MAE). We perform pre-training and defect detection using a dataset of less than 10,000 scanning acoustic microscopy (SAM) images. Our experimental results show that our approach leads to substantial performance gains compared to a) supervised ViT, b) ViT pre-trained on natural image datasets, and c) state-of-the-art CNN-based defect detection models used in microelectronics. Additionally, interpretability analysis reveals that our self pre-trained models attend to defect-relevant features such as cracks in the solder material, while baseline models often attend to spurious patterns. This shows that our approach yields defect-specific feature representations, resulting in more interpretable and generalizable transformer models for this data-sparse domain.

This paper presents an approach to abstract capabilities of hardware components, such as a robot, and to store the necessary data for accessing that capability in Asset Administration Shells (AAS). It uses an ontology based on the Capability, Skills, and Services (CSS) model, enhanced with the possibility to describe physical relations between components. Capabilities describe abstract abilities realized through technology-neutral skills, exemplified in Docker containers using a standardized interface to communicate. The case study, bin picking, illustrates the system's application and requirements. Two user types are identified: the hardware manufacturers who can abstract and store component capabilities in an AAS, and the user who can load these shells in order to orchestrate the process program. This paper details the system's design, implementation, and evaluation, demonstrating its functionality through the case study.

Predictive maintenance and reliability analysis have notoriously imperfect and unbalanced data because of few failure tests. This challenge limits the capability of supervised learning. Data-driven methods, especially deep learning, show great potential in capturing such complex relationships in many domains. Those methods rely heavily on the amount of data to extract meaningful representations. In contrast, data-poor domains depend on the use of simple architectures with a limited amount of trainable parameters. In this work, we show how pretraining with supervised contrastive learning can lead to better representations in data-poor domains such as the lifetime estimation on the example of solder joint. With such pre-trained models we are able to outperform end-to-end learning methods. Furthermore, we show that the learned representations improve classification tasks as well as rearession tasks.

Automation mechanisms are increasingly established in the field of visual quality control. UAVs can be used for particularly large components, such as those used in aircraft or ship production, but also for critical infrastructures. This paper concentrates on the problem of visual quality control in the field of perspective-dependent route planning. It is shown how the requirements for such a system can be implemented and elaborated. Furthermore we investigate how sensor positions can be calculated offline, based on optical- and geometrical requirements and how a trajectory can be planned which contains the found sensor positions for each given area on the component. It is shown how the systems architecture can be designed in order to be able to adapt it to different requirements for the planning of sensor positions and trajectory. The implementation was tested in a simulation environment, evaluated using a benchmark data set and it was shown how above-average results can be achieved on this data set.

To increase the utility of semantic industrial information models we propose a methodology to incorporate extracted operator knowledge, which we assume to be present in the form of rules, in knowledge graphs. To this end, we present multiple modelling patterns that can be combined depending on the required complexity. Aiming to combine information models with learning systems we contemplate desired behaviours of embeddings from a predictive quality perspective and provide a suited embedding methodology. This methodology is evaluated on a real world dataset of a fused deposition modelling process.

In socio-technical systems, such as manufacturing processes, human operators are often entrusted with cognitive tasks that rely on tacit knowledge. Extracting the operators' tacit knowledge is beneficial to facilitate knowledge transfer and enable semantic machine learning. We improve upon an existing methodology that relies on operators' insights into influences on their decision making processes to extract tacit knowledge. By introducing a data-based weighting of the operators' information, we are able to control varying degrees of worker reliability and other individual biases, increasing the quality of the aggregated knowledge. We evaluate several methods to weigh and aggregate knowledge on a real-world dataset collected in the domain of fused deposition modelling (FDM) showing an improvement of 34% over a previously published baseline applied to our data. Applicability of the approach in the same domain is demonstrated by a case study, where the aggregated knowledge is utilised to shorten the time required for parametrisation.

Scanning Acoustic Microscopy (SAM) measurements of thermally aged LED solder joints are translated to the thermal properties of the sample as characterized by Transient Thermal Analyses (TTA) using Artificial Neural Networks in order to improve the comparability of these two measurement methods. The dataset of 1800 samples with five solder pastes and nine LED types is used to study the inter- and extrapolation abilities of the trained models with respect to differences in solders and component structure. The effect of solder joint degradation due to thermal shock cycles on the ability of the model to translate is also studied with four different aging states. The architecture used is a combination of convolutional layers with max pooling and fully connected layers.

In this work, we describe a ROS-based method for whole-body control (WBC) of mobile manipulators in the context of safe human-robot interaction. Our method is based on cyclic quadratic programming (QP) with a set of simultaneously active tasks that define constraints. The importance of different tasks is captured through priorities and weights. Robot behavior can be changed at run-time by re-configuring the active tasks through ROS interfaces. We evaluate the suitability of our method for safe human-robot collaboration in a Gazebo simulation. We show that our method lets the mobile manipulator perform evasive motions while staying consistent with other tasks if possible. At the same time, self-collisions and static obstacles are avoided. If a given safety threshold is crossed, the robot comes to a safe stop. Operation continues once the distance is high enough again.

The industry of tomorrow is changing from central hierarchical industrial and robot controls to distributed controls on the industrial shop floor. These fundamental changes in network structure make it possible to implement technologies such as Plug & Produce. In other words, to integrate, change and remove devices without much effort at runtime. In order to achieve this goal, a uniform architecture with defined interfaces is necessary to establish real-time communication between the varying devices. Therefore, we propose an approach to use the combination of OPC UA and TSN to automatically configure real-time capable communication paths between robots and other cyber-physical components and execute real-time critical tasks in the distributed control system.

The inspection of large structures is increasingly carried out with the help of Unmanned Aerial Vehicles (UAVs). When navigating relative to the structure, multiple data sources can be used to determine the position of the UAV. Examples include track data from an installed camera and sensor data from the orientation sensors of the UAV. This paper deals with the fusion of this data and its use for navigation alongside the structure. For the sensor fusion, a concept is developed using a Kalman filter and evaluated simulatively in a prototype. The calculated position data are also fed into a vector flight control system, which dynamically calculates and flies a trajectory along the component using the potential field method. This is done taking into account obstacles detected by the onboard sensors of the UAV. The established concept is then implemented with the Robot Operating System (ROS) and evaluated simulatively.

Industrial manufacturing is currently undergoing a transformation from mass production with inflexible production systems to individual production with adaptable cells. In order to ensure this adaptability of these systems, technologies such as plug & produce are needed, to integrate, modify and remove devices at runtime. Therefor an exact description of the system, the products and the capabilities / skills of the devices is essential as well as a network for communication between the devices. Deterministic data transmission is particularly important for distributed control systems. We propose an architecture for plug & produce mechanisms with hard real-time capable communication paths between the cyber-physical components using OPC UA PubSub over TSN and the ability to load and execute real-time critical tasks at runtime.

Applying unmanned aerial vehicles (UAV) has benefits for many different use-cases. Existing implementations of ground control stations (GCS) to manage UAVs in such scenarios already provide some support for the operation of multi-unit systems, i.e., ensembles. However, since they are usually designed for the operation of only one copter at once, this is often not sufficient to react quickly in dangerous situations, e.g., search and rescue scenarios. To address this problem, we propose an approach for easy observation and control of complete autonomous UAV ensembles: The Intention of our approach is to greatly reduce the number of personnel required for the operation of an UAV ensemble. Thereby, we generate the possibility for rapid intervention in potentially dangerous situations in order to prevent damage to the UAVs and the environment. In this paper, we present a software architecture for this safety-critical multi UAV ground control station including a fully implemented prototype which we also tested in a realistic environment.

The Robot Operating System (ROS) offers developers a large number of ready-made packages for developing robot programs. The multitude of packages and the different interfaces or adapters is also the reason why ROS projects often tend to become confusing. Concepts of model-driven software development using a domain-specific modeling language could counteract this and at the same time speed up the development process of such projects. This is investigated in this paper by transferring the core concepts from ROS 2 into a graphical programming interface. Elements of established graphical programming tools are compared and approaches from modeling languages such as UML are used to create a novel approach for graphical development of ROS projects. The resulting interface is evaluated through the development of a project built on ROS, and the approach shows promise towards facilitating work with the Robot Operating System.

Through the mechanisms of the Semantic Web, it is possible not only to describe web content syntactically but also to relate it semantically. The properties and capabilities of hardware, instead, are hidden in documents, code documentations, repository descriptions, etc. This paper presents a methodology and architecture that can be used to describe and relate the properties and capabilities of hardware. The decentralized storage of the descriptions on a hardware adapter allows the information to be evaluated at runtime. For domain-specific applications a Model-Domain-Domainmodel Architecture (MDDM) is presented so that code can also be executed at runtime using these hardware descriptions. The architecture is presented using a home automation system with single-board computers and microcontrollers, in which sensors and actuators can be exchanged and integrated.

Carbon fiber reinforced polymers (CFRP) are light yet strong composite materials designed to reduce the weight of aerospace or automotive components – contributing to reduced emissions. Resin transfer molding (RTM) is a manufacturing process for CFRP that can be scaled up to industrial-sized production. It is prone to errors such as voids or dry spots, resulting in high rejection rates and costs. At runtime, only limited in-process information can be made available for diagnostic insight via a grid of pressure sensors. We propose FlowFrontNet, a deep learning approach to enhance the in-situ process perspective by learning a mapping from sensors to flow front “images” (using upscaling layers), to capture spatial irregularities in the flow front to predict dry spots (using convolutional layers). On simulated data of 6 million single time steps resulting from 36k injection processes, we achieve a time step accuracy of 91.7 38 × 30 sensor grid 1 cm sensor distance in x- and y-direction. On a sensor grid of 10 × 8 , with a sensor distance of 4 cm, we achieve 83.7

In the field of industry 4.0 and smart factories of the future, dynamic teams of robots will play an important role in the manufacturing of custom-tailored products with small lot sizes. Especially the planning for such multi-functional robot cells forms a key challenge, finding appropriate tasks for cooperating robots to perform efficient production. In this paper, we present two approaches for automated planning of assembly programs for multiple robots that separate between domain and automation level. While one uses a decomposition approach with plan-space planning and explicit decomposition rules, the other considers heuristic searching in state space. Both approaches are evaluated with different planning problems in the blocks world domain (i.e., LEGO® DUPLO®). The evaluation shows that both approaches outperform the results achieved with classical planning based on state-space search such as \(A^*\). Furthermore, a concept for post-processing programs to optimize execution time by dependency-based interleaving of robot tasks is introduced that fully integrates into both planning approaches.

The programming of cooperating teams of robots in automation and in particular for assembling small batch sizes is a tedious task. Apart from that, the use of AI-based planning for multiple robots is computationally expensive. Using state space techniques, the state space increases dramatically for numerous possibilities in an indiscrete, continuous world. Hence, in this work we present a two-layer planning algorithm for multi-robot assembly which automatically partitions the state space in order to reduce complexity and to speed up planning time. We depict formal considerations about planning complexity and show how and why the state space size and, thus, planning time is strongly lower compared to common approaches.

Unmanned aerial vehicles (UAV) can support various scenarios, e.g., serve as measuring instruments for climate research, help rescue forces in disaster scenarios or autonomously inspect critical infrastructure. To accomplish their respective tasks, UAVs are often equipped with different sensors and in some scenarios used in ensembles to work together cooperatively. In this paper, we present the Block Definition Language (BDL) for modeling and executing UAV missions. The BDL supports both the use of exchangeable sensors and appropriate coordination mechanisms for robot ensembles. We demonstrate our plugin mechanism in a proof of concept using the BDL in conjunction with the robotic middleware ROS for hardware control and robot synchronization.