We introduce a unified, end-to-end framework that seamlessly integrates object detection and pose estimation with a versatile onboarding process. Our pipeline begins with an onboarding stage that generates object representations from either traditional 3D CAD models or, in their absence, by rapidly reconstructing a high-fidelity neural representation (NeRF) from multi-view images. Given a test image, our system first employs the CNOS detector to localize target objects. For each detection, our novel pose estimation module, OPFormer, infers the precise 6D pose. The core of OPFormer is a transformer-based architecture that leverages a foundation model for robust feature extraction. It uniquely learns a comprehensive object representation by jointly encoding multiple template views and enriches these features with explicit 3D geometric priors using Normalized Object Coordinate Space (NOCS). A decoder then establishes robust 2D-3D correspondences to determine the final pose. Evaluated on the challenging BOP benchmarks, our integrated system demonstrates a strong balance between accuracy and efficiency, showcasing its practical applicability in both model-based and model-free scenarios.

3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based benchmark that evaluates 6D pose estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using robots.

We present a novel and complete vision-based pipeline for 6DoF object pose estimation in challenging industrial bin-picking scenarios, characterized by significant clutter, occlusions, and reflective surfaces. Our approach addresses the limitations of both computationally expensive fine-tuning methods and the current immaturity of foundation models in handling such complex environments. The key contribution lies in a balanced approach leveraging synthetic data augmentation and a streamlined architecture to achieve robust performance without extensive per-object optimization. The pipeline integrates existing state-of-the-art object detection, coarse pose estimation, and a render-and-compare refinement strategy, enabling accurate pose estimation from monocular images. We introduce a new benchmark on the recently released dataset, establishing a baseline for future research. Unlike existing industrial approaches, our system minimizes reliance on multi-sensor configurations, offering a cost-effective and easily deployable solution. We demonstrate the impact of error propagation across a complete pipeline using datasets that mirror real-world industrial conditions, in contrast to commonly used but less representative datasets.

Current generalizable object pose estimators, i.e., approaches that do not need to be trained per object, rely on accurate 3D models. Predominantly, CAD models are used, which can be hard to obtain in practice. At the same time, it is often possible to acquire images of an object. Naturally, this leads to the question of whether 3D models reconstructed from images are sufficient to facilitate accurate object pose estimation. We aim to answer this question by proposing a novel benchmark for measuring the impact of 3D reconstruction quality on pose estimation accuracy. Our benchmark provides calibrated images suitable for reconstruction and registered with the test images of the YCB-V dataset for pose evaluation under the BOP benchmark format. Detailed experiments with multiple state-of-the-art 3D reconstruction and object pose estimation approaches show that the geometry produced by modern reconstruction methods is often sufficient for accurate pose estimation. Our experiments lead to interesting observations: (1) Standard metrics for measuring 3D reconstruction quality are not necessarily indicative of pose estimation accuracy, which shows the need for dedicated benchmarks such as ours. (2) Classical, non-learning-based approaches can perform on par with modern learning-based reconstruction techniques and can even offer a better reconstruction time-pose accuracy tradeoff. (3) There is still a sizable gap between performance with reconstructed and with CAD models. To foster research on closing this gap, the benchmark is made available at https://github.com/VarunBurde/reconstruction_pose_benchmark.

Object pose estimation is a prominent task in computer vision. The object pose gives the orientation and translation of the object in real-world space, which allows various applications such as manipulation, augmented reality, etc. Various objects exhibit different properties with light, such as reflections, absorption, etc. This makes it challenging to understand the object's structure in RGB and depth channels. Recent research has been moving toward learning-based methods, which provide a more flexible and generalizable approach to object pose estimation utilizing deep learning. One such approach is the render-and-compare method, which renders the object from multiple views and compares it against the given 2D image, which often requires an object representation in the form of a CAD model. We reason that the synthetic texture of the CAD model may not be ideal for rendering and comparing operations. We showed that if the object is represented as an implicit (neural) representation in the form of Neural Radiance Field (NeRF), it exhibits a more realistic rendering of the actual scene and retains the crucial spatial features, which makes the comparison more versatile. We evaluated our NeRF implementation of the render-and-compare method on transparent datasets and found that it surpassed the current state-of-the-art results.

Using robots and UAVs(unmanned aerial vehicles) for surveying transmission lines and towers is standard practice. Applications such as painting and NDT (non-destructive testing) require the UAV to be in close proximity to the surface of the structure. Typically, UAV systems consist of GPS(global positioning system), cameras, and high computing devices mounted to perform SLAM to locate and navigate. Most vision-based SLAM (simultaneous localization and mapping) algorithms use spatial features to localize and map. Structures such as transmission towers are highly symmetric, which may make the SLAM system difficult to localize. Even with accurate localization, it is hard to calculate the trajectory which covers the surface area with proximity, which is another problem to solve to automate the process. The paper proposes a CAD (computer-aided design) model-based navigation method for UAVs around transmission towers that can be deployed in low-power devices. The CAD model provides crucial information, which can be just enough to plan the path/motion around the transmission tower. The algorithm pipeline starts with finding the flat surfaces of the CAD model and sampling them. Using those sampled points, estimate the possible site around the tower where the drone could fly. After that, sampled points are filtered to find the possible trajectory to cover the surface.

This paper deals with motion modeling of a 5-axis industrial Delta robot. The robot has extra rotational two degrees-of-freedom (DoF) realized with a wrist arm driven through two co-axial telescopic shafts as compared to the basic 3-DoF Delta robot. The kinematic model is derived with fully symbolic Jacobian matrices. Using the derived Jacobians, a novel simplified dynamic model is proposed based on the virtual work principle and the trajectory dependent artificial mass distribution. As compared to the existing literature, the proposed dynamic model does not require Lagrangian multiplier calculation or recursive and parallel computing so that it provides advantage for model-based control design. Also a linear regression model is provided to identify the dynamic parameters. The presented models are suitable to be employed for basic Delta and the extended Delta robots with parallel telescopic shafts as well. The derived models are verified through a Simulink model where the 3D CAD files of robot bodies having the information of real dimensions, masses and moments of inertia are used. The adequate agreement of the proposed dynamic model with the simulation results is illustrated via performing three different generated trajectory profiles. We also demonstrate the better accuracy of the proposed dynamic model as compared to a simplified and widely employed model for basic 3-DoF Delta robot. The simulation model is shared online to serve as a research and test platform for performing tasks such as motion planning, model prototyping, and control design.

For deploying any robotic solution, the most cumbersome yet essential task remains calibration. Almost every robotic system uses a camera to extract information to produce some output. In the case of the pick-and-place system, camera information is used to detect objects and estimate their position in real-world space. The accuracy of tasks like sorting and stacking the objects in the box depends on their estimated pose in the space, especially when they need to be arranged in a confined space. We propose the automatic calibration of the workspace using distinct markers. Our experimentation was carried out with loose packets of recycled paper, which consists of small parts inside. We evaluate our results on the basis of translation errors of the estimated centroid position of objects with respect to the position given by the calibrated 3D camera. In conclusion, we also assess the typical pick-and-place performance on the success rate of stacking the packets in a small box.

The exponential proliferation of big data and computation-intensive tasks, such as Artificial Intelligence (AI) applications in factories, poses a significant challenge for the current datacenter-focused technological architecture. The “Big data pRocessing and Artificial Intelligence at the Network Edge” (BRAINE) project addresses this problem by introducing an innovative system architecture designed explicitly for compute-intensive edge deployments. BRAINE focuses on decentralizing the computation tasks, enabling a significant reduction in latency, and optimizing the placement of applications within a cloud-edge continuum to ensure optimal operational efficiency. This paper presents the design, implementation, and testing of our novel system architecture in the context of an AI digital twin for factory robotics. Our empirical results indicate substantial improvements in performance metrics such as processing speed and latency compared to traditional architectures and approaches.

Robotic cells in industrial automation use different programming and description languages, which are typically almost hard-wired in the program solutions and do not allow a goal product to be changed easily. Using a unification description language allows focusing on the solution itself. An independent description of capabilities and intentions offers a way to allow for changing the goal and also for distributing the process to different locations. Moreover, based on the description, a knowledge model can be created to check if a solution exists for the defined goal and resource capabilities. If the knowledge model is implemented in a database, the check can be performed in a very Efficient way, which allows it to be used in real-life production scenarios. A reasoner tries to reach the solution-defined goal based on an initial state and actions following the predicates’ rules. Using the database to search only for the first possible solution, it can be checked if the defined domain model can be realized. The check can be done automatically and much faster than using a planner. In an implementation in our Testbed for Industry 4.0 at CIIRC/CTU Prague, we employ a scalable system based on a PDDL description and an automated translation to TypeDB to efficiently compute production plans for changing goals, tools and resources.

The Linear time-invariant dynamic models are widely adopted in the industry. In the machine learning domain, such models are known as time-invariant continuous-state hidden Gauss-Markov models. Their super-class, the linear time-varying dynamic models, have relatively sparse applications as predictive models and classifiers of time series. This is typically due to the model complexity and the need for a significantly larger training set than time-invariant models. Without a large training set, a better modeling performance is counteracted by a less robust model. In this paper, we propose the continuity preference of the time-varying parameters of the model, which significantly reduces the required amount of training data while maintaining the modeling performance. We also derive a simple modification of the Expectation-Maximization algorithm incorporating continuity in parameters. The modified algorithm shows robust learning performance. The model performance is demonstrated by experiments on real 6-axis robotic manipulators in a laboratory, the Skoda Auto car producer body shop, and also on a public benchmark data set. (C) 2022 Elsevier Inc. All rights reserved.

This work presents the BRAINE Project approaches and solutions to enable AI-empowered industrial applications to efficiently and securely operate at the network edge. First, two use cases are presented to highlight the motivations for secure edge computing in industrial plants, including low-latency applications and the need to locally handle production data in a secure way. Then, AI-based proactive and reactive approaches are discussed to provide comprehensive cyber security at the industrial edge. Finally, two specific hardware accelerated implementations for cyber security are presented. The first implementation focuses on data encryption at network interface cards, the second implementation provides deep packet inspection for reconnaissance detection in switching elements. Results that hardware accelerated solutions enable the effective deployment of security solutions guaranteeing wire-speed operations while freeing CPU resources for industrial applications.

Modern manufacturing faces a challenge of integrating data models from various sources/domains which may differ both semantically and technically when particular domain specific data models are designed by different users and stored in different formats. This paper introduces an approach for facilitating the design of domain-specific data models using semantic web technologies. In this approach, all the information required for managing the production (including a description of a product, processes involved in the production, and existing resources and their specifications) is captured in an ontology. The proposed Product, Process, and Resource (PPR) ontology defines fundamental conceptualization of the production that can be easily applied to the arbitrary domain. Application of the PPR ontology is demonstrated in the case of simple truck assembling by means of robots. Capturing the knowledge in the form of ontology provides the advantage of employing supporting tools such as reasoners for consistency checking or query languages for information extraction. The paper demonstrates the utilization of SQWRL for searching resources suitable to manipulate given truck parts on the basis of semantic matching between properties of particular elements.

A novel mathematical formulation of the energy optimisation problem for robotic lines is presented, which allows minimising the energy consumption in a robotic cell while keeping the required production cycle time. Different energy saving modes of the robots are utilised as well as the fact that the robot energy consumption during its movement depends on the movement duration. This dependency is modelled with a so-called energy function, which can be obtained by measurements, physical modelling of the robots or simulation. Each of these areas is covered by the presented work. The achieved results show there is a good potential to achieve energy savings at existing robotic cells and their series, and an even bigger potential if the presented approach is used during the design phase of new robotic cells.

This paper tackles the problem of processing measured values in time series of energy consumption data obtained in robotic production cells. The consumed energy is measured at each robot in the cell to get information about the robotic operations that are performed. Such knowledge may serve as a basis for further steps such as minimization of the energy consumption or diagnosis of the robot behavior. For the modeling of the robots, Continuous State Hidden Gaussian-Markov Models (CS-HGMM) were developed in the previous work, which rely on a set of training examples of sequences for unsupervised training. In this paper, segmentation based on signal information contents and unsupervised clustering of the acquired segments is presented. The used clustering methods have been adapted from the OPTICS algorithm, which is a generalization of the popular DBSCAN algorithm. This approach has resulted in the ability to process irregular artefacts in measured data that do not represent any particular robotic operation, and to process and cluster segment candidates that do not have the same length which happens quite often in the industrial applications.

This study focuses on the energy optimization of industrial robotic cells, which is essential for sustainable production in the long term. A holistic approach that considers a robotic cell as a whole toward minimizing energy consumption is proposed. The mathematical model, which takes into account various robot speeds, positions, power-saving modes, and alternative orders of operations, can be transformed into a mixed-integer linear programming formulation that is, however, suitable only for small instances. To optimize complex robotic cells, a hybrid heuristic accelerated by using multicore processors and the Gurobi simplex method for piecewise linear convex functions is implemented. The experimental results showed that the heuristic solved 93% of instances with a solution quality close to a proven lower bound. Moreover, compared with the existing works, which typically address problems with three to four robots, this study solved real-size problem instances with up to 12 robots and considered more optimization aspects. The proposed algorithms were also applied on an existing robotic cell in Skoda Auto. The outcomes, based on simulations and measurements, indicate that, compared with the previous state (at maximal robot speeds and without deeper power-saving modes), the energy consumption can be reduced by about 20% merely by optimizing the robot speeds and applying power-saving modes. All the software and generated datasets used in this research are publicly available.

This paper focuses on identifying the operations of industrial manipulators that are often realised as robots with six degrees of freedom. The identification is based on the measurement of the power consumption of the whole robot without separating it to the individual axes. Such a case corresponds to industrial use cases. The robot is taken as an unobservable system with respect to its internal states. This disadvantage is overcome using hidden time-varying Markov chains. The results are verified on a simple one-axis manipulator and further discussion about the extension of the model is provided.

The paper deals with a modelling of the energy consumption of industrial articulated robots. The proposed modelling way is based on graphically-oriented computer-aided concept that exploits CAD software Solidworks and simulation environment MATLAB/Simulink with SimMechanics and SimPowerSystems libraries. These software tools are used for the composition of a dynamical simulation model that represents both mechanical robot structure and robot drives during robot motions. The paper addresses mathematical analysis and interpretation of the considered simulation model. Here, equations of the motion for the mechanical robot structure and appropriate dynamical equations of the robot drives are introduced. Using these equations, the energy consumption equation is defined. The proposed way is demonstrated by simulation experiments for several different velocities of the robot motion along a selected trajectory. For the experiments, industrial articulated robot KUKA KR 5 arc driven by PMSM drives is considered.

This paper focuses on the analysis of the robotic operations in a welding cell based on the measurement of the robot energy consumption. In such a way it is possible to have information about the production status of the welding cell without a single intervention or disruption of the main controller program that controls the welding cell. Methods of behaviour pattern recognition are used.

This paper presents a way how a simulation model of an industrial Ethernet network is designed and how it is parametrised according to a real network. The parameter set up is done by capturing the Ethernet frames in the network and by performing their detailed analysis. The device types in the simulation have been designed as generic that can be parametrised externally and automatically based on the identification of a real network. Thus the parameters of the real devices such as switch delay are used to adapt the simulation and the simulation results are compared to the real traffic in a series of long-term observations. Such a setting can be used to perform diagnostics of industrial Ethernet networks to identify e.g. bursts of messages and thus discovering potential bottlenecks and problems aiming at interrupting or disturbing the connected application process, eventually.