A requirement for a significant amount of training as well as the exploration of potentially expensive or safety-critical states limits the applicability of reinforcement learning for real-world robotics. One potential solution is given by pretraining models in simulations before transferring them to the real world. In this chapter, we investigate the concept of domain randomization to train robust agents in simulation to control an industrial robot. We examine the effects of different degrees of randomization with respect to the transferability to the real world. In addition, we use attention maps to gain insights into the agents’ decision-making processes. We find that attention maps enable a qualitative assessment for the data-efficiency of a pretrained agent when transferred to the real-world setup.

The black box problem of artificial neural networks (ANNs) is still a very relevant issue. When communicating basic concepts of ANNs, they are often depicted as node-link diagrams. Despite this being a straight forward way to visualize them, it is rarely used outside an educational context. However, we hypothesize that large-scale node-link diagrams of full ANNs could be useful even to machine learning experts. Hence, we present a visualization tool that depicts convolutional ANNs as node-link diagrams using immersive virtual reality. We applied our tool to a use-case in the field of machine learning research and adapted it to the specific challenges. Finally, we performed an expert review to evaluate the usefulness of our visualization. We found that our node-link visualization of ANNs was perceived as helpful in this professional context.

In the production industry, the digital transformation enables a significant optimization potential. The concept of reinforcement learning offers a suitable approach to train agents on learning control strategies, further advancing automation. While applications training directly on real-world processes are rare due to economical and safety constraints, simulations offer a way to develop and evaluate agents prior to deployment. With the rise of service-based business models, the simulation owner and the machine learning expert are likely to be different stakeholders in a joint project. Due to different requirements for both simulations and reinforcement-learning agents, the stakeholders may be reluctant or unable to grant full access to the respective software. This poses a serious impediment to the potential of the digital transformation. In this paper, a distributed architecture is proposed, which allows the remote training of reinforcement learning agents on a simulation. It is shown that this architecture allows the cooperation between two stakeholders by exposing a suitable technical interface to the simulation. The proposed architecture is implemented for a simulation of the multi-step metal forming process of heavy plate rolling. Furthermore, the implemented architecture is used to successfully train a reinforcement-learning agent on the task of designing optimal parameter schedules.

While it is apparent that the transfer of knowledge between tasks is beneficial for training efficiency, the application of trained deep reinforcement learning agents to solve new tasks is not trivial. Especially when tasks are differently structured, retraining and fine tuning is not necessarily beneficial. Instead, it is often the most convenient approach to train a new agent from scratch. One potential solution for effectively reusing learned knowledge may be found in hierarchical reinforcement learning. In this paper we investigate the possibility of reusing low-level policies to improve training efficiency when learning manipulation tasks with an industrial robot. We consider four different scenarios and demonstrate for three of them an increased sample efficiency when training a high-level policy on top of pre-trained low-level skills. In the fourth scenario we uncover the reason for a failed transfer to be an ambitious higher hierarchy level enforcing a relearning of the low-level skills.

The concept of reinforcement learning enables an agent to learn a task based on trial and error. Especially for the automation of industrial processes, this approach promises significant advantages in terms of flexibility and adaptability when compared to rule-based solutions. While previous works have uncovered the potential of reinforcement learning and the applicability to real-world scenarios was shown, the algorithm relies on a discretization of time, where every time step comprises a self-contained sequence of observation, execution and feedback. However, this design poses a major obstacle for tasks, which do not allow for a distinct separation of steps. A prominent example is motion planning for industrial robotics, where reinforcement-learning solutions to date result in non-fluent trajectories. In this work, we address this shortcoming of reinforcement learning by presenting an asynchronous update strategy, which enables the agent to plan its next trajectory while executing the previous one. We use Bézier curves as actions due to the ability to characterize complex trajectories with relatively few parameters. We show that our modifications further improve the smoothness of the robot’s motion and allow for a smoother velocity profile without a drop in performance when compared to previous solutions.

Motion planning processes for industrial robots are complex tasks often done manually by human domain experts and result in robotic motion that lacks flexibility and adaptability in response to dynamic environments. In our work, we propose an automated control agent utilizing a convolutional neural network embedded in an actor-critic architecture that learns adaptive continuous motion behavior via reinforcement learning. The learning is based on direct sensory input without the need of directly programming the robot’s motion. We show that the learned behavior can account for uncontrollable dynamic environmental circumstances and helps to decrease time and cost of ramp up processes.