The commissioning and operation of future large-scale scientific experiments will challenge current tuning and control methods. Reinforcement learning (RL) algorithms are a promising solution thanks to their capability of autonomously tackling a control problem based on a task parameterized by a reward function. The conventionally utilized machine learning (ML) libraries are not intended for microsecond latency applications, as they mostly optimize for throughput performance. On the other hand, most of the programmable logic implementations are meant for computation acceleration, not being intended to work in a real-time environment. To overcome these limitations of current implementations, RL needs to be deployed on-the-edge, i.e. on to the device gathering the training data. In this paper we present the design and deployment of an experience accumulator system in a particle accelerator. In this system deep-RL algorithms run using hardware acceleration and act within a few microseconds, enabling the use of RL for control of ultra-fast phenomena. The training is performed offline to reduce the number of operations carried out on the acceleration hardware. The proposed architecture was tested in real experimental conditions at the Karlsruhe research accelerator (KARA), serving also as a synchrotron light source, where the system was used to control induced horizontal betatron oscillations in real-time. The results showed a performance comparable to the commercial feedback system available at the accelerator, proving the viability and potential of this approach. Due to the self-learning and reconfiguration capability of this implementation, its seamless application to other control problems is possible. Applications range from particle accelerators to large-scale research and industrial facilities.

Machine learning has emerged as a powerful solution to the modern challenges in accelerator physics. However, the limited availability of beam time, the computational cost of simulations, and the high-dimensionality of optimisation problems pose significant challenges in generating the required data for training state-of-the-art machine learning models. In this work, we introduce Cheetah, a PyTorch-based high-speed differentiable linear-beam dynamics code. Cheetah enables the fast collection of large data sets by reducing computation times by multiple orders of magnitude and facilitates efficient gradient-based optimisation for accelerator tuning and system identification. This positions Cheetah as a user-friendly, readily extensible tool that integrates seamlessly with widely adopted machine learning tools. We showcase the utility of Cheetah through five examples, including reinforcement learning training, gradient-based beamline tuning, gradient-based system identification, physics-informed Bayesian optimisation priors, and modular neural network surrogate modelling of space charge effects. The use of such a high-speed differentiable simulation code will simplify the development of machine learning-based methods for particle accelerators and fast-track their integration into everyday operations of accelerator facilities.

Abstract Online tuning of particle accelerators is a complex optimisation problem that continues to require manual intervention by experienced human operators. Autonomous tuning is a rapidly expanding field of research, where learning-based methods like Bayesian optimisation (BO) hold great promise in improving plant performance and reducing tuning times. At the same time, reinforcement learning (RL) is a capable method of learning intelligent controllers, and recent work shows that RL can also be used to train domain-specialised optimisers in so-called reinforcement learning-trained optimisation (RLO). In parallel efforts, both algorithms have found successful adoption in particle accelerator tuning. Here we present a comparative case study, assessing the performance of both algorithms while providing a nuanced analysis of the merits and the practical challenges involved in deploying them to real-world facilities. Our results will help practitioners choose a suitable learning-based tuning algorithm for their tuning tasks, accelerating the adoption of autonomous tuning algorithms, ultimately improving the availability of particle accelerators and pushing their operational limits.

Real-world applications of reinforcement learning (RL) face challenges such as the need for numerous interactions and achieving stable training under dynamic conditions. Meta-RL emerges as a solution, particularly in environments where simulations cannot perfectly mimic real-world conditions. This study demonstrates Meta-RL's potential in the CERN's AWAKE project, focusing on the electron line's control. By incorporating Model-Agnostic Meta-Learning (MAML), we showcase how Meta-RL facilitates rapid adaptation to environmental changes with minimal interaction steps. Our findings indicate Meta-RL's efficacy in managing Partially Observable Markov Decision Processes (POMDPs) with evolving hidden parameters, underlining its significance in high-dimensional control challenges prevalent in particle physics experiments and beyond.

Accelerator physics relies on numerical algorithms to solve optimization problems in online accelerator control and tasks such as experimental design and model calibration in simulations. The effectiveness of optimization algorithms in discovering ideal solutions for complex challenges with limited resources often determines the problem complexity these methods can address. The accelerator physics community has recognized the advantages of Bayesian optimization algorithms, which leverage statistical surrogate models of objective functions to effectively address complex optimization challenges, especially in the presence of noise during accelerator operation and in resource-intensive physics simulations. In this review article, we offer a conceptual overview of applying Bayesian optimization techniques towards solving optimization problems in accelerator physics. We begin by providing a straightforward explanation of the essential components that make up Bayesian optimization techniques. We then give an overview of current and previous work applying and modifying these techniques to solve accelerator physics challenges. Finally, we explore practical implementation strategies for Bayesian optimization algorithms to maximize their performance, enabling users to effectively address complex optimization challenges in real-time beam control and accelerator design.

Online tuning of real-world plants is a complex optimisation problem that continues to require manual intervention by experienced human operators. Autonomous tuning is a rapidly expanding field of research, where learning-based methods, such as Reinforcement Learning-trained Optimisation (RLO) and Bayesian optimisation (BO), hold great promise for achieving outstanding plant performance and reducing tuning times. Which algorithm to choose in different scenarios, however, remains an open question. Here we present a comparative study using a routine task in a real particle accelerator as an example, showing that RLO generally outperforms BO, but is not always the best choice. Based on the study's results, we provide a clear set of criteria to guide the choice of algorithm for a given tuning task. These can ease the adoption of learning-based autonomous tuning solutions to the operation of complex real-world plants, ultimately improving the availability and pushing the limits of operability of these facilities, thereby enabling scientific and engineering advancements.

We have evaluated the data-efficient Bayesian optimization method for the specific task of injection tuning in a circular accelerator. In this paper, we describe the implementation of this method at the Karlsruhe Research Accelerator with up to nine tuning parameters, including the determination of the associated hyperparameters. We show that the Bayesian optimization method outperforms manual tuning and the commonly used Nelder-Mead optimization algorithm in both simulation and experiment. The algorithm was also successfully used to ease the commissioning phase after the installation of new injection magnets and is regularly used during accelerator operations. We demonstrate that the introduction of context variables that include intrabunch scattering effects, such as the Touschek effect, further improves the control and robustness of the injection process.

The injection at the KIT storage ring KARA (Karlsruhe Research Accelerator) is tuned by many parameters, such as the strength of various magnets and the RF frequency. The tuning process is currently performed manually by machine operators, which is time consuming and can get stuck in local optima. To address this, Bayesian optimisation is applied, i.e. a technique for optimising noisy black-box functions. Using Gaussian processes (GPs) for regression we obtain a probabilistic model, which allows the integration of prior knowledge about the physical process. The model can be queried during the optimization procedure to efficiently explore the given parameter space, leading to comparably fast convergence. In this contribution, we demonstrate the implementation of Bayesian optimization to automate and optimize the injection process. Chenran Xu acknowledges the support by the DFG-funded Doctoral School Karlsruhe School of Elementary and Astroparticle Physics: Science and Technology.

The characteristics of an electron bunch created with a photoinjector are strongly influenced by the properties of the driving laser. Therefore, we plan to install spatial light modulators (SLMs) in the laser beam path to shape and manipulate the infrared laser pulses, which are converted to the ultraviolet spectrum, and eventually also the generated electron bunch in the transverse as well as the longitudinal plane. SLMs should also allow the flexible mitigation of aberrations in the beam profile. The SLMs are planned to be installed in FLUTE (Ferninfrarot Linac- und Test-Experiment) at KIT, a compact linac-based test facility for accelerator R&D and source of intense THz radiation. In this presentation, the first test-setup is shown.

Reinforcement Learning algorithms have risen in popularity in recent years in the accelerator physics community, showing potential in beam control and in the optimization and automation of tasks in accelerator operation. The Helmholtz AI project "Machine Learning toward Autonomous Accelerators" is a collaboration between DESY and KIT that works on investigating and developing RL applications for the automatic start-up of electron linear accelerators. The work is carried out in parallel at two similar research accelerators: ARES at DESY and FLUTE at KIT, giving the unique opportunity of transfer learning between facilities. One of the first steps of this project is the establishment of a common interface between the simulations and the machine, in order to test and apply various optimization approaches interchangeably between the two accelerators. In this paper we present the first results on the common interface and its application to beam focusing in ARES, and the idea of laser shaping with spatial light modulators at FLUTE.