Typical schemes to encode classical data in variational quantum machine learning (QML) lead to quantum Fourier models with 𝒪(exp(n)) Fourier basis functions in the number of qubits. Despite this, in order for the model to be efficiently trainable, the number of parameters must scale as 𝒪(poly(n)). This imbalance implies the existence of correlations between the Fourier modes, which depend on the structure of the circuit. In this work, we demonstrate that this phenomenon exists and show cases where these correlations can be used to predict ansatz performance. For several popular ansatzes, we numerically compute the Fourier coefficient correlations (FCCs) and construct the Fourier fingerprint, a visual representation of the correlation structure. We subsequently show how, for the problem of learning random Fourier series, the FCC correctly predicts relative performance of ansatzes whilst the widely-used expressibility metric does not. Finally, we demonstrate how our framework applies to the more challenging problem of jet reconstruction in high-energy physics. Overall, our results demonstrate how the Fourier fingerprint is a powerful new tool in the problem of optimal ansatz choice for QML.

The gradients used to train neural networks are typically computed using backpropagation. While an efficient way to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is biologically implausible. Forward gradients are an approach to approximate the gradients from directional derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal projections. We demonstrate that increasing the number of tangents improves both approximation quality and optimization performance across various tasks.

Federated Learning (FL) is an approach for training a shared Machine Learning (ML) model with distributed training data and multiple participants. FL allows bypassing limitations of the traditional Centralized Machine Learning CL if data cannot be shared or stored centrally due to privacy or technical restrictions – the participants train the model locally with their training data and do not need to share it among the other participants. This paper investigates the practical implementation and effectiveness of FL in a real-world scenario, specifically focusing on unmanned aerial vehicle (UAV)-based thermal images for common thermal feature detection in urban environments. The distributed nature of the data arises naturally and makes it suitable for FL applications, as images captured in two German cities are available. This application presents unique challenges due to non-identical distribution and feature characteristics of data captured at both locations. The study makes several key contributions by evaluating FL algorithms in real deployment scenarios rather than simulation. We compare several FL approaches with a centralized learning baseline across key performance metrics such as model accuracy, training time, communication overhead, and energy usage. This paper also explores various FL workflows, comparing client-controlled workflows and server-controlled workflows. The findings of this work serve as a valuable reference for understanding the practical application and limitations of the FL methods in segmentation tasks in UAV-based imaging.

Backpropagation has long been criticized for being biologically implausible, relying on concepts that are not viable in natural learning processes. This paper proposes an alternative approach to solve two core issues, i.e., weight transport and update locking, for biological plausibility and computational efficiency. We introduce Feed-Forward with delayed Feedback (F$^3$), which improves upon prior work by utilizing delayed error information as a sample-wise scaling factor to approximate gradients more accurately. We find that F$^3$ reduces the gap in predictive performance between biologically plausible training algorithms and backpropagation by up to 96%. This demonstrates the applicability of biologically plausible training and opens up promising new avenues for low-energy training and parallelization.

The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings.

In this work, we propose a framework in the form of a Python package, specifically designed for the analysis of Quantum Machine Learning models. This framework is based on the PennyLane simulator and facilitates the evaluation and training of Variational Quantum Circuits. It provides additional functionality ranging from the ability to add different types of noise to the classical simulation, over different parameter initialisation strategies, to the calculation of expressibility and entanglement for a given model. As an intrinsic property of Quantum Fourier Models, it provides two methods for calculating the corresponding Fourier spectrum: one via the Fast Fourier Transform and another analytical method based on the expansion of the expectation value using trigonometric polynomials. It also provides a set of predefined approaches that allow a fast and straightforward implementation of Quantum Machine Learning models. With this framework, we extend the PennyLane simulator with a set of tools that allow researchers a more convenient start with Quantum Fourier Models and aim to unify the analysis of Variational Quantum Circuits.

Atmospheric composition is an essential part of weather, climate and Earth system modeling. However, modeling atmospheric composition is a computationally expensive and time-consuming task that requires a significant amount of energy. As models scale to finer spatial and temporal resolutions, maintaining real-time performance becomes increasingly challenging. To address this, optimization and acceleration techniques are essential. One promising approach is the use of deep neural networks, which have demonstrated the capability to efficiently approximate complex systems with high accuracy. Predictions using these neural networks are notably faster compared to traditional methods, significantly reducing the computational burden. In this study, we present the development of a surrogate model designed to emulate ISORROPIA, a traditional model used for calculating the concentrations of chemical compounds in the ICON-ART (ICOsahedral Nonhydrostatic model with Aerosol and Reactive Trace gases) model. Specifically, ISORROPIA is an aerosol thermodynamic equilibrium model used by ART that requires substantial computational resources, occupying a significant portion of the overall calculation time, making it particularly well-suited for emulation. The methodology involved generating a comprehensive dataset using the traditional model, which served as the training data for the neural network. This dataset encompassed a wide range of chemical concentrations and conditions, ensuring the neural network could effectively learn the underlying patterns and relationships for real-life scenarios. A simple feedforward architecture was used and fine-tuned with the primary goal of maintaining a low approximation error while also striving to achieve the lowest possible inference timing. After training, the new neural network model was compared to ISORROPIA on ICON-ART simulation data. The results demonstrated that the neural network model successfully achieved the desired outcomes, maintaining low approximation error across the globe and efficient inference timing.

The future German HEP computing strategy will partially rely on national scientific HPC centers for providing the CPU pledges to the WLCG. The pilot phase for this endeavor is starting in 2025 from which on the share provided by HPC resources will gradually increase. To make this courageous step a success, it is essential that the integrated HPC sites can be utilized just as reliable and efficient for HEP workflows as the traditional, dedicated Grid sites. Motivated by I/O limitations observed at HoreKa, the scientific HPC cluster at KIT, the integration was optimized and an XRootD-based approach for data access bottleneck mitigation was developed and deployed, resulting in a comparable performance of the cluster. This is an important result supporting the future HEP computing strategy in Germany and shows that national scientific HPC centers are capable of delivering the performance and reliability required to be used for providing pledges to the WLCG. Moreover, the findings underline the potential of integrating HPC clusters into the Grid, paving the way for a scalable and sustainable computing infrastructure in the HL-LHC era.

The rapid growth of environmental data and the complexity of data pre-processing tasks poses significant challenges to environmental scientists. Repetitive and error-prone manual data preparation methods not only consume valuable research time but also introduce potential data quality issues. Also, individually pre-processed datasets are hardly reproducible. The V-FOR-WaTer virtual research environment (VRE) addresses these challenges as a powerful tool that seamlessly integrates data access, data pre-processing, and data exploration capabilities. V-FOR-WaTer has an automated data pre-processing workflow to improve data preparation by eliminating the need for manual data cleaning, standardization, harmonization, and formatting. This approach significantly reduces the risk of human error while freeing up researchers to focus on their actual research questions without being hampered by data preparation. The pre-processing tools integrated in the virtual research environment are designed to handle a wide range of data formats, ensuring consistent and reliable data preparation across diverse disciplines. This empowers researchers to seamlessly integrate data from various sources in a standardized manner. The web portal's user-centric design facilitates data exploration and selection through map operations and filtering options, empowering researchers to efficiently identify and focus on relevant data for their analyses. The scalability and extensibility of the V-FOR-WaTer web portal ensures that it can accommodate the ever-growing volume of environmental data and adapt to the evolving research landscape. Its ability to integrate user-developed tools reflects the dynamic nature of environmental research and ensures that the virtual research environment stays up-to-date with the latest advancements in data processing. The comprehensive features and user-friendly interface position it as a valuable tool for environmental scientists, fostering collaboration, streamlining data analysis, and accelerating the advancement of knowledge in the field of hydrology.

We introduce WOKIE, an open-source, modular, and ready-to-use pipeline for the automated translation of SKOS thesauri. This work addresses a critical need in the Digital Humanities (DH), where language diversity can limit access, reuse, and semantic interoperability of knowledge resources. WOKIE combines external translation services with targeted refinement using Large Language Models (LLMs), balancing translation quality, scalability, and cost. Designed to run on everyday hardware and be easily extended, the application requires no prior expertise in machine translation or LLMs. We evaluate WOKIE across several DH thesauri in 15 languages with different parameters, translation services and LLMs, systematically analysing translation quality, performance, and ontology matching improvements. Our results show that WOKIE is suitable to enhance the accessibility, reuse, and cross-lingual interoperability of thesauri by hurdle-free automated translation and improved ontology matching performance, supporting more inclusive and multilingual research infrastructures.

Noisy Intermediate-Scale Quantum (NISQ) computers, despite their limitations, present opportunities for near-term quantum advantages in Nuclear and High-Energy Physics (NHEP) when paired with specially designed quantum algorithms and processing units. This study focuses on core algorithms that solve optimization problems through the quadratic Ising or Quadratic Unconstrained Binary Optimisation model, specifically Quantum Annealing and the Quantum Approximate Optimisation Algorithm (QAOA). In particular, we estimate runtimes and scalability for the task of particle Track Reconstruction (TR), a key computing challenge in NHEP, and investigate how the classical parameter space in QAOA, along with techniques like a Fourieranalysis based heuristic, can facilitate future quantum advantages. The findings indicate that lower frequency components in the parameter space are crucial for effective annealing schedules, suggesting that heuristics can improve resource efficiency while achieving near-optimal results. Overall, the study highlights the potential of NISQ computers in NHEP and the significance of co-design approaches and heuristic techniques in overcoming challenges in quantum algorithms.

Transformers have recently gained prominence in long time series forecasting by elevating accuracies in a variety of use cases. Regrettably, in the race for better predictive performance the overhead of model architectures has grown onerous, leading to models with computational demand infeasible for most practical applications. To bridge the gap between high method complexity and realistic computational resources, we introduce the Residual Cyclic Transformer, ReCycle. ReCycle utilizes primary cycle compression to address the computational complexity of the attention mechanism in long time series. By learning residuals from refined smoothing average techniques, ReCycle surpasses state-of-the-art accuracy in a variety of application use cases. The reliable and explainable fallback behavior ensured by simple, yet robust, smoothing average techniques additionally lowers the barrier for user acceptance. At the same time, our approach reduces the run time and energy consumption by more than an order of magnitude, making both training and inference feasible on low-performance, low-power and edge computing devices. Code is available at https://github.com/Helmholtz-AI-Energy/ReCycle

With the emergence of the research field of Quantum Machine Learning, interest in finding advantageous real-world applications is growing as well. However, challenges concerning the number of available qubits on Noisy Intermediate Scale Quantum (NISQ) devices and accuracy losses due to hardware imperfections still remain and limit the applicability of such approaches in real-world scenarios. Therefore, for simplification, most studies assume nearly noise-free conditions as they are expected with logical, i.e. error-corrected, qubits instead of real qubits provided by hardware. However, the number of logical qubits is expected to scale slowly as they require a high number of real qubits for error correction. This is our motivation to deal with noise as an unavoidable, non-negligible problem on NISQ devices. As an application, we use the example of particle decay tree reconstruction as a highly complex combinatoric problem in High Energy Physics. We investigate methods to reduce the noise impact of such devices and propose a hybrid architecture that extends a classical graph neural network by a parameterized quantum circuit. While we have shown that such a hybrid architecture enables a reduction of the amount of trainable parameters compared to the fully classical case, we are now specifically interested in the actual performance in more realistic, i.e. noise prone scenarios. Using simple synthetic Decay Trees, we train the network in classical simulations to allow for efficient optimization of the parameters. The trained parameters are validated in noisy simulations based on devices by "IBM Quantum" and are used in interpretability and significance studies, enabling improvements in the accuracy on real devices.

Ensemble learning is a widespread technique to improve the prediction performance of neural networks. However, it comes at the price of increased memory and inference time. In this work we propose a novel model fusion technique called Neuron Transplantation (NT) in which we fuse an ensemble of models by transplanting important neurons from all ensemble members into the vacant space obtained by pruning insignificant neurons. An initial loss in performance post-transplantation can be quickly recovered via fine-tuning, consistently outperforming individual ensemble members of the same model capacity and architecture. Furthermore, NT enables all the ensemble members to be jointly pruned and jointly trained in a combined model. Comparing it to alignment-based averaging (like Optimal-Transport-fusion), it requires less fine-tuning than the corresponding OT-fused model, the fusion itself is faster and requires less memory, while the resulting model performance is comparable or better. The code is available under the following link: https://github.com/masterbaer/neuron-transplantation.

Clustering in data mining involves grouping similar objects into categories based on their characteristics. As the volume of data continues to grow and advancements in high-performance computing evolve, a critical need has emerged for algorithms that can efficiently process these computations and exploit the various levels of parallelism offered by modern supercomputing systems. Exploiting Single Instruction Multiple Data (SIMD) instructions enhances parallelism at the instruction level and minimizes data movement within the memory hierarchy. To fully harness a processor’s SIMD capabilities and achieve optimal performance, adapting algorithms for better compatibility with vector operations is necessary. In this paper, we introduce a vectorized implementation of the Density-based Clustering for Applications with Noise (DBSCAN) algorithm suitable for the execution on both shared and distributed memory systems. By leveraging SIMD, we enhance the performance of distance computations. Our proposed Vectorized HPDBSCAN (VHPDBSCAN) demonstrates a performance improvement of up to two times over the state-of-the-art parallel version, Highly Parallel DBSCAN (HPDBSCAN), on the ARM-based A64FX processor on two different datasets with varying dimensions. We have parallelized computations which are essential for the efficient workload distribution. This has significantly enhanced the performance on higher dimensional datasets. Additionally, we evaluate VHPDBSCAN’s energy consumption on the A64FX and Intel Xeon processors. The results show that in both processors, due to the reduced runtime, the total energy consumption of the application is reduced by 50% on the A64FX Central Processing Unit (CPU) and by approximately 19% on the Intel Xeon 8368 CPU compared to HPDBSCAN.

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals that an orthogonal basis within each multidimensional weight's SVD representation stabilizes during training. Building upon this, we introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel training method exploiting the intrinsic orthogonality of neural networks. OIALR seamlessly integrates into existing training workflows with minimal accuracy loss, as demonstrated by benchmarking on various datasets and well-established network architectures. With appropriate hyperparameter tuning, OIALR can surpass conventional training setups, including those of state-of-the-art models.

Communication bottlenecks hinder the scalability of distributed neural network training, particularly on distributed-memory computing clusters. To significantly reduce this communication overhead, we introduce AB-training, a novel data-parallel training method that decomposes weight matrices into low-rank representations and utilizes independent group-based training. This approach consistently reduces network traffic by 50 scenarios, increasing the training potential on communication-constrained systems. Our method exhibits regularization effects at smaller scales, leading to improved generalization for models like VGG16, while achieving a remarkable 44.14 : 1 compression ratio during training on CIFAR-10 and maintaining competitive accuracy. Albeit promising, our experiments reveal that large batch effects remain a challenge even in low-rank training regimes.

Given the urgency to reduce fossil fuel energy production to make climate tipping points less likely, we call for resource-aware knowledge gain in the research areas on Universe and Matter with emphasis on the digital transformation. A portfolio of measures is described in detail and then summarized according to the timescales required for their implementation. The measures will both contribute to sustainable research and accelerate scientific progress through increased awareness of resource usage. This work is based on a three-days workshop on sustainability in digital transformation held in May 2023.

Data-driven medium-range weather forecasts have recently outperformed classical numerical weather prediction models, with Pangu-Weather (PGW) being the first breakthrough model to achieve this. The Transformer-based PGW introduced novel architectural components including the three-dimensional attention mechanism (3D Transformer) in the Transformer blocks. Additionally, it features an Earth-specific positional bias term which accounts for weather states being related to the absolute position on Earth. However, the effectiveness of different architectural components is not yet well understood. Here, we reproduce the 24 h forecast model of PGW based on subsampled 6-hourly data. We then present an ablation study of PGW to better understand the sensitivity to the model architecture and training procedure. We find that using a two-dimensional attention mechanism (2D Transformer) yields a model that is more robust to training, converges faster, and produces better forecasts compared to using the 3D Transformer. The 2D Transformer reduces the overall computational requirements by 20 %-30 %. Further, the Earth-specific positional bias term can be replaced with a relative bias, reducing the model size by nearly 40 %. A sensitivity study comparing the convergence of the PGW model and the 2D-Transformer model shows large batch effects; however, the 2D-Transformer model is more robust to such effects. Lastly, we propose a new training procedure that increases the speed of convergence for the 2D-Transformer model by 30 % without any further hyperparameter tuning.

Predicting the performance of various infrastructure design options in complex federated infrastructures with computing sites distributed over a wide area network that support a plethora of users and workflows, such as the Worldwide LHC Computing Grid (WLCG), is not trivial. Due to the complexity and size of these infrastructures, it is not feasible to deploy experimental test-beds at large scales merely for the purpose of comparing and evaluating alternate designs. An alternative is to study the behaviours of these systems using simulation. This approach has been used successfully in the past to identify efficient and practical infrastructure designs for High Energy Physics (HEP). A prominent example is the Monarc simulation framework, which was used to study the initial structure of the WLCG. New simulation capabilities are needed to simulate large-scale heterogeneous computing systems with complex networks, data access and caching patterns. A modern tool to simulate HEP workloads that execute on distributed computing infrastructures based on the SimGrid and WRENCH simulation frameworks is outlined. Studies of its accuracy and scalability are presented using HEP as a case-study. Hypothetical adjustments to prevailing computing architectures in HEP are studied providing insights into the dynamics of a part of the WLCG and candidates for improvements.