Abstract To handle increased data flow, Jefferson Lab (JLab) is partnering with ESnet for development of an AI/ML directed compute work Load Balancer (LB) of UDP streamed data. The LB is FPGA based featuring dynamically configurable, low latency and high throughput destination address switching. The LB provides integration of edge and core computing to support JLab experimental programs, the Electron-Ion Collider, as well as data centers of the future. In the ESnet/JLab FPGA Accelerated Transport (EJFAT) initiative, the function of the LB Data Plane (DP) is to redirect data streams to selectable (but unknown to sender) destination hosts based on current worload and within that host to destination ports as a function of substream id. This effects hierarchical scaling, first across compute machines for processing over a series of events and second, across ports so different data source sub-streams may be assigned to different processors for further parallelization. The LB Control Plane (CP) programs the DP using compute farm telemetry to direct and balance workloads across a compute cluster as the operating conditions require. While Proportional / Integrative / Derivative (PID) controllers are often seen in similar applications, here we investigate the feasibility of a Reinforcement Learning (RL) based schedule manager running in the CP to provide dynamic updates to the DP scheduling policy.

The JIRIAF (JLab Integrated Research Infrastructure Across Facilities) framework is designed to streamline resource management and optimize high-performance computing (HPC) workloads across heterogeneous environments. Central to JIRIAF is the JIRIAF Resource Manager (JRM), which effectively leverages Kubernetes and Virtual Kubelet to manage resources dynamically, even in environments with restricted user privileges. By operating in userspace, JRM facilitates the execution of user applications as containers across diverse computing sites, ensuring unified control and monitoring. The framework's effectiveness is demonstrated through a case study involving the deployment of data-stream processing pipelines on the Perlmutter system at NERSC, showcasing its capability to manage large-scale HPC applications efficiently. Additionally, we discuss the integration of a digital twin model for a simulated queue system related to a streaming system, using a Dynamic Bayesian Network (DBN) to enhance real-time monitoring and control, providing valuable insights into system performance and optimization strategies.

Thomas Jefferson National Accelerator Facility (JLab) has partnered with Energy Sciences Network (ESnet) to define and implement an edge to compute cluster computational load balancing acceleration architecture. The ESnet-JLab FPGA Accelerated Transport (EJFAT) architecture focuses on FPGA acceleration to address compression, fragmentation, UDP packet destination redirection (Network Address Translation (NAT)) and decompression and reassembly. EJFAT seamlessly integrates edge and cluster computing to support direct processing of streamed experimental data. This will directly benefit the JLab science program as well as data centers of the future that require high throughput and low latency for both time-critical data acquisition systems and data center workflows. The EJFAT project will be presented along with how it is synergistic with other DOE activities such as an Integrated Research Infrastructure (IRI), and recent results using data sources at JLab, an EJFAT LB at ESnet, and computational cluster resources at Lawrence Berkeley National Laboratory (LBNL).

The JIRIAF project aims to combine geographically diverse computing facilities into an integrated science infrastructure. This project starts by dynamically evaluating temporarily unallocated or idled compute resources from multiple providers. These resources are integrated to handle additional workloads without affecting local running jobs. This paper describes our approach to launch best-effort batch tasks that exploit these underutilized resources. Our system measures the real-time behavior of jobs running on a machine and learns to distinguish typical performance from outliers. Unsupervised ML techniques are used to analyze hardware-level performance measures, followed by a real-time crosscorrelation analysis to determine which applications cause performance degradation. We then facilitate bad behavior by throttling these processes. We demonstrate that problematic performance interference can be detected and acted on, which makes it possible to continue to share resources between applications and simultaneously maintain high utilization levels in a computing cluster. For a case study, we relocated the CLAS12 data processing workflow to a remote data processing facility, preventing file migration and temporal data persistency.

With the exponential growth in the volume and complexity of data generated at high-energy physics and nuclear physics research facilities, there is an imperative demand for innovative strategies to process this data in real or near-real-time. Given the surge in the requirement for high-performance computing, it becomes pivotal to reassess the adaptability of current data processing architectures in integrating new technologies and managing streaming data. This paper introduces the ERSAP framework, a modern solution that synergizes flow-based programming with the reactive actor model, paving the way for distributed, reactive, and high performance in data stream processing applications. Additionally, we unveil a novel algorithm focused on time-based clustering and event identification in data streams. The efficacy of this approach is further exemplified through the data-stream processing outcomes obtained from the recent beam tests of the EIC prototype calorimeter at DESY.

The 26th International Conference on Computing in High Energy and Nuclear Physics (CHEP), organized by Jefferson Lab, took place in Norfolk, Virginia from 5–11 May 2023. The conference attracted 581 registered participants from 28 different countries. There were scientific presentations made over the 5 days of the conference. These were divided between 20 long talks and 2 keynotes, which were presented in plenary sessions; 450+ short talks, which were presented in parallel sessions; and 140+ posters split over two dedicated sessions.

The Thomas Jefferson National Accelerator Facility collaborates with Lawrence Berkeley National Lab to implement a dynamic UDP load balancer (LB) for high-throughput scientific data processing. This study employs a simulation to compare the efficacy of Proportional, Integrative, Derivative (PID) controllers and Q-Learning based controllers for configuring the load balancer. Two cluster configurations, homogeneous and heterogeneous, were examined. The simulation results indicate that PID control is superior in both configurations. In homogeneous clusters, PID achieved a 50% reduction in aggregate queue levels and maintained an even distribution across computational nodes (CNs). In contrast, Q-Learning was less effective in heterogeneous environments, exacerbating queue levels compared to the no-control case and failing to achieve balance across the cluster. Our findings suggest that PID control should be used for the ESnet-JLab FPGA Accelerated Transport (EJFAT) system.

This article presents a framework based on a flow-based programming (FBP) paradigm to design data-stream processing applications for Nuclear Physics (NP). The developed framework encourages a functional decomposition of the overall data-processing application into small monofunctional artifacts that are easy to understand, develop, and debug. The fact that these artifacts (actors) are programmatically independent means that they can be scaled and optimized independently, which is difficult for monolithic application components. One of the advantages of this approach is fault tolerance, where independent actors can come and go in the data stream without stopping or crashing the entire application. Because actors are loosely coupled and data carries context, they can run in heterogeneous environments and utilize wide-ranging accelerators. This article describes the main design concepts of this framework, presenting a proof-of-concept application and the results of processing on-beam calorimeter streaming data.

Over several months, representatives from the U.S. Department of Energy (DOE) Office of Science and National Institutes of Health (NIH) had a number of meetings that lead to the conclusion that innovations in the Nation's health care could be realized by more directed interactions between NIH and DOE. It became clear that the expertise amassed and instrumentation advances developed at the DOE physical science laboratories to enable cutting-edge research in particle physics could also feed innovation in medical healthcare. To meet their scientific mission, the DOE laboratories created advances in such technologies as particle beam generation, radioisotope production, high-energy particle detection and imaging, superconducting particle accelerators, superconducting magnets, cryogenics, high-speed electronics, artificial intelligence, and big data. To move forward, NIH and DOE initiated the process of convening a joint workshop which occurred on July 12th and 13th, 2021. This Special Report presents a summary of the findings of the collaborative workshop and introduces the goals of the next one.

To increase the science rate for high data rates/volumes, Thomas Jefferson National Accelerator Facility (JLab) has partnered with Energy Sciences Network (ESnet) to define an edge to data center traffic shaping / steering transport capability featuring data event-aware network shaping and forwarding. The keystone of this ESnet JLab FPGA Accelerated Transport (EJFAT) is the joint development of a dynamic compute work Load Balancer (LB) of UDP streamed data. The LB is a suite consisting of a Field Programmable Gate Array (FPGA) executing the dynamically configurable, low fixed latency LB data plane featuring real-time packet redirection at high throughput, and a control plane running on the FPGA host computer that monitors network and compute farm telemetry in order to make dynamic decisions for destination compute host redirection / load balancing. The LB provides for three forms of scaling. It provides horizontal scale by adding more FPGAs for increased bandwidth. Second it scales out to the number of core compute hosts independent of the number of source DAQs. Thirdly it allows for a flexible number of CPUs and threads per host, treating each receiving thread as an independent LB destination. The LB provides seamless integration of edge / core computing to support direct experimental data processing. Immediate use will be at JLab science programs and others such as the EIC (Electron Ion Collider). Data centers of the future will need high throughput and low latency for both live streamed and recorded data for running experiment data acquisition analysis and data center use cases. EJ-FAT is a development for production use within DOE. When completed, it will have an operational impact for integrated research infrastructure as called for in [8], [9], and [10]. It demonstrates a new load balancing architecture, when compared with prior solutions like Server Load Balancing.

This report describes the physics case, the resulting detector requirements, and the evolving detector concepts for the experimental program at the Electron-Ion Collider (EIC). The EIC will be a powerful new high-luminosity facility in the United States with the capability to collide high-energy electron beams with high-energy proton and ion beams, providing access to those regions in the nucleon and nuclei where their structure is dominated by gluons. Moreover, polarized beams in the EIC will give unprecedented access to the spatial and spin structure of the proton, neutron, and light ions. The studies leading to this document were commissioned and organized by the EIC User Group with the objective of advancing the state and detail of the physics program and developing detector concepts that meet the emerging requirements in preparation for the realization of the EIC. The effort aims to provide the basis for further development of concepts for experimental equipment best suited for the science needs, including the importance of two complementary detectors and interaction regions. This report consists of three volumes. Volume I is an executive summary of our findings and developed concepts. In Volume II we describe studies of a wide range of physics measurements and the emerging requirements on detector acceptance and performance. Volume III discusses general-purpose detector concepts and the underlying technologies to meet the physics requirements. These considerations will form the basis for a world-class experimental program that aims to increase our understanding of the fundamental structure of all visible matter

The CLAS12 Data Acquisition System was designed and built as part of the CLAS12 detector project in Hall B at Jefferson Laboratory. This article contains a full description of the system, including requirements, design, hardware, and software descriptions, as well as the achieved performance. The associated computing, network, and slow controls systems are also described.

Over the last decade advances in electronics, computing, and software have changed the assumptions upon which data acquisition system designs for nuclear physics experiments are based. This is true at Jefferson Lab (JLab) as well as at other laboratories. Looking forward to future experiments that are in various stages of planning we must reevaluate the nearand long-term course that the development of readout systems for JLab experiments will take. One technique that is growing in favor in the data acquisition community is streaming mode readout, whereby detectors are continuously read out in parallel streams of data. The goal of this document is to introduce streaming mode readout, discuss the advantages and disadvantages in the context of JLab experiments, and make comparisons to the pipelined trigger mode currently used by JLab experiments.

We have assembled a small-scale streaming data acquisition system based on the SAMPA front-end ASIC. We report on measurements performed on the SAMPA chip and preliminary cosmic ray data acquired from a Gas Electron Multiplier (GEM) detector read out using the SAMPA.

Imagine being able to predict — with unprecedented accuracy and precision — the structure of the proton and neutron, and the forces between them, directly from the dynamics of quarks and gluons, and then using this information in calculations of the structure and reactions of atomic nuclei and of the properties of dense neutron stars (NSs). Also imagine discovering new and exotic states of matter, and new laws of nature, by being able to collect more experimental data than we dream possible today, analyzing it in real time to feed back into an experiment, and curating the data with full tracking capabilities and with fully distributed data mining capabilities. Making this vision a reality would improve basic scientific understanding, enabling us to precisely calculate, for example, the spectrum of gravity waves emitted during NS coalescence, and would have important societal applications in nuclear energy research, stockpile stewardship, and other areas. This review presents the components and characteristics of the exascale computing ecosystems necessary to realize this vision.

The design of the Trigger Interface and Distribution (TID) board for the 12 GeV Upgrade at Thomas Jefferson National Accelerator Facility (TJNAF) is described. The TID board distributes a low jitter system clock, synchronized trigger, and synchronized multi-purpose SYNC signal. The TID also initiates readout for the data acquisition front-end crate. With the TID boards, a multi-crate system can be setup for large scale nuclear physics experiments. The TID board can be selectively populated as a Trigger Interface (TI) board, or a Trigger Distribution (TD) board for the 12 GeV upgrade experiments. When the TID is populated as a TI, The TID can be located in the VXS crate and distribute the CLOCK/TRIGGER/SYNC (CTS) through the VXS/P0 connector; it can also be located in the standard VME64 crate, and distribute the CTS through the VME/P2 connector or front panel connectors. It initiates the data acquisition for the front-end crate where the TI is positioned in. When the TID is populated as a TD, it fans out the CTS from the trigger supervisor to the front-end crates through optical fibres. The TD board monitors the trigger processing on the TI boards, and sends feedback to the Trigger Supervisor (TS) board for event readout flow control. A Field Programmable Gate Arrays (FPGA) is utilised on the TID board to provide programmability. The TID board was intensively tested on the bench. The TID production version has been released to industry for contract manufacturing.

Data from high-energy physics (HEP) experiments are collected with significant financial and human effort and are mostly unique. An inter-experimental study group on HEP data preservation and long-term analysis was convened as a panel of the International Committee for Future Accelerators (ICFA). The group was formed by large collider-based experiments and investigated the technical and organisational aspects of HEP data preservation. An intermediate report was released in November 2009 addressing the general issues of data preservation in HEP. This paper includes and extends the intermediate report. It provides an analysis of the research case for data preservation and a detailed description of the various projects at experiment, laboratory and international levels. In addition, the paper provides a concrete proposal for an international organisation in charge of the data management and policies in high-energy physics.