Jupyter notebooks are executable documents that are displayed in a web browser. The notebook elements consist of human-authored contextual elements and computer code, and computer-generated output from executing the computer code. Such outputs can include tables and plots. The notebook elements can be executed interactively, and the whole notebook can be saved, re-loaded and re-executed, or converted to read-only formats such as HTML, LaTeX and PDF. Exploiting these characteristics, Jupyter notebooks can be used to improve the effectiveness of computational and data exploration, documentation, communication, reproducibility and re-usability of scientific research results. They also serve as building blocks of remote data access and analysis as is required for facilities hosting large data sets and initiatives such as the European Open Science Cloud (EOSC). In this contribution we report from our experience of using Jupyter notebooks for data analysis at research facilities, and outline opportunities and future plans.
EXPERIMENTS AT THE EUXFEL Dall'Antonia, Fabio; Beg, Marijan; Bergemann, Martin; Bielecki, Johan; Bondar, Valerii; Carinan, Cammille; Costa Junior, Raul; Danilevski, Cyril; Ehsan, Wajid; Esenov, Sergey; Fabbri, Riccardo; Flucke, Gero; Fullà-Marsà, Daniel; Giovanetti, Gabriele; Göries, Dennis; Hickin, David; Jarosiewicz, Tobiasz; Kamil, Ebad; Kirienko, Yury; Kirkwood, Henry; Klimovskaia, Anna; Kluyver, Thomas; Mamchyk, Denys; Michelat, Thomas; Mohacsi, Istvan; Parenti, Andrea; Rosca, Robert; Rück, Denivy; Santos, Hugo; Schaffer, Robert; Silenzi, Alessandro; Spirzewski, Michal; Trojanowski, Sebastian; Youngman, Christopher; Zhu, Jun; Mancuso, Adrian P.; Fangohr, Hans; Brockhauser, Sandor: European XFEL GmbH, Schenefeld, GER
The Karabo distributed control system has been developed to address the challenging requirements of the European X-ray Free Electron Laser facility, including complex and custom-made hardware, high data rates and volumes, and close integration of data analysis for distributed processing and rapid feedback. Karabo is a pluggable, distributed application management system forming a supervisory control and data acquisition environment as part of a distributed control system. Karabo provides integrated control of hardware, monitoring, data acquisition and data analysis on distributed hardware, allowing rapid control feedback based on complex algorithms. Services exist for access control, data logging, configuration management and situational awareness through alarm indicators. The flexible framework enables quick response to the changing requirements in control and analysis, and provides an efficient environment for development, and a single interface to make all changes immediately available to operators and experimentalists.
We describe the data analysis structure that is integrated into the Karabo framework to support scientific experiments and data analysis at European XFEL. The photon science experiments have a range of data analysis requirements, including online (i.e. near real-time during the actual measurement) and offline data analysis. The Karabo data analysis framework supports execution of automatic data analysis for routine tasks, supports complex experiment protocols including data analysis feedback integration to instrument control, and supports integration of external applications. The online data analysis is carried out using distributed and accelerator hardware (such as GPUs) where required to balance load and achieve near real-time data analysis throughput. Analysis routines provided by Karabo are implemented in C++ and Python, and make use of established scientific libraries. The XFEL control and analysis software team collaborates with users to integrate experiment specific analysis codes, protocols and requirements into this framework, and to make it available for the experiments and subsequent offline data analysis.
X-ray microscopy is a commonly used method especially in material science application, where the large penetration depth of X-rays is necessary for three-dimensional structural studies of thick specimens with high-Z elements. In this paper it is shown that full-field X-ray microscopy at 6.2 keV can be utilized for imaging of biological specimens with high resolution. A full-field Zernike phase-contrast microscope based on diffractive optics is used to study lipid droplet formation in hepatoma cells. It is shown that the contrast of the images is comparable with that of electron microscopy, and even better contrast at tender X-ray energies between 2.5 keV and 4 keV is expected.
Beamline P11 at PETRA III is dedicated to structural investigations of biological samples. It provides two experimental stations, one for macromolecular crystallography and one for X-ray microscopy. The microscope will provide full field Zernike phase contrast and scanning microscopy both in 2D and in tomographic mode. Full field microscopy with a field of view of 50 x 50 μm2 will allow to generate an overview of the sample and to select regions of interest for later inspection of the element distribution by X-ray fluorescence and diffraction in scanning mode. Central part of the microscope is an inhouse developed flexure based x,y,z scanner on top of a rotation stage. The scanner is operated in closed loop with piezo motors, has a travel range of 4 mm in horizontal and of 3 mm in vertical direction. With laser interferometers for closed loop operation a positioning accuracy of better than 5 nm is achieved in all directions. For precise sample rotation an in-vacuum air-bearing has been developed. An open bore in the center of the air-bearing allows cryogenic sample cooling by a cold He or N2 gas stream. Different optical elements such as beam defining pinholes, a condensor, zone plates, OSA, phase rings, etc. can be centered in the beam path by piezomotor driven x,y flexure elements mounted on a rail system which allows further positioning along the beam path. Different 2D detectors and two fluoresence detectors can be attached to the microscope.
Sessions C468Manganites with perovskite structure are the class of compounds referred as the rare earth manganites that is characterized by extremely interesting structural and physical properties and phenomena, including Colossal Magnetoresistance (CMR) [1].The structure of the RE 1-x M x MnO (RE= rare earth) oxides is close to that of the cubic type perovskite (CaTiO3), but the structure of manganites generally corresponds to a lower rhombohedral symmetry (LiNbO3) or orthorhombic (GdFeO3) structure [2] such as LaSrMnO3, S.G.Pnma.We present preliminary studies and results of the RESrMnO 3 system where substitutions RE=Dy,Yb for La in perovskite type structure were carried out.Samples were prepared by the solid-state reaction method in air at ambient pressure.Process of synthesis was followed by thermal analysis (TGA and DTA) and X-ray powder diffraction (XRD).Morphology of resultant samples have been observed by Scanning Electron Microscopy (SEM) and the stoichiometry has been analyzed by Electron Dispersive X-Ray Spectroscopy (EDX).
Agile Project Management (Agile PM), coupled with the DevOps concept, has been worked out as a fundamental approach in a highly uncertain and unpredictable environment to achieve mature software development and to efficiently support concurrent operation [1]. At the European XFEL[2], Agile PM and DevOps have been applied to provide adaptability and efficiency in the development and operation of its control system: Karabo[3,4]. In this context, the Control and Analysis Software Group (CAS) has developed in-house a management platform composed of the following macro-artefacts: (1) Agile Process; (2) Release Planning; (3) Testing Infrastructure; (4) Roll-out and Deployment Strategy; (5) Automated tools for Monitoring Control Points (i.e. Configuration Items[5]) and; (6) Incident Management[6]. The software engineering management platform is also integrated with User Relationship Management to establish and maintain a proper feedback loop with our