Calibration Processing At The European Xfel - Implementation And Concepts

The European X-ray Free Electron Laser is a high-intensity X-ray light source currently being constructed in the Hamburg region that will provide spatially coherent X-rays in the energy range between 0.25 keV - 25 keV. The machine will deliver 10 trains/s, consisting of up to 2700 pulses, with a 4.5 MHz repetition rate. The LPD, DSSC and AGIPD detectors are being developed to provide high dynamic range Mpixel imaging capabilities at the mentioned repetition rates. A consequence of these detectors' characteristics is that they generate raw data volumes of up to 12.8 Gbyte/s. In addition, the detectors' on-sensor memory-cell and multi-/non-linear gain architectures pose unique challenges in data correction and calibration, especially given the in the order of one billion calibration parameters. We present how these challenges are addressed for the AGIPD and LPD prototypes within XFEL's control and analysis framework Karabo, which integrates access to hardware conditions, acquisition settings and distributed computing. Implementation is mainly in Python, using self-optimizing (py)CUDA code, numpy and iPython.parallel to achieve near-real time performance for calibration application.