The ESO Data Processing System (EDPS) is a new software infrastructure to run the European Southern Observatory’s (ESO) science data processing modules (“pipeline recipes”) that is currently in an advanced status of development. These recipes are used for quality control purposes, unsupervised production of science and calibration products for the archive, and are embedded in an interactive data reduction system that is also offered to external science users. EDPS aims to unify all these use cases and thereby replace individual dedicated systems that are currently used at the European Southern Observatory (ESO). All pipelines are based on self-contained recipes with well-defined inputs, outputs, and parameters. They are strung together into different workflows and fed with data selected by pre-defined rules. Even though the dependencies of the recipes imply an execution cascade that is identical for all use cases, the actual workflows and data selection rules for the different use cases are significantly different. EDPS will be a Python based system that generates workflows for ESO’s science data processing use cases from a general specification for the dependencies of the recipes. By starting from a single specification for different workflows, EDPS will improve the reliability and reproducible of data processing. At the same time, EDPS is expected to drastically lower implementation and maintenance costs for the workflows.
原文链接
分享
引用
