An empirical evaluation on the applicability of the DALiuGE execution framework

The Square Kilometre Array (SKA) project is an international cooperation project to build the largest radio telescope worldwide. Data processing is one of the biggest challenges of building the SKA telescope. As a distributed execution framework, the Data Activated Liu Graph Engine (DALiuGE) was proposed to be one of the candidates for addressing the massive data of the SKA. DALiuGE has many distinctive features, but its actual ability to handle scientific data is still not clear. In this paper, we perform an objective evaluation of the usability of DALiuGE for the execution performance, developer workload, and implementation difficulty of porting the SAGECal to DALiuGE. The evaluation results show that the DALiuGE enables fast integration of astronomical software, but there are significant differences in the efficiency of different parallel granularities. Even with the deep optimization of the program, there is still a gap between the current DALiuGE and the traditional MPI in execution performance. Therefore, we come to a preliminary conclusion that the DALiuGE has no performance advantage in batch processing of massive data. At the same time, it may be more suitable for application scenarios with more customized computational tasks, such as SKA science regional centers. (C) 2021 Elsevier B.V. All rights reserved.