DATACLOUDDSL: Textual and Visual Presentation of Big Data Pipelines

This paper describes the DATACLOUDDSL language and the DEF-PIPE tool for describing Big Data pipelines. DATACLOUDDSL has both a textual and a visual form and supports requirements obtained both from analyzing existing data pipeline specification tools and from interviews with relevant industrial actors. Particularly, DATACLOUDDSL supports (i) separation of concerns between design and run-time issues, (ii) reuse of previously developed pipeline steps and pipelines in designing new pipelines, (iii) flexible data transfer between pipelines steps and containerization of pipelines and pipeline steps, and (iv) integration of description and simulation components in Big Data pipeline orchestration systems. Additionally, it provides an interface to the discovery and deployment tools of the DataCloud toolbox.