WorkflowDSL: Scalable workflow execution with provenance for data analysis applictions

Autoren Tharidu Fernando
Nikita Gureev
Michael Zwick
Thomas Natschläger
Mihhail Matskin
Editoren
Titel WorkflowDSL: Scalable workflow execution with provenance for data analysis applictions
Buchtitel Proceedings of the 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC 2018)
Typ in Konferenzband
Verlag IEEE
ISBN 78-1-5386-2666-5
DOI 10.1109/COMPSAC.2018.00115
Monat July
Jahr 2018
Seiten 774-779
SCCH ID# 18003
Abstract

Data analysis projects typically use different programming languages (from Python for prototyping to C++ for support of runtime constraints) at their different stages by different experts. This creates a need for a data processing framework that is re-usable across multiple programming languages and supports collaboration of experts. In this work, we discuss implementation of a framework which uses a Domain Specific Language (DSL), called WorkflowDSL, that enables domain experts to collaborate on fine-tuning workflows. The framework includes support for parallel execution without any specialized code. It also provides a provenance capturing framework that enables users to analyse past executions and retrieve complete lineage of any data item generated. Graph database is used for storing provenance data. Advantages of usage of a graph database compare to relational databases are demonstrated. Experiments which were performed using a real-world scientific workflow from the bioinformatics domain and industrial data analysis models show that users were able to execute workflows efficiently when using WorkflowDSL for workflow composition and Python for task implementations. Moreover, we show that capturing provenance data can be useful for analysing past workflow executions.