WorkflowDSL: Scalable workflow execution with provenance for data analysis applictions

Authors Tharidu Fernando
Nikita Gureev
Michael Zwick
Thomas Natschläger
Mihhail Matskin
Title WorkflowDSL: Scalable workflow execution with provenance for data analysis applictions
Booktitle Proceedings of the 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC 2018)
Type in proceedings
Publisher IEEE
ISBN 78-1-5386-2666-5
DOI 10.1109/COMPSAC.2018.00115
Month July
Year 2018
Pages 774-779
SCCH ID# 18003

Data analysis projects typically use different programming languages (from Python for prototyping to C++ for support of runtime constraints) at their different stages by different experts. This creates a need for a data processing framework that is re-usable across multiple programming languages and supports collaboration of experts. In this work, we discuss implementation of a framework which uses a Domain Specific Language (DSL), called WorkflowDSL, that enables domain experts to collaborate on fine-tuning workflows. The framework includes support for parallel execution without any specialized code. It also provides a provenance capturing framework that enables users to analyse past executions and retrieve complete lineage of any data item generated. Graph database is used for storing provenance data. Advantages of usage of a graph database compare to relational databases are demonstrated. Experiments which were performed using a real-world scientific workflow from the bioinformatics domain and industrial data analysis models show that users were able to execute workflows efficiently when using WorkflowDSL for workflow composition and Python for task implementations. Moreover, we show that capturing provenance data can be useful for analysing past workflow executions.