Automated data quality monitoring

Authors Lisa Ehrlinger
Wolfram Wöß
Title Automated data quality monitoring
Booktitle Proceedings of the 22nd Annual MIT International Conference on Information Quality (ICIQ 2017)
Type in proceedings
ISBN 978-0-692-05026-2
Month December
Year 2017
Pages 15-1 to 15-9
SCCH ID# 17048

Most existing methodologies agree that the assessment of data quality (DQ) is a cyclic process, which has to be carried out continuously. Nevertheless, the majority of DQ tools allow the evaluation of data sources only at specific points in time, and the automation and scheduling is therefore in the responsibility of the user. In contrast, automated DQ monitoring allows the evaluation of applied DQ improvements as well as the comparability between different system states. The reproducibility of DQ assessments is also an important topic for the scientific community in order to review algorithms that improve the DQ of an information system. We are developing a tool for DQ monitoring and our research covers the investigation of suitable DQ metrics for continuous monitoring as well as the development of a standardized approach to storing DQ assessment results over time. In addition, statistical methods to analyze and visualize the resulting time series data are selected and applied.