Live replay of screen videos: Automatically executing real applications as shown in recordings

Authors Rudolf Ramler
Marko Gattringer
Josef Pichler
Editors
Title Live replay of screen videos: Automatically executing real applications as shown in recordings
Booktitle Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution, and Reengineering (SANER'20)
Type in proceedings
Publisher IEEE
ISBN 978-1-7281-5143-4
Month February
Year 2020
Pages 664-665
SCCH ID# 20001
Abstract

Screencasts and videos with screen recordings are becoming an increasingly popular source of information for users to understand and learn about software applications. However, searching for answers to specific questions in screen videos is notoriously difficult due to the effort for locating specific events of interest and reproducing the application's state up to this event. To increase the efficiency when working with screen videos, we propose a solution for replaying recorded sequences shown in videos directly on live applications. In this paper, we describe the analysis of screen videos to automatically identify and extract user interactions and the construction of visual scripts, which are used to run the application in sync with replaying the video. Currently, a first prototype has been developed to demonstrate the technical feasibility of the approach. The paper provides an overview of the implemented solution concept and discusses technical challenges, open issues, as well as future application scenarios.