Vollautomatisiertes Ausführen einer Applikation auf Basis eines Bildschirmvideos

• Marko Gattringer

Screen videos are a popular way to spread information and knowledge. Screen videos are videos that show user interaction with a computer program. Most often, such screen videos are recorded to create a tutorial for a specific application. Today, there are already countless such screen videos and platforms on which they can be distributed. There are already very good tools that allow you to create such screen videos in a simple way. As interesting and helpful such videos are in many cases, it is often difficult to reproduce the shown workflow by yourself. Furthermore, videos are poorly or not at all searchable for specific user interactions. This would require a computer program to recognize and extract the different user interactions in a screen video. Current solutions to this problem usually only extract individual content from such videos, but ignore related actions and the shown workflow. This often prevents an effective and efficient automated use of such screen videos.

This thesis presents a prototype that automatically extracts a workflow from a screen video. With the help of an video player this workflow is applied to the live application. In this way, the workflow shown in the screen video can be reproduced. The entire task is broken down into two steps. First the video is analyzed and all shown actions are extracted. Then the extracted actions are reproduced. Due to the complexity of this task, the prototype presented in this thesis is limited to the detection of left mouse clicks. The prototype shall work at least for the Windows application called "Calculator". To recognize this mouse clicks two different variants are presented. Both variants are based on a neural network. To reproduce the extracted workflow the tool Sikuli is used. The prototype was tested with screen videos showing a workflow in the calculator app as well as with screen videos of other applications. These tests have shown that both variants for the detection of mouse clicks achieve very good results when the screen video shows a workflow in the calculator app. If a screen video shows a workflow in another application than the calculator, both mouse click detection solutions were not convincing. The tests for replaying the extracted workflow have shown that the application in which the workflow is shown, does not matter.