Towards score following In sheet music images

Autoren Matthias Dorfer
Andreas Arzt
Gerhard Widmer
Titel Towards score following In sheet music images
Buchtitel roceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR 2016)
Typ in Konferenzband
Monat August
Jahr 2016
Seiten 789-795
SCCH ID# 16102

This paper addresses the matching of short music audio snippets to the corresponding pixel location in images of sheet music. A system is presented that simultaneously learns to read notes, listens to music and matches the currently played music to its corresponding notes in the sheet. It consists of an end-to-end multi-modal convolutional neural network that takes as input images of sheet music and spectrograms of the respective audio snippets. It learns to predict, for a given unseen audio snippet (covering approximately one bar of music), the corresponding position in the respective score line. Our results suggest that with the use of (deep) neural networks – which have proven to be powerful image processing models – working with sheet music becomes feasible and a promising future research direction.