Emotion and Expressivity in Music Performance: A Multimodal Approach

Natalia Kotsani; Spyros Kantarelis; Vasilis Lyberatos; Edmund Dervakos; Giorgos Stamou

Emotion and Expressivity in Music Performance: A Multimodal Approach
Image credit: Natalia Kotsani; Spyros Kantarelis; Vasilis Lyberatos; Edmund Dervakos; Giorgos Stamou

Abstract:

The relationship between musical performance, perceived, intended, and induced emotion by audience and musician is a complex topic that has been studied for centuries. In the era of Artificial Intelligence (AI), this topic becomes particularly important for the development of applications that assist in the study and creative aspects of musical performance, enhancing both the interpretative process and emotional expression. In this paper we present a protocol for creating datasets that contain live performances of music compositions, continuous and categorical emotion annotations from the audience and the performing musician, in addition to an array of biosignals recorded from performers and listeners. The protocol is designed to contain a variety of musical contexts, from rigid etudes simulating practice to free improvisation emulating pure expression. It is replicable and agnostic of instrumentation, and its main purpose is to facilitate the development of AI applications that will enhance musical expression. We provide the design and details of the protocol, preliminary results from its implementation with professional musicians and we discuss limitations and potential future research directions.