A Web Interface for Real-Time Interaction with Machine Learning in Musical Performance

Hongzhe Zhu; Charles Martin

A Web Interface for Real-Time Interaction with Machine Learning in Musical Performance
Image credit: Hongzhe Zhu; Charles Martin

Abstract:

Interactive machine learning systems are increasingly enabling new forms of real-time musical collaboration between human performers and artificial intelligence, transforming how musicians access and manipulate performance data. However, many existing tools function as black boxes, obscuring the internal logic of the generative models and limiting the performer’s sense of agency and trust. This research addresses this challenge by developing a transparent, web-based interface for an interactive machine learning system which visualises the state of mixture density recurrent neural networks (MDRNNs) in real-time. We present the design and evaluation of this system, detailing how its architecture and two-way visualisation strategy support multiple interaction paradigms. An exploratory user study suggests that while the conceptual complexity of model training presents a learning curve for novices, the system’s real-time visual feedback can support user trust and enable expressive co-creation. We conclude that exposing the data provenance and generative processes of AI systems can help transform them from opaque tools into intelligible musical partners. This work contributes a framework for human-centered AI interface design in the NIME community, suggesting that transparency is an important design factor for creative agency in intelligent instruments.