Performing with the inclusive machine: An interdisciplinary roadmap for the design of AI collaborative musical instruments

Pablo Mollenhauer; Alejandra Pérez Núñez

Performing with the inclusive machine: An interdisciplinary roadmap for the design of AI collaborative musical instruments
Image credit: Pablo Mollenhauer; Alejandra Pérez Núñez

Abstract:

Computer musical instruments that use traditional programming paradigms have leaned towards predictability and direct causality based on deterministic mappings, rule-based synthesis, and explicit parameter control. Many studies have implemented instruments using machine learning for mapping and sound synthesis, but yet, few studies have explored the implications of these technologies on agency, causality, and power relations in the design process and performance practice. This paper explores the political-agential balances and aesthetics that emerge when machine learning technologies are integrated into the design process of computer music instruments. The hypothesis is that control and agency is distributed by machine learning’s algorithms, in which designer and performer, have to navigate through the feedback system made of gestures, latent space and spatialized sound. The methodology comprises several chained machine learning stages: a single pose tracking system driven by pre-trained models, a supervised neural network for mapping parameters of the latent space of models for sound synthesis. The design stage is informed by first-person accounts of experience using micro-phenomenology.