Sounds from Mismatch: Sensorimotor Prediction Error as Sonic Material in Augmented Reality

Domenico Stefani; Alberto Boem; Luca Turchet

Sounds from Mismatch: Sensorimotor Prediction Error as Sonic Material in Augmented Reality
Image credit: Domenico Stefani; Alberto Boem; Luca Turchet

Abstract:

Sounds from Mismatch (SfM) is an augmented reality (AR) sound experience in which sound emerges from the mismatch between a user’s bodily actions and the system’s expectations of those actions. SfM builds an internal model of user interactions by maintaining individual predictors for future positions of multiple tracked hand joints. Rather than treating prediction error solely as a quantity to be minimized, SfM frames sensorimotor and expectation mismatch as an expressive material for interaction. Using hand tracking in AR, the system generates sound from the dynamic discrepancy between the user’s movements and a continuously updated “ghost” representation of anticipated gestures. This mismatch is mapped to granular sound textures, producing an embodied auditory experience that unfolds through exploration rather than control. SfM invites users to attend to the shifting relationship between visual perception, proprioception, and action, foregrounding anticipation and deviation as core elements of musical interaction. This paper describes the conceptual framing, interaction design, and technical implementation of SfM, and discusses its implications for designing embodied AR sound experiences based on expectation rather than performative accuracy.