Imagined Movement as Sonic Gesture: Auditory Expression from a Deep Learning-Based Motion Decoding BCI

Niall McShane; Karl McCreadie; Attila Korik; Damien Coyle

Imagined Movement as Sonic Gesture: Auditory Expression from a Deep Learning-Based Motion Decoding BCI
Image credit: Niall McShane; Karl McCreadie; Attila Korik; Damien Coyle

Abstract:

Continuous motion trajectory decoding (MTD) brain-computer interfaces (BCIs) translate imagined limb movements into continuous control signals, traditionally presented through visual feedback such as virtual limbs. This paper extends the multimodal expressivity of MTD-BCIs by introducing embodied sonification as a primary interaction modality. Using previously trained CNN-LSTM decoders for three-dimensional imagined arm movement, we mapped decoded motion and velocity signals in real time to a layered granular synthesis system. The framework employs velocity magnitude to modulate textural density and spectral characteristics, temporal accumulation to shape harmonic evolution, and rest-state detection to define acoustic boundaries between imagined gesture phases. Rather than treating sound as supplementary feedback, this approach positions decoded motion as sonic gesture and a continuous expressive articulation of imagined movement grounded in embodied cognition principles. The multi-layered synthesis architecture demonstrates that complex, many-to-many parameter mappings preserve perceptual coherence between movement dynamics and auditory change, creating a unified motion-audio-visual loop. This proof-of-concept establishes sonification as a viable interaction paradigm for motion-based BCIs, with implications for expressive musical performance, accessible sound-based control, and neurocognitive research into multimodal feedback in embodied human-computer interaction.