Audionce: An Audiovisual Improvisation Environment Co-Designed by Deaf and Hearing Designers

Alon Ilsar; Matt Hughes; Yuka Maruyama; Andrew Johnston

oral
Presence: remote
Duration: 16
Type: long
Session: Togethering

Abstract:

Audionce is an audiovisual installation designed to support Deaf and hearing musicians and non-musicians to experience collaborative musical improvisation through multimodal interaction. The installation challenges the assumption that improvisation is primarily an auditory practice by reframing musical agency through gesture, vibration, touch, and visual feedback. The system combines a physical array of loudspeakers and a subwoofer with a virtual `mirror’ environment constructed from point-cloud scans of the same speakers. Using depth-camera tracking, animated lines connect participants to the speakers as their presence creates audiovisual content. As participants move through the space, additional audiovisual layers emerge, including drones, spatially responsive ostinati, and tactile kick-drum manipulation activated through direct contact with speaker surfaces. This paper presents the design motivations, system architecture, interaction mappings, artistic reflections. Audionce proposes a model for inclusive musical interaction that treats improvisation as a shared, embodied, and multi-sensory experience rather than one constrained by hearing ability.