Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance

Tristan WU; Ruiji YU; Gus XIA

Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance
Image credit: Tristan WU; Ruiji YU; Gus XIA

Abstract:

While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. This paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. Specifically, we build a low-latency pipeline that captures brush motion with an attachable sensor and maps it to control signals for real-time symbolic music generation. Using a generative model, the system produces multi-track MIDI in performance settings, while brush-derived control signals constrain event timing and activate additional musical layers.The generated melody is then extended with real-time harmony and additional voices, and finally rendered through a DAW for live staging.

Calliphony contributes: (1) a performance-oriented prototype that uses calligraphic motion as an external control layer for a real-time symbolic music generation model, controlling note density, pitch constraints, and accompaniment-layer activation; and (2) a cross-modal performance scenario that extends calligraphy beyond a primarily visual practice into an audiovisual, AI-assisted setting.