Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System

Jaeran Choi; Juhan Nam; Hikari Kuriyama; Gou Koutaki

Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System
Image credit: Jaeran Choi; Juhan Nam; Hikari Kuriyama; Gou Koutaki

Abstract:

This study presents a semi-automated robotic flute system that coordinates performance onset timing using a human performer’s gesture cues, and examines how such control influences experience in a human–robot ensemble. In the proposed system, the performer produces sound through breath while the robot actuates the flute’s keys via a servo-driven mechanism, establishing a shared-performance structure. A camera-based motion tracking system detects preparatory head gestures in real time and predicts intended onset timing using a gesture cue–onset ratio model.

We compared three conditions: timer-based onset, gesture cue-based onset with visual feedback, and gesture cue-based onset without visual feedback. Quantitative measures assessed onset asynchrony, and qualitative measures examined perceived partnership, agency, leadership, and trust. Results indicate that gesture cue-based control enhances the sense of partnership and performer agency, while timer-based control yields higher timing stability. These findings suggest that gesture-driven semi-automated musical robots can shift perception from playback device to ensemble partner.