Project description

When two musicians improvise together, each listens and responds while forming expectations about how the other’s timing, phrasing, and expressive gestures will unfold. Most of these expectations come from the sound itself and from knowing a partner’s playing; body movement adds cues, especially when timing is uncertain. Two research areas have explored this largely separately. At IRCAM (the Institute for Research and Coordination in Acoustics/Music, a Paris-based center for music research and creation), co-improvisation systems such as Somax2 and Dicy2 respond to musicians mainly by listening to their audio or MIDI. Research on multimodal performance shows that musicians’ movements carry information about timing and interaction. This project brings the two together, asking whether performer-specific models of physiological and movement data improve a computational partner’s anticipation beyond audio and MIDI alone.

We synchronize audio, MIDI, MIDI Polyphonic Expression when available, gesture and motion capture, facial measurements, EEG (brain activity), and EMG (muscle activity). Models trained on each performer’s recordings predict pauses, phrase changes, and shifts in intensity using only data recorded before each event, so they can run live. The system responds over MIDI, including through a Steinway Spirio self-playing piano, and records the performer’s reaction, closing a feedback loop.