{"id":194,"date":"2026-11-01T16:39:47","date_gmt":"2026-11-01T16:39:47","guid":{"rendered":"https:\/\/research.bowdoin.edu\/experimental-multimedia-art\/?page_id=194"},"modified":"2026-11-01T17:42:34","modified_gmt":"2026-11-01T17:42:34","slug":"human-machine-co-creation","status":"publish","type":"page","link":"https:\/\/research.bowdoin.edu\/experimental-multimedia-art\/human-machine-co-creation\/","title":{"rendered":"Human\u2013Machine Co-Creation"},"content":{"rendered":"
\n
All research <\/span><\/a>ONGOING RESEARCH<\/span><\/div>\n
\n
\n

Studies in human\u2013machine co-creative collaborations<\/p>\n

HUMAN\u2013MACHINE
CO-CREATION<\/h1>\n
\n

Project description<\/h2>\n

When two musicians improvise together, each listens and responds while forming expectations about how the other’s timing, phrasing, and expressive gestures will unfold. Most of these expectations come from the sound itself and from knowing a partner’s playing; body movement adds cues, especially when timing is uncertain. Two research areas have explored this largely separately. At IRCAM<\/a> (the Institute for Research and Coordination in Acoustics\/Music, a Paris-based center for music research and creation), co-improvisation systems such as Somax2<\/a> and Dicy2<\/a> respond to musicians mainly by listening to their audio or MIDI. Research on multimodal performance shows that musicians’ movements carry information about timing and interaction. This project brings the two together, asking whether performer-specific models of physiological and movement data improve a computational partner’s anticipation beyond audio and MIDI alone.<\/p>\n

We synchronize audio, MIDI, MIDI Polyphonic Expression when available, gesture and motion capture, facial measurements, EEG (brain activity), and EMG (muscle activity). Models trained on each performer’s recordings predict pauses, phrase changes, and shifts in intensity using only data recorded before each event, so they can run live. The system responds over MIDI, including through a Steinway Spirio self-playing piano, and records the performer’s reaction, closing a feedback loop.<\/p>\n<\/div>\n<\/div>\n

Proposed closed-loop architecture<\/figcaption>
<\/span>Performer improvises<\/a>Capture and align musical
and bodily signals<\/a>Extract features and apply
a personalized model<\/a>Generate a musical response<\/a>The performer hears
and responds<\/a><\/div>\n
\n

The performer improvises using a MIDI-capable instrument. Audio and selected bodily signals provide complementary observations.<\/p>\n

\n
\n

Research team<\/h2>\n

CEMA \u00b7 Bowdoin College<\/p>\n<\/div>\n

\n
\"Badie<\/a><\/p>\n

Principal investigator<\/p>\n

Badie Khaleghian<\/a><\/h3>\n

Music \u00b7 CEMA<\/p>\n<\/article>\n

\"Jeov\u00e1<\/a><\/p>\n

Co-principal investigator<\/p>\n

Jeov\u00e1 Farias<\/a><\/h3>\n

Computer Science<\/p>\n<\/article>\n

\n
\"Jintae<\/div>\n

Student researcher<\/p>\n

Jintae Park<\/h3>\n

Bowdoin College<\/p>\n<\/article>\n

\n
\"Shahd<\/div>\n

Student researcher<\/p>\n

Shahd Hekal<\/h3>\n

Bowdoin College<\/p>\n<\/article>\n

\n
\"Johann<\/div>\n

Student researcher<\/p>\n

Johann Ocampo<\/h3>\n

Bowdoin College<\/p>\n<\/article>\n<\/div>\n

Funded by<\/span>Hastings Initiative for AI and Humanity<\/a>Bowdoin College<\/span><\/div>\n<\/section>\n
\n