SH4DOW

The performance harvests what you feel. You are the dataset.

SH4DOW (Skyggen) is an immersive AI performance in which the machine builds its model of the audience from the audience itself. Visitors become participants in a live gamified system run from their phones during the show not knowing exactly what they are training.

The audience-facing system is a single-page application built around a joystick interface. On every touch move it streams a distance, angle, direction and intensity of the user’s response to whatever is shown on stage over WebSocket to the backend. The backend manages WebSocket connections, user state, and a database of emotional stimulus images scored along valence and arousal axes.

A set of HTTP control endpoints lets operators step through the performance arc in real time: advancing to the next image or question, eliminating users, inviting finalists on stage. An OSC output relays all joystick positions live to the visuals system, so the physical performance on stage is driven by data flowing in from the phones in the audience’s hands. A separate facial expression analysis service processes audience video and forwards emotion scores over the same OSC pipeline.

Shown at Ars Electronica (2020), Click Festival (2021), Dunkers Kulturhus, Helsingborg (2022), and Aveny-T Copenhagen (2023).

Made possible with grants from The Bikubenfonden, Danish Arts Foundation, AI LAB & Helsingør kommune