Physical Computing in Music and Sound Art

The term physical computing describes the use of micro-controller-based solutions for connecting software processes to the physical world. Sensors for obtaining physical quantities and actuators for manipulating objects open up manifold applications in sound art and music. Due to the rise of low cost, easy to program development boards like the Arduino, creative projects have easy access to these possibilities, giving rise to a variety of individual interfaces, controllers and installation concepts. Many projects presented at conferences like NIME are based on such solutions.

The Arduino is the most popular and established platform. It will be used in the examples and tutorials in this section. Other examples in the following sections are based on the Teensy Board or the ESP32 , which are programmed in the same way as the Arduino.


Sensor-Based Projects

BinBong

The first version of the BinBong (von Coler et al, 2017) is a haptic musical interface, using a MIDI over USB wired communication. A Teensy board is used to capture the force applied to four valve-like mechanics and the wooden top. The goal of this research-oriented instrument is to investigate the expressive means of force-based control in monophonic electronic musical instruments.

Front view of the BinBong with valve mechanics and wooden top.

Front view of the BinBong with valve mechanics and wooden top.

Rear view of the BinBong with Teensy and octave switches.

Rear view of the BinBong with Teensy and octave switches.


BinBong MKII

Based on the findings of the BinBong, a second, wireless version was designed, using a RedBear Duo board with integrated WiFi communication (Schmied, 2018). It has been designed to control spatial sound synthesis (von Coler et al., 2020) with an additional IMU and is used within user studies on mapping and expressivity (von Coler, 2021).

BinBong MKII.

BinBong MKII.


GameTrak: The Gallows

The GameTrak is a motion controller for the Playstation, launched in 2004. Nowadays, it is obsolete as a gaming device and used devices are cheap. The actual control unit is basically a joystick with a string attached, allowing to measure displacement and lengthening via potentiometers:

GameTrak sensor.

The GameTrak is easily hacked in many ways and has thus been used in many experimental music projects and by laptop orchestras (Huberth & Nanou, 2016). It can be easily integrated into projects using PD, SC and other environments, while the potentiometers are very reliable. The follwing short demo video shows a small version of The Gallows with a simple PD sample synthesis patch:

Visit the Gallows Website for more information and audio recordings or get the code: https://github.com/anwaldt/The_Gallows


PS_Gloves

The PS_Gloves are a project by Paul Schuladen, developed between 2018 and 2023 and now being finalized in his master's thesis. They track posture and orientation of the hand, using two inertial measurement units (IMU) and various capacitive sensors at the finger tips. Within a final study, they are used to play physical models of string instruments.

/images/basics/ps_glove_1.jpeg

Actuator-Based Projects

Cybernetic-playing Guitar

Building a self-playing and tuning guitar in a student project in a 2011 class with Douglas Repetto at TU Berlin. These videos show only the isolated attempts at plucking the strings with selenoids and tuning them with servo motors:


Ton-Technik & Pot-Shot

In his installations Ton-Technik and Pot-Shot, Andre Bartetzki uses solenoids to excite clay pots. An Arduino Mega is used in combination with SuperCollider to trigger the selenoids in specific patterns:


References

2021

  • Henrik von Coler. A System for Expressive Spectro-spatial Sound Synthesis. PhD thesis, TU Berlin, 2021.
    [details] [BibTeX▼]

2020

  • Henrik von Coler, Steffen Lepa, and Stefan Weinzierl. User-defined mappings for spatial sound synthesis. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME). 2020.
    [details] [BibTeX▼]

2018

2017

  • Henrik von Coler, Gabriel Treindl, Hauke Egermann, and Stefan Weinzierl. Development and evaluation of an interface with four-finger pitch selection. In Proceedings of the 142nd Audio Engineering Society Convention. 2017.
    [details] [BibTeX▼]

2016

  • Madeline Huberth and Chryssie Nanou. Notation for motion tracking controllers: a gametrak case study. In Proceedings of the International Conference on New Interfaces for Musical Expression, 1. 2016.
    [details] [BibTeX▼]