Physical Computing in Music and Sound Art

Henrik von Coler

The term physical computing describes the use of micro-controller-based solutions for connecting software processes to the physical world. Sensors for obtaining physical quantities and actuators for manipulating objects open up manifold applications in sound art and music. Due to the rise of low cost, easy to program development boards like the Arduino, creative projects have easy access to these possibilities, giving rise to a variety of individual interfaces, controllers and installation concepts. Many projects presented at conferences like NIME are based on such solutions.

The Arduino is the most popular and established platform. It will be used in the examples and tutorials in this section. Other examples in the following sections are based on the Teensy Board or the ESP32 , which are programmed in the same way as the Arduino.

Sensor-Based Projects

BinBong

The first version of the BinBong (von Coler et al, 2017) is a haptic musical interface, using a MIDI over USB wired communication. A Teensy board is used to capture the force applied to four valve-like mechanics and the wooden top. The goal of this research-oriented instrument is to investigate the expressive means of force-based control in monophonic electronic musical instruments.

Rear view of the BinBong with Teensy and octave switches.

BinBong MKII

Based on the findings of the BinBong, a second, wireless version was designed, using a RedBear Duo board with integrated WiFi communication (Schmied, 2018). It has been designed to control spatial sound synthesis (von Coler et al., 2020) with an additional IMU and is used within user studies on mapping and expressivity (von Coler, 2021).

GameTrak: The Gallows

The GameTrak is a motion controller for the Playstation, launched in 2004. Nowadays, it is obsolete as a gaming device and used devices are cheap. The actual control unit is basically a joystick with a string attached, allowing to measure displacement and lengthening via potentiometers:

The GameTrak is easily hacked in many ways and has thus been used in many experimental music projects and by laptop orchestras (Huberth & Nanou, 2016). It can be easily integrated into projects using PD, SC and other environments, while the potentiometers are very reliable. The follwing short demo video shows a small version of The Gallows with a simple PD sample synthesis patch:

Visit the Gallows Website for more information and audio recordings or get the code: https://github.com/anwaldt/The_Gallows

PS_Gloves

The PS_Gloves are a project by Paul Schuladen, developed between 2018 and 2023 and now being finalized in his master's thesis. They track posture and orientation of the hand, using two inertial measurement units (IMU) and various capacitive sensors at the finger tips. Within a final study, they are used to play physical models of string instruments.

Actuator-Based Projects

Cybernetic-playing Guitar

Building a self-playing and tuning guitar in a student project in a 2011 class with Douglas Repetto at TU Berlin. These videos show only the isolated attempts at plucking the strings with selenoids and tuning them with servo motors:

Ton-Technik & Pot-Shot

In his installations Ton-Technik and Pot-Shot, Andre Bartetzki uses solenoids to excite clay pots. An Arduino Mega is used in combination with SuperCollider to trigger the selenoids in specific patterns:

References

2021

Henrik von Coler. A System for Expressive Spectro-spatial Sound Synthesis. PhD thesis, TU Berlin, 2021.
[details] [BibTeX▼]

@phdthesis{voncoler2021asystem,
    author = "von Coler, Henrik",
    date = "2021-03-25",
    institution = "TU Berlin",
    school = "TU Berlin",
    title = "A System for Expressive Spectro-spatial Sound Synthesis",
    year = "2021"
}

2020

Henrik von Coler, Steffen Lepa, and Stefan Weinzierl. User-defined mappings for spatial sound synthesis. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME). 2020.
[details] [BibTeX▼]

@inproceedings{voncoler2020user,
    author = "von Coler, Henrik and Lepa, Steffen and Weinzierl, Stefan",
    booktitle = "{Proceedings of the International Conference on New Interfaces for Musical Expression (NIME)}",
    location = "Birmingham, UK",
    title = "User-Defined Mappings for Spatial Sound Synthesis",
    year = "2020"
}

2018

Anton Schmied. Developement and stress testing of the BinBong MKII. Master's thesis, TU Berlin, Berlin, Germany, 2018.
[details] [BibTeX▼]

@mastersthesis{schmied2018development,
    author = "Schmied, Anton",
    address = "Berlin, Germany",
    school = "TU Berlin",
    title = "Developement and Stress Testing of the {BinBong MKII}",
    year = "2018"
}

2017

Henrik von Coler, Gabriel Treindl, Hauke Egermann, and Stefan Weinzierl. Development and evaluation of an interface with four-finger pitch selection. In Proceedings of the 142nd Audio Engineering Society Convention. 2017.
[details] [BibTeX▼]

@inproceedings{voncoler2017design,
    author = "von Coler, Henrik and Treindl, Gabriel and Egermann, Hauke and Weinzierl, Stefan",
    title = "Development and Evaluation of an Interface with Four-Finger Pitch Selection",
    booktitle = "Proceedings of the 142nd Audio Engineering Society Convention",
    year = "2017",
    location = "Berlin, Germany"
}

2016

Madeline Huberth and Chryssie Nanou. Notation for motion tracking controllers: a gametrak case study. In Proceedings of the International Conference on New Interfaces for Musical Expression, 1. 2016.
[details] [BibTeX▼]

@inproceedings{huberth2016notation,
    author = "Huberth, Madeline and Nanou, Chryssie",
    booktitle = "Proceedings of the International Conference on New Interfaces for Musical Expression",
    pages = "1",
    title = "Notation for motion tracking controllers: A gametrak case study",
    year = "2016"
}