Tones: An Interactive Audio Generator

November 20, 2011

Artist Statement
Tones is an interactive software piece that generates musical patterns based on colour data from a live camera input. Visuals are fed from the camera and downscaled to a 10×24 pixel grid, where the darkest pixel of each column is marked in orange. Sound is continuously generated and reiterated based on the height of these marked pixels, where the height is either proportional to the pitch of a note or the volume of a track.

The work attempts to create a codependent relationship between two distinct forms of sensory communication, audio and visual, looking at how data can be repurposed and translated from one function to another. Marking the darkest pixel of each column enables users to plainly see how their presence affects the data provided to the computer, but they must uncover on their own how their position and movements contribute to the generated sound. Users are invited to explore how their interaction with the piece contributes to the sound generation without being given any definitive guideline or affordance, in turn creating both a generative output and generative user interaction.

Connection to Class Material
In Jim Campbell’s Control and Choice in Interactive Art, Campbell argues for interactive art that denies a precisely calculated response to the user’s actions (what he describes as “controlling” the work), posing the guiding question, “as a program, what can I measure?” Tones is centred around measuring and mapping arbitrary data sets, creating a large set of possible sound outcomes. The user’s interaction affects the data input in a passive manner – while the software will respond to the user’s presence and movements, it will respond without any user as well. In fact, it is possible that any combination of data read based on the user’s presence can also be read by capturing an empty room.

What is Generative Art? by Philip Galanter mentions electronic music and algorithmic composition as one kind of instance of generative art. I like to view Tones as a generative work more so than an interactive work, although it maintains characteristics of both. Galanter uses “complexity, order, and disorder” as general overarching traits of generative art – in particular, order and disorder are demonstrated in Tones: percussion tracks can be deliberately controlled by the user based on where they position their bodies (adjusting the volume of a particular track), while the algorithm for generating the melody and harmony are based around the randomized selection of dark values from the camera.

Galanter frequently mentions randomization as a typical (though not always necessary) attribute of generative works.

Technical Description
Tones uses a chain of data and arbitrary data relationships to translate input to output. A camera input is scaled down to 10×24 pixels and then stretched across the screen (creating blocks of “wide” pixels, the primary visual aesthetic of the piece). The decision to scale the work to 10×24 is based on sound graphs, where amplitude is expressed on the y-axis and frequency on the x-axis (the y- axis can represent values from -24dB to +24dB, which are used to measure the volume of the percussion tracks for example).

Using the pixel array from the camera feed, loops are used to compare pixel data in each column. The HSB colour mode is used to extract and compare the brightness value from each pixel, allowing the program to determine the darkest pixel in every column at any given time. These “dark pixels” are used to control the volume of the percussion tracks (where each track corresponds to a pre- defined column). Dark pixels are also mapped to frequency values, which are assigned to three different “instruments” (a sine, triangle and square wave representing melody, harmony, and bass). Musical patterns are generated every 16 beats by grabbing these frequencies from a randomly selected set of four pixels (the algorithm selects four x-coordinates, then searches for the darkest pixel in each of these columns to map and assign a frequency to each note in the progression). The outcome is an atonal musical composition that is regenerated every four measures, entirely interpreted from colour data.