PCM sound representation is good for sound storage and playing. It lets us operate sound samples like a piece of magnetic tape—to cut, shuffle its parts, reverse, and glue back together. Also it lets us change and measure the overall volume of the sound. But PCM is inadequate for more advanced sound analysis and processing. The reason being that humans cannot hear separate audio samples, only frequencies in sound in short time intervals. The collection of amplitudes of each frequency in a short time interval is called spectrum of the sound. Therefore, sound processing methods should work using frequencies-spectrum language. This differs sound processing from image and video processing as they work well with pixels independently.
In this section, we will not dip into the mathematical aspects of spectrum computing, but will learn how to compute it using the openFrameworks functions and use it in projects.
The spectrum in openFrameworks is calculated for sound, which is formed by playing samples using the ofSoundPlayer objects.
The spectrum is an array of float numbers. It is obtained using the ofSoundGetSpectrum( N ) function, where N is the number of spectrum bands. Normally, N is the power of two:
float *val = ofSoundGetSpectrum( 256 );
Here val is an array of size 256. First array items correspond to lower frequencies, and last array items correspond to higher frequencies. With increasing N, you will have a more detailed description of the spectrum, but the data accuracy in time will decrease. The values of the spectrum are normalized so you can think they lie in the range [0, 1], though for loud sounds, values can exceed 1.
Having a spectrum array, you can get its values and use these for changing the controlling parameters for physics and visualization in your project. It is a good idea to smooth the spectrum because it jitters. Also, when using it for crucial projects, you do not need to use one spectrum band but perform smoothing (filtration) spectrum values over a number of bands.
The technology of filtering the regions of a spectrum is widely used in the VJ software for detecting a track's BPM (beats per minute, or tempo), and tracking separate beats of drums and other instruments. If you need to do a really advanced sound analysis of the music track for visualization, it may be a good idea to use Max/MSP, VDMX, or any other VJ software for analysis, and then send its result to your openFrameworks project via the OSC protocol. For more details on using the OSC protocol, see Chapter 11, Networking.
This is an example of an audio-reactive visual project. We are going to play a music track, get its spectrum, and use it for controlling point cloud parameters. So the cloud is rendered on the screen and shakes synchronously with drum beats in the music.
The example is based on the emptyExample project in openFrameworks. Before running it, copy the surface.wav file into the bin/data folder of your project.
In the testApp.h file, in the class testApp declaration, add declarations of a sound sample:
ofSoundPlayer sound; //Sound sample
Now let's consider the testApp.cpp file. For simplicity, we place the constants and variables not in class testApp definition, but right into the cpp file, after the #include "testApp.h" line:
const int N = 256; //Number of bands in spectrum float spectrum[ N ]; //Smoothed spectrum values float Rad = 500; //Cloud radius parameter float Vel = 0.1; //Cloud points velocity parameter int bandRad = 2; //Band index in spectrum, affecting Rad value int bandVel = 100; //Band index in spectrum, affecting Vel value const int n = 300; //Number of cloud points //Offsets for Perlin noise calculation for points float tx[n], ty[n]; ofPoint p[n]; //Cloud's points positions float time0 = 0; //Time value, used for dt computing
You can see that the spectrum is stored in the spectrum array, with size N = 256. Cloud has two control parameters—radius Rad and velocity Vel. Radius depends on the spectrum band bandRad = 2, and velocity depends on spectrum band bandVel = 100. These bands were selected specifically for the given music track, so Rad and Vel jump up on the base drum and snare drum beats respectively. Visually, the cloud expands on the base drum beat, and the points in the cloud begin to shuffle on the snare drum beat. The cloud is made from array points p, with size n = 300. Points are moved by Perlin noise (see more details in Appendix B, Perlin Noise).
The setup() function does sound sample loading and sets Perlin noise offsets for points initialization:
void testApp::setup(){
//Set up sound sample
sound.loadSound( "surface.wav" );
sound.setLoop( true );
sound.play();
//Set spectrum values to 0
for (int i=0; i<N; i++) {
spectrum[i] = 0.0f;
}
//Initialize points offsets by random numbers
for ( int j=0; j<n; j++ ) {
tx[j] = ofRandom( 0, 1000 );
ty[j] = ofRandom( 0, 1000 );
}
}The update() function gets the spectrum of the currently played sound, computes its smoothed values to the spectrum array, and recalculates the Rad and Vel parameters. Finally, it calculates new point positions:
void testApp::update(){
//Update sound engine
ofSoundUpdate();
//Get current spectrum with N bands
float *val = ofSoundGetSpectrum( N );
//We should not release memory of val,
//because it is managed by sound engine
//Update our smoothed spectrum,
//by slowly decreasing its values and getting maximum with val
//So we will have slowly falling peaks in spectrum
for ( int i=0; i<N; i++ ) {
spectrum[i] *= 0.97; //Slow decreasing
spectrum[i] = max( spectrum[i], val[i] );
}
//Update particles using spectrum values
//Computing dt as a time between the last
//and the current calling of update()
float time = ofGetElapsedTimef();
float dt = time - time0;
dt = ofClamp( dt, 0.0, 0.1 );
time0 = time; //Store the current time
//Update Rad and Vel from spectrum
//Note, the parameters in ofMap's were tuned for best result
//just for current music track
Rad = ofMap( spectrum[ bandRad ], 1, 3, 400, 800, true );
Vel = ofMap( spectrum[ bandVel ], 0, 0.1, 0.05, 0.5 );
//Update particles positions
for (int j=0; j<n; j++) {
tx[j] += Vel * dt; //move offset
ty[j] += Vel * dt; //move offset
//Calculate Perlin's noise in [-1, 1] and
//multiply on Rad
p[j].x = ofSignedNoise( tx[j] ) * Rad;
p[j].y = ofSignedNoise( ty[j] ) * Rad;
}
}The draw() function draws a spectrum and the cloud. Cloud's points are rendered as small circles. Additionally, pairs of points with distance less than the threshold dist = 40 are joined by a line segment:
void testApp::draw(){
ofBackground( 255, 255, 255 ); //Set up the background
//Draw background rect for spectrum
ofSetColor( 230, 230, 230 );
ofFill();
ofRect( 10, 700, N * 6, -100 );
//Draw spectrum
ofSetColor( 0, 0, 0 );
for (int i=0; i<N; i++) {
//Draw bandRad and bandVel by black color,
//and other by gray color
if ( i == bandRad || i == bandVel ) {
ofSetColor( 0, 0, 0 ); //Black color
} else {
ofSetColor( 128, 128, 128 ); //Gray color
}
ofRect( 10 + i * 5, 700, 3, -spectrum[i] * 100 );
}
//Draw cloud
//Move center of coordinate system to the screen center
ofPushMatrix();
ofTranslate( ofGetWidth() / 2, ofGetHeight() / 2 );
//Draw cloud's points
ofSetColor( 0, 0, 0 );
ofFill();
for (int i=0; i<n; i++) {
ofCircle( p[i], 2 );
}
//Draw lines between near points
float dist = 40; //Threshold parameter of distance
for (int j=0; j<n; j++) {
for (int k=j+1; k<n; k++) {
if ( ofDist( p[j].x, p[j].y, p[k].x, p[k].y )
< dist ) {
ofLine( p[j], p[k] );
}
}
}
//Restore coordinate system
ofPopMatrix();
}When running this example, you will hear a music track and will see the moving point cloud in the center of the screen. At the bottom of the screen, you will see the sound spectrum:
You can see that spectrum bands bandRad = 2 and bandVel = 100 are drawn in black (numeration from 0). Note that band 2 jumps on the base drum beat, and band 100 jumps on the snare drum beat, but in a lesser range, and the cloud extends and shuffles in correspondence to these beats.
You can extend the example by associating the radius and color of the point's circles with some spectrum bands.