As a baseline template, let's copy over our AVPlaybackSoundController class from Chapter 9 into the Space Rocks! project. We will then gut the code to remove the things we don't need. We will also add a few new methods to make it easier to integrate with Space Rocks! In truth, it would probably be just as easy to use Apple's AVAudioPlayer directly in the BBSceneController, but I wanted a separate place to put the interruption delegate callbacks to keep things clean.

Starting with the AVPlaybackSoundController header, let's delete all the old methods. Then delete the separate speech and music player. In its place, we'll create a generic "stream" player. The idea is that if the game needs multiple streams, we can instantiate multiple instances of this class. I suppose this makes this class less of a "controller," but oh well.

The class also conformed to the AVAudioSessionDelegate protocol. We will let the OpenAL controller class continue to set up and manage the audio session, so we can delete this, too. We need a way to specify an arbitrary sound file, so we'll make a new designated initializer called initWithSoundFile:, which takes an NSString* parameter. Finally, we'll add some new methods and properties to control playing, pausing, stopping, volume, and looping. The modified AVPlaybackSoundController.h file should look like this:

#import <AVFoundation/AVFoundation.h>
#import <UIKit/UIKit.h>

@interface AVPlaybackSoundController : NSObject <AVAudioPlayerDelegate>
{
   AVAudioPlayer* avStreamPlayer;
}

@property(nonatomic, retain) AVAudioPlayer* avStreamPlayer;
@property(nonatomic, assign) NSInteger numberOfLoops;
@property(nonatomic, assign) float volume;

- (id) initWithSoundFile:(NSString*) sound_file_basename;

- (void) play;
- (void) pause;
- (void) stop;

@end

In our implementation, we delete almost everything except the AVAudioPlayerDelegate methods. We will purge the audioPlayerDidFinishPlaying:successfully:-specific implementation though. We then just implement the new methods. Most of them are direct pass-throughs to AVAudioPlayer. The one exception is our new initializer. This code will create our new AVAudioPlayer instance. We also copy and paste the file-detection code we use from the OpenAL section to locate a file without requiring an extension. The new AVPlaybackSoundController.m file looks like this:

#import "AVPlaybackSoundController.h"

@implementation AVPlaybackSoundController

@synthesize avStreamPlayer;


- (id) initWithSoundFile:(NSString*)sound_file_basename

{
   NSURL* file_url = nil;
   NSError* file_error = nil;

   // Create a temporary array containing the file extensions we want to handle.
   // Note: This list is not exhaustive of all the types Core Audio can handle.
   NSArray* file_extension_array = [[NSArray alloc]
     initWithObjects:@"caf", @"wav", @"aac", @"mp3", @"aiff", @"mp4", @"m4a", nil];
   for(NSString* file_extension in file_extension_array)
   {
       // We need to first check to make sure the file exists;
       // otherwise NSURL's initFileWithPath:ofType will crash if the file doesn't exist
       NSString* full_file_name = [NSString stringWithFormat:@"%@/%@.%@",
         [[NSBundle mainBundle] resourcePath], sound_file_basename, file_extension];
       if(YES == [[NSFileManager defaultManager] fileExistsAtPath:full_file_name])
       {
          file_url = [[[NSURL alloc] initFileURLWithPath:[[NSBundle mainBundle]
             pathForResource:sound_file_basename ofType:file_extension]] autorelease];
          break;
       }
    }
    [file_extension_array release];

    if(nil == file_url)
    {
       NSLog(@"Failed to locate audio file with basename: %@", sound_file_basename);
       return nil;
    }

    self = [super init];
    if(nil != self)
    {
       avStreamPlayer = [[AVAudioPlayer alloc] initWithContentsOfURL:file_url
          error:&file_error];
       if(file_error)
       {
          NSLog(@"Error loading stream file: %@", [file_error localizedDescription]);
       }
       avStreamPlayer.delegate = self;
       // Optional: Presumably, the player will start buffering now instead of on play.
       [avStreamPlayer prepareToPlay];
    }
    return self;
}

- (void) play
{
   [self.avStreamPlayer play];

}

- (void) pause
{
   [self.avStreamPlayer pause];
}

- (void) stop
{
   [self.avStreamPlayer stop];
}

- (void) setNumberOfLoops:(NSInteger)number_of_loops
{
   self.avStreamPlayer.numberOfLoops = number_of_loops;
}

- (NSInteger) numberOfLoops
{
   return self.avStreamPlayer.numberOfLoops;
}

- (void) setVolume:(float)volume_level
{
   self.avStreamPlayer.volume = volume_level;
}

- (float) volume
{
   return self.avStreamPlayer.volume;
}

- (void) dealloc
{
   [avStreamPlayer release];
   [super dealloc];
}

#pragma mark AVAudioPlayer delegate methods

- (void) audioPlayerDidFinishPlaying:(AVAudioPlayer*)which_player
    successfully:(BOOL)the_flag
{
}

- (void) audioPlayerDecodeErrorDidOccur:(AVAudioPlayer*)the_player
    error:(NSError*)the_error
{
   NSLog(@"AVAudioPlayer audioPlayerDecodeErrorDidOccur: %@", [the_error
         localizedDescription]);
}

- (void) audioPlayerBeginInterruption:(AVAudioPlayer*)which_player
{
}

- (void) audioPlayerEndInterruption:(AVAudioPlayer*)which_player

{
   [which_player play];
 }
@end

We probably should do something interesting with audioPlayerDidFinishPlaying:successfully: to integrate it with the rest of the game engine, similar to what we did with the OpenAL resource manager. But since this example is concerned only with playing background music, which we will infinitely loop, we won't worry about that here.

Now for the integration. Remember to add AVFoundation.framework to the project so it can be linked. In BBConfiguration.h, we'll add our background music file. Prior to this point, I've been making my own sound effects, or in a few instances, finding public domain stuff on the Internet. But creating a good soundtrack exceeds my talents, and it is very hard to find one in the public domain or free for commercial use. Fortunately for us, Ben Smith got permission from the musician he used for Snowferno to reuse one of his soundtracks for Space Rocks! for our book. Please do not use this song for your own projects, as this permission does not extend beyond this book's example.

Attribution Credits:
Music by Michael Shaieb
© Copyright 2009 FatLab Music
From "Snowferno" for iPhone/iPod touch

Also, for demonstration purposes, I have compressed the soundtrack using AAC into an .m4a container file. This is to demonstrate that we can use restricted compression formats for our audio files in our game. Chapter 11 covered audio file formats. Remember that the hardware decoder can handle only one file in compression format at a time. While compressing it, I converted it down to 22 kHz to match all our other sample rates for performance. You'll find the file D-ay-Z-ray_mix_090502.m4a in the completed project. Make sure to add it to your project if you are following along.

Add this line to BBConfiguration.h:

#define BACKGROUND_MUSIC @"D-ay-Z-ray_mix_090502"

In BBSceneController.h, we're going to add a new instance variable:

@class AVPlaybackSoundController;

@interface BBSceneController : NSObject <EWSoundCallbackDelegate> {
  ...
  AVPlaybackSoundController* backgroundMusicPlayer;
}

In BBSceneController.m, we are going to create a new instance of the backgroundMusicPlayer in the init method. We want to use our background music file and set the player to infinitely loop. To avoid overwhelming all the other audio, we will reduce the volume of the music. Once this is all set up, we tell the player to play.

- (id) init

{
   self = [super init];
   if(nil != self)
   {
      SetPreferredSampleRate(22050.0);
      [[OpenALSoundController sharedSoundController] setSoundCallbackDelegate:self];
      [self invokeLoadResources];
      [[OpenALSoundController sharedSoundController]
          setDistanceModel:AL_INVERSE_DISTANCE_CLAMPED];
      [[OpenALSoundController sharedSoundController] setDopplerFactor:1.0];
      [[OpenALSoundController sharedSoundController] setSpeedOfSound:343.3];
      backgroundMusicPlayer = [[AVPlaybackSoundController alloc]
          initWithSoundFile:BACKGROUND_MUSIC];
      backgroundMusicPlayer.numberOfLoops = −1; // loop music
      backgroundMusicPlayer.volume = 0.5;
   }
   return self;
}

Conversely, in our dealloc method, we should remember to delete the player for good measure.

[backgroundMusicPlayer stop];
[backgroundMusicPlayer release];

And, of course, we need to actually start playing the music. We could do this in init, but the startScene method seems to be more appropriate. So add the following line there:

[backgroundMusicPlayer play];

Finally, in the audio session initialization code in our OpenALSoundController class, we should switch the mode from Ambient to Solo Ambient, because we are using an .m4a file for our background music, and we would like to minimize the burden on the CPU by using the hardware decoder. If we don't do this, the music will play using the software decoder, which will work but could significantly degrade the performance of Space Rocks!, since the game already pushes the CPU pretty hard, even without audio. And it doesn't make a lot of sense to keep using Ambient mode, which would allow mixing iPod music, now that we have our own soundtrack. Alternatively, we could avoid using a restricted compression format for our audio and not worry about this. But since we have an idle hardware decoding unit and can benefit from higher compression, we will exploit these capabilities. The remaining streaming examples will also switch to Solo Ambient for these same reasons.

Find our call to InitAudioSession in OpenALSoundController.m's init method, and change the first parameter to kAudioSessionCategory_SoloAmbientSound.

InitAudioSession(kAudioSessionCategory_SoloAmbientSound, MyInterruptionCallback, self, PREFERRED_SAMPLE_OUTPUT_RATE);

Congratulations! Not only do you have background music, but you have also learned all the basic essential elements in creating a complete game solution with respect to audio. Many developers will often be satisfied here. They can play OpenAL sound effects and stream audio easily with AVFoundation.

But I encourage you to stick around for the next section on the OpenAL streaming solution. While it's more difficult, you'll continue to get features such as spatialized sound. You'll also be able to reuse more infrastructure, so things will integrate more smoothly, instead of having two disparate audio systems that don't really talk to each other. For example, we currently do nothing with AVAudioPlayer's audioPlayerDidFinishPlaying:successfully: method because we would have to think about how it would integrate with the rest of the system. We already defined all those behaviors for our OpenAL system, so there is less ambiguity on how we should handle it with OpenAL streaming.

Following the pattern we used for OpenALSoundController in Chapter 10, let's examine the changes we need to make for OpenALStreamingController.m's initOpenAL method.

- (void) initOpenAL
{
   openALDevice = alcOpenDevice(NULL);
   if(openALDevice != NULL)
   {

// Use the Apple extension to set the mixer rate
      alcMacOSXMixerOutputRate(44100.0);

      // Create a new OpenAL context
      // The new context will render to the OpenAL device just created
      openALContext = alcCreateContext(openALDevice, 0);
      if(openALContext != NULL)
      {
         // Make the new context the current OpenAL context
         alcMakeContextCurrent(openALContext);
      }
      else
      {
          NSLog(@"Error, could not create audio context.");
          return;
      }
   }
   else
   {
       NSLog(@"Error, could not get audio device.");
       return;
   }

   alGenSources(1, &streamingSource);
   alGenBuffers(MAX_OPENAL_QUEUE_BUFFERS, availableALBufferArray);
   availableALBufferArrayCurrentIndex = 0;

   // File is from Internet Archive Open Source Audio, US Army Band, public domain
   // http://www.archive.org/details/TheBattleHymnOfTheRepublic_993
   NSURL* file_url = [NSURL fileURLWithPath:[[NSBundle mainBundle]
        pathForResource:@"battle_hymn_of_the_republic" ofType:@"mp3"]];
   if(file_url)
   {
      streamingAudioRef = MyGetExtAudioFileRef((CFURLRef)file_url,
        &streamingAudioDescription);
   }
   else
   {
       NSLog(@"Could not find file!
");
       streamingAudioRef = NULL;
   }

   intermediateDataBuffer = malloc(INTERMEDIATE_BUFFER_SIZE);
}

The only new/customized code is at the bottom of the method after the OpenAL context is initialized.

First, we create an OpenAL source, which we will use to play our music using alGenSources. Next, we create five OpenAL buffers using the array version of the function. The number of buffers was picked somewhat arbitrarily. We'll talk about picking the number a little later, but for now, just know that our program will have a maximum of five buffers queued at any one time.

Then we open the music file we are going to play. I am reusing the same file from the AVPlayback example in Chapter 9.

The first semi-new thing is that we call MyGetExtAudioFileRef. Recall that in our side quest in Chapter 10 when we loaded files into OpenAL using Core Audio, we deliberately broke up the function into three pieces, but we used only the third piece, MyGetOpenALAudioDataAll, which is built using the other two pieces. MyGetOpenALAudioDataAll loads the entire file into a buffer. But since we don't want to load the entire file for this streaming example, we need to use the two other pieces. The first piece, MyGetExtAudioFileRef, will just open the file using Core Audio (Extended File Services) and return a file handle (ExtAudioFileRef). We save this file reference in an instance variable, because we need to read the data from it later as we stream. We also have streamingAudioDescription as an instance variable, because we need to keep the audio description metadata around for the other function. (We'll get to the second piece a little later.)

Next, we allocate memory for a PCM buffer. This is memory we will have Extended File Services decode file data into so we can use it. I have hard-coded the buffer to be 32KB, which means this demo is going to stream audio data in 32,768-byte chunks at a time.

That concludes our initialization. Let's now go to the heart of the program, which is in the animationCallback: method in OpenALStreamingController. This is where all the important stuff happens.

We need to do two things with OpenAL streaming: queue buffers to play and unqueue buffers that have finished playing (processed buffers). We will start with unqueuing (and reclaiming) the processed buffers. I like to do this first, because I can then turn around and immediately use that buffer again for the next queue.

First, OpenAL will let us query how many buffers have been processed using alGetSourcei with AL_BUFFERS_PROCESSED. In OpenALStreamingController.m's animationCallback: method, we need to have the following:

ALint buffers_processed = 0;
alGetSourcei(streamingSource, AL_BUFFERS_PROCESSED, &buffers_processed);

Next, we can write a simple while loop to unqueue and reclaim each processed buffer, one at a time.^[31] To actually unqueue a buffer, OpenAL provides the function alSourceUnqueueBuffers. You provide it the source you want to unqueue from, the number of buffers you want to unqueue, and a pointer to where the unqueued buffer IDs will be returned.

while(buffers_processed > 0)
{
   ALuint unqueued_buffer;
   alSourceUnqueueBuffers(streamingSource, 1, &unqueued_buffer);

availableALBufferArrayCurrentIndex--;
   availableALBufferArray[availableALBufferArrayCurrentIndex] = unqueued_buffer;

   buffers_processed--;
}

We keep an array of available buffers so we can use them in the next step. We use the array like a stack, hence the incrementing and decrementing of the availableALBufferArrayCurrentIndex. Don't get too hung up on this part. Just consider it an opaque data structure. Because we're doing a minimalist example, I wanted to avoid using Cocoa data structures or introducing my own. The next example will not be minimalist.

Now we are ready to queue a buffer. When we queue a buffer, we need to load a chunk of the data from the file, get the data into OpenAL, and queue it to the OpenAL source. We are going to queue only one buffer in a single function pass. The idea is that this function will be called repeatedly (30 times a second or so). The assumption is the function call rate will be faster than OpenAL can use up the buffer, so there will always be multiple buffers queued in a given moment, up to some maximum number of queue buffers that we define. For this example, MAX_OPENAL_QUEUE_BUFFERS is hard-coded as follows:

#define MAX_OPENAL_QUEUE_BUFFERS 5

We continue adding code to animationCallback: in OpenALStreamingController.m. The first thing we do is check to make sure we have available buffers to queue. If not, then we don't need to do anything.

if(availableALBufferArrayCurrentIndex < MAX_OPENAL_QUEUE_BUFFERS)
{

Once we establish we need to queue a new buffer, we do it.

ALuint current_buffer = availableALBufferArray[availableALBufferArrayCurrentIndex];

  ALsizei buffer_size;
  ALenum data_format;
  ALsizei sample_rate;

  MyGetDataFromExtAudioRef(streamingAudioRef, &streamingAudioDescription,
     INTERMEDIATE_BUFFER_SIZE, &intermediateDataBuffer, &buffer_size,
    &data_format, &sample_rate);
  if(0 == buffer_size) // will loop music on EOF (which is 0 bytes)
  {
     MyRewindExtAudioData(streamingAudioRef);
     MyGetDataFromExtAudioRef(streamingAudioRef, &streamingAudioDescription,
       INTERMEDIATE_BUFFER_SIZE, &intermediateDataBuffer, &buffer_size,
       &data_format, &sample_rate);
  }
  alBufferData(current_buffer, data_format, intermediateDataBuffer, buffer_size,
    sample_rate);
  alSourceQueueBuffers(streamingSource, 1, &current_buffer);

availableALBufferArrayCurrentIndex++;

We use our MyGetDataFromExtAudioRef function (which is our second piece of the Core Audio file loader from Chapter 10) to fetch a chunk of data from our file. We provide the file handle and the audio description, which you can consider as opaque data types for Core Audio. We provide the buffer size and buffer to which we want the data copied. The function will return by reference the amount of data we actually get back, the OpenAL data format, and the sample rate. We can then feed these three items directly into alBufferData. We use alBufferData for simplicity in this example. For performance, we should consider using alBufferDataStatic, since this function is going to be called a lot, and we are going to be streaming in the middle of game play where performance is more critical. We will change this in the next example.

Finally, we call OpenAL's function alSourceQueueBuffers to queue the buffer. We specify the source, the number of buffers, and an array containing the buffer IDs.

There is a corner case handled in the preceding code. If we get 0 bytes back from MyGetDataFromExtAudioRef, it means we hit the end of the file (EOF). For this example, we want to loop the music, so we call our custom helper function MyRewindExtAudioData, which rewinds the file pointer to the beginning. We then grab the data again. If you were thinking you could use OpenAL's built-in loop functionality, this won't work. OpenAL doesn't have access to the full data anymore, since we broke everything into small pieces and unqueued the old data (the beginning of the file). We must implement looping ourselves.

We've now queued the buffer, but there is one more step we may need to take. It could happen that we were not quick enough in queuing more data, and OpenAL ran out of data to play. If OpenAL runs out of data to play, it must stop playing. This makes sense, because OpenAL can't know whether we were too slow at queuing more data or we actually finished playing (and don't plan to loop). I call the case where we were too slow a buffer underrun.

To remedy the potential buffer underrun case, our task is to find out if the OpenAL source is still playing. If it is not playing, we need to determine if it is because of a buffer underrun or because we don't want to play (e.g., finished playing, wanted to pause, etc.).

We will use alGetSourcei to find the source state to figure out if we are not playing. Then we can use alGetSourcei to get the number of buffers queued to help determine if we are in a buffer underrun situation. (We will have one queued buffer now since we just added one in the last step.) Then we make sure the user didn't pause the player, which would be a legitimate reason for not playing even though we have buffers queued. If we determine we should, in fact, be playing, we simply call alSourcePlay.

ALenum current_playing_state;
alGetSourcei(streamingSource, AL_SOURCE_STATE, &current_playing_state);
// Handle buffer underrun case
if(AL_PLAYING != current_playing_state)
{

ALint buffers_queued = 0;

   alGetSourcei(streamingSource, AL_BUFFERS_QUEUED, &buffers_queued);

   if(buffers_queued > 0 && NO == self.isStreamingPaused)
   {
      // Need to restart play
      alSourcePlay(streamingSource);
   }
}

We use alSourcePause for the first time in this example. It probably behaves exactly as you think it does, so there is not much to say about it.

You may be wondering why we removed the fast-forward and rewind buttons from the AVPlayback example. The reason is that seeking is more difficult with OpenAL streaming than it is with AVAudioPlayer.

OpenAL does have support for seeking. There are three different attributes you can use with alSource* and alGetSource*: AL_SEC_OFFSET, AL_SAMPLE_OFFSET, and AL_BYTE_OFFSET. These deal with the positions in seconds, samples, or bytes. With fully loaded buffers, seeking is pretty easy. But with streaming, you will need to do some additional bookkeeping.

The problem is that with streaming, OpenAL can seek only within the range of what is currently queued. If you seek to something that is outside that range, it won't work. In addition, you need to make sure your file handle is kept in sync with your seek request; otherwise, you will be streaming new data in at the wrong position. Thus, you really need to seek at the file level instead of the OpenAL level.

Generally, emptying the current queue and rebuffering it at the new position is the easiest solution. (If you don't empty the queue, you may have a lag between when the users requested a seek and when they actually hear it.) But, of course, as you've just emptied your queue and need to start over, you may have introduced some new latency issues. For this kind of application, probably no one will care though. For simplicity, this was omitted from the example. For the curious, notice that our MyRewindExtAudioData function wraps ExtAudioFileSeek. This is what you'll probably want to use to seek using Extended File Services.

Recall that in this example, we fill only one buffer per update loop pass. This means the game loop must be run through at least five times before we can fill our queue to our designated maximum of MAX_OPENAL_QUEUE_BUFFERS. This brings up the question, "What is the best strategy for filling the queue?"

There are two extremes to consider. The first extreme is what we did in the example. We fill a single buffer and move on. The advantages of this strategy are that it is easy to implement and it has a cheap startup cost. It allows us to spread the buffer fill-up cost over a longer period of time. The disadvantage is that when we first start up, we don't have a safety net. We are particularly vulnerable to an untimely system usage spike that prevents us from being able to queue the next buffer in time.

The other extreme is that we fill up our queue to our designated maximum when we start playing. So, in this example, rather than filling just one buffer, we would fill up five buffers. While this gives us extra protection from starvation, we have a trade-off of a longer startup cost. This may manifest itself as a performance hiccup in the game if the startup cost is too great. For example, the player may fire a weapon that we tied to streaming. If we spend too much time up front to load more buffers, the user may perceive a delay between the time he touched the fire button and when the weapon actually seemed to fire in the game.

You will need to experiment to find the best pattern for you and your game. You might consider hybrids of the two extremes, where on startup, you queue two or three buffers instead of the maximum. You might also consider varying the buffer sizes and the number of buffers you queue based on metrics such as how empty the queue is or how taxed the CPU is. And, of course, multithreading/concurrency can be used, too.

Finally, you may be wondering why we picked five buffers and made our buffers 32KB. This is somewhat of a guessing game. You generally need to find values that give good performance for your case. You are trying to balance different things.

The buffer size will affect how long it takes to load new data from the file. Larger buffers take more time. If you take too much time, you will starve the queue, because you didn't get data into the queue fast enough. In my personal experience, I have found 1KB to 64KB to be the range for buffer sizes. More typically, I tend to deal with 8KB to 32KB. Apple seems to like 16KB to 64KB. Some of Apple's examples include a CalculateBytesForTime function, which will dynamically decide the size based on some criteria. (It's worth a look.)

I also like powers of two because of a hard-to-reproduce bug I had way back with the Loki OpenAL implementation. When not using power-of-two buffer sizes, I had certain playback problems. That implementation is dead now, but the habit stuck with me. Some digital processing libraries, like for a Fast Fourier Transform (FFT), often prefer arrays in power-of-two sizes for performance reasons. So it doesn't hurt.^[32]

As for the number of buffers, you basically want enough to prevent buffer underruns. The more buffers you have queued, the less likely it will be that you deplete the queue before you can add more. However, if you have too many buffers, you are wasting memory. The point of streaming is to save memory. And, obviously, there is a relationship with the buffer size. Larger buffer sizes will last longer, so you need fewer buffers.

Also, you might think about the startup number of buffers. In our example, we queue only one buffer to start with. One consequence of that is we must make our buffer larger to avoid early starvation. If we had queued more buffers, the buffer size could be smaller.

Let's start with the new class. We will name it EWStreamBufferData and create .h and .m files for it. The purpose of this class is analogous to the EWSoundBufferData class we made earlier, except it will be for streamed data.

@interface EWStreamBufferData : NSObject
{
   ALenum openalFormat;
   ALsizei dataSize;
   ALsizei sampleRate;

In Chapter 10, we implemented a central/shared database of audio files with soundFileDictionary and EWSoundBufferData. We will not be repeating that pattern with EWStreamBufferData. This is because it makes no sense to share streamed data. With fully loaded sounds, we can share the explosion sound between the UFO and the spaceship. But with streamed data, we can't do this because we have only a small chunk of the data in memory at any given time. So, in the case of the explosion sound, the spaceship might be starting to explode while the UFO is ending its explosion. Each needs a different buffer segment in the underlying sound file. For streamed data, if both the UFO and spaceship use the same file, they still need to have separate instances of EWStreamBufferData. Because of the differences between the two classes, I have opted to not make EWStreamBufferData a subclass of EWSoundBufferData. You could do this if you want, but you would need to modify the existing code to then be aware of the differences.

This class will contain the multiple OpenAL buffers needed for buffer queuing. In the previous example, we used five buffers. This time, we will use 32. (I'll explain the increase in buffers later.) We will use the alBufferDataStatic extension for improved performance, so we also need 32 buffers for raw PCM data.

#define EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS 32
   ALuint openalDataBufferArray[EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS];
   void* pcmDataBufferArray[EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS];

We will also need to keep track of which buffers are queued and which are available. In the previous example, I apologized for being too minimalist for using a single C array. This time, we will use two NSMutableArrays for more clarity.

NSMutableArray* availableDataBuffersQueue;
  NSMutableArray* queuedDataBuffersQueue;

This class will also contain the ExtAudioFileRef (file handle) to the audio file (and AudioStreamBasicDescription) so we can load data into the buffers from the file as we need it.

ExtAudioFileRef streamingAudioRef;
  AudioStreamBasicDescription streamingAudioDescription;

We also have a few properties. We have a streamingPaused property in case we need to pause the audio, which will allow us to disambiguate from a buffer underrun, as in the previous example. (We will not actually pause the audio in our game.) We add audioLooping so it can be an option instead of hard-coded. And we add an atEOF property so we can record if we hit the end of file.

BOOL audioLooping;
   BOOL streamingPaused;
   BOOL atEOF;
}
@property(nonatomic, assign, getter=isAudioLooping) BOOL audioLooping;
@property(nonatomic, assign, getter=isStreamingPaused) BOOL streamingPaused;
@property(nonatomic, assign, getter=isAtEOF) BOOL atEOF;

We will also add three methods:

+ (EWStreamBufferData*) streamBufferDataFromFileBaseName:(NSString*)sound_file_basename;
- (EWStreamBufferData*) initFromFileBaseName:(NSString*)sound_file_basename;
- (BOOL) updateQueue:(ALuint)streaming_source;
@end

The initFromFileBaseName: instance method is our designated initializer, which sets the file to be used for streaming. For convenience, we have a class method named streamBufferDataFromFileBaseName:, which ultimately does the same thing as initFromFileBaseName:, except that it returns an autoreleased object (as you would expect, following standard Cocoa naming conventions).

You might have noticed that this design contrasts slightly with the method soundBufferDataFromFileBaseName:, which we placed in OpenALSoundController instead of EWSoundBufferData. The main reason is that for EWSoundBufferData objects, we had a central database to allow resource sharing. Since OpenALSoundController is a singleton, it was convenient to put the central database there. EWStreamBufferData differs in that we won't have a central database. Since this class is relatively small, I thought we might take the opportunity to relieve the burden on OpenALSoundController. Ultimately, the two classes are going to work together, so it doesn't matter too much. But for symmetry, we will add a convenience method to OpenALSoundController named streamBufferDataFromFileBaseName:, which just calls the one in this class.

The updateQueue: method is where we are going to do most of the unqueuing and queuing work. We could do this in the OpenALSoundController, but as you saw, the code is a bit lengthy. So again, I thought I we might take the opportunity to relieve the burden on OpenALSoundController.

Finally, because we are using alBufferDataStatic, we want an easy way to keep our PCM buffer associated with our OpenAL buffer, particularly with our NSMutableArray queues. We introduce a helper class that just encapsulates both buffers so we know they belong together.

@interface EWStreamBufferDataContainer : NSObject
{
   ALuint openalDataBuffer;
   void* pcmDataBuffer;
}
@property(nonatomic, assign) ALuint openalDataBuffer;
@property(nonatomic, assign) void* pcmDataBuffer;
@end

In our initialization code for EWStreamBufferData, we need to allocate a bunch of memory: 32 OpenAL buffers (via alGenBuffers), 32 PCM buffers (via malloc), and 2 NSMutableArrays, which are going to mirror the buffer queue state by recording which buffers are queued and which are available. Let's first look at the code related to initialization:

@implementation EWStreamBufferData
@synthesize audioLooping;
@synthesize streamingPaused;
@synthesize atEOF;

- (void) createOpenALBuffers
{

   for(NSUInteger i=0; i<EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS; i++)
   {
       pcmDataBufferArray[i] = malloc(EW_STREAM_BUFFER_DATA_INTERMEDIATE_BUFFER_SIZE);
   }

   alGenBuffers(EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS, openalDataBufferArray);

   availableDataBuffersQueue = [[NSMutableArray alloc]
         initWithCapacity:EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS];

queuedDataBuffersQueue = [[NSMutableArray alloc]
      initWithCapacity:EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS];

   for(NSUInteger i=0; i<EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS; i++)
   {
       EWStreamBufferDataContainer* stream_buffer_data_container =
         [[EWStreamBufferDataContainer alloc] init];
       stream_buffer_data_container.openalDataBuffer = openalDataBufferArray[i];
       stream_buffer_data_container.pcmDataBuffer = pcmDataBufferArray[i];
       [availableDataBuffersQueue addObject:stream_buffer_data_container];
       [stream_buffer_data_container release];
    }
}

- (id) init
{
   self = [super init];
   if(nil != self)
    {
      [self createOpenALBuffers];
    }
    return self;
}

- (EWStreamBufferData*) initFromFileBaseName:(NSString*)sound_file_basename
{
   self = [super init];
   if(nil != self)
   {
      [self createOpenALBuffers];

      NSURL* file_url = nil;

// Create a temporary array containing all the file extensions we want to handle.
// Note: This list is not exhaustive of all the types Core Audio can handle.
NSArray* file_extension_array = [[NSArray alloc]
  initWithObjects:@"caf", @"wav", @"aac", @"mp3", @"aiff", @"mp4", @"m4a", nil];
for(NSString* file_extension in file_extension_array)
{
    // We need to first check to make sure the file exists;
    // otherwise NSURL's initFileWithPath:ofType will crash if the file doesn't exist
    NSString* full_file_name = [NSString stringWithFormat:@"%@/%@.%@",
      [[NSBundle mainBundle] resourcePath], sound_file_basename, file_extension];
    if(YES == [[NSFileManager defaultManager] fileExistsAtPath:full_file_name])
    {
       file_url = [[NSURL alloc] initFileURLWithPath:[[NSBundle mainBundle]
          pathForResource:sound_file_basename ofType:file_extension]];
       break;
    }
 }
 [file_extension_array release];

 if(nil == file_url)
 {
    NSLog(@"Failed to locate audio file with basename: %@", sound_file_basename);
    [self release];
    return nil;

}

 streamingAudioRef = MyGetExtAudioFileRef((CFURLRef)file_url,
    &streamingAudioDescription);
 [file_url release];
 if(NULL == streamingAudioRef)
 {
    NSLog(@"Failed to load audio data from file: %@", sound_file_basename);
    [self release];
    return nil;
 }
}
        return self;
}


+ (EWStreamBufferData*) streamBufferDataFromFileBaseName:(NSString*)sound_file_basename
{
        return [[[EWStreamBufferData alloc] initFromFileBaseName:sound_file_basename] autorelease];
}

Let's examine the initFromFileBaseName: method. The first part of the method is just a copy-and-paste of our file-finding code that tries to guess file extensions. The following is the only important line in this function:

streamingAudioRef = MyGetExtAudioFileRef((CFURLRef)file_url, &streamingAudioDescription);

This opens our file and returns the file handle and AudioStreamBasicDescription.

The streamBufferDataFromFileBaseName: convenience class method just invokes the initFromFileBaseName: instance method.

Finally, in the createOpenALBuffers method, the final for loop fills our availableDataBuffersQueue with our EWStramBufferDataContainer wrapper object. That way, it is easy to get at both the PCM buffer and OpenAL buffer from the array object.

You might be wondering why we have two arrays when the previous example had only one. With two arrays, we can easily know if a buffer is queued or available. We are doing a little extra work here by mirroring (or shadowing) the OpenAL state, but it's not much more work.

For correctness, we also should write our dealloc code:

- (void) dealloc
{
   [self destroyOpenALBuffers];

   if(streamingAudioRef)
   {
      ExtAudioFileDispose(streamingAudioRef);
   }

   [super dealloc];
}

- (void) destroyOpenALBuffers
{
   [availableDataBuffersQueue release];
   [queuedDataBuffersQueue release];

   alDeleteBuffers(EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS,
       openalDataBufferArray);

   for(NSUInteger i=0; i<EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS; i++)
   {
       free(pcmDataBufferArray[i]);
       pcmDataBufferArray[i] = NULL;
   }
}

There shouldn't be any surprises here, except that this class also keeps a file handle, so we need to remember to close the file handle if it is open.

The real meat of this class is the updateQueue: method. As I said, we are going to put all the stuff in the updateQueue update loop from the BasicOpenALStreaming example. This loop should look very familiar to you, as it is the same algorithm. For brevity, I won't reproduce the entire body of the code here, but it's included with the completed project example. However, we will zoom in on subsections of the method next.

As in the BasicOpenALStreaming example, let's start with unqueuing the processed buffers in the updateQueue method.

ALint buffers_processed = 0;
alGetSourcei(streaming_source, AL_BUFFERS_PROCESSED, &buffers_processed);
while(buffers_processed > 0)
{
   ALuint unqueued_buffer;
   alSourceUnqueueBuffers(streaming_source, 1, &unqueued_buffer);

   [availableDataBuffersQueue insertObject:[queuedDataBuffersQueue lastObject]
       atIndex:0];
   [queuedDataBuffersQueue removeLastObject];

   buffers_processed--;
}

Astute observers might notice that we do nothing with the unqueued_buffer we retrieve from OpenAL. This is because we are mirroring (shadowing) the OpenAL buffer queue with our two arrays, so we already know which buffer was unqueued.^[33]

Continuing in updateQueue, let's queue a new buffer if we haven't gone over the number of queued buffers specified by our self-imposed maximum, which is 32 buffers. Since we have a data structure that holds all our available buffers, we can just check if the data structure has any buffers.

if([availableDataBuffersQueue count] > 0 && NO == self.isAtEOF)
{
    // Have more buffers to queue
    EWStreamBufferDataContainer* current_stream_buffer_data_container =
      [availableDataBuffersQueue lastObject];
    ALuint current_buffer = current_stream_buffer_data_container.openalDataBuffer;
    void* current_pcm_buffer = current_stream_buffer_data_container.pcmDataBuffer;

    ALenum al_format;
    ALsizei buffer_size;
    ALsizei sample_rate;

    MyGetDataFromExtAudioRef(streamingAudioRef, &streamingAudioDescription,
      EW_STREAM_BUFFER_DATA_INTERMEDIATE_BUFFER_SIZE,
      &current_pcm_buffer, &buffer_size, &al_format, &sample_rate);
    if(0 == buffer_size) // will loop music on EOF (which is 0 bytes)
    {
       if(YES == self.isAudioLooping)
       {
          MyRewindExtAudioData(streamingAudioRef);
          MyGetDataFromExtAudioRef(streamingAudioRef, &streamingAudioDescription,
            EW_STREAM_BUFFER_DATA_INTERMEDIATE_BUFFER_SIZE,
            &current_pcm_buffer, &buffer_size, &al_format, &sample_rate);
       }
       else
       {
          self.atEOF = YES;
       }
   }

   if(buffer_size > 0)
   {
      alBufferDataStatic(current_buffer, al_format, current_pcm_buffer, buffer_size,
         sample_rate);
      alSourceQueueBuffers(streaming_source, 1, &current_buffer);

      [queuedDataBuffersQueue insertObject:current_stream_buffer_data_container
         atIndex:0];
         [availableDataBuffersQueue removeLastObject];

There are just a few minor new things here. First, audio looping is now an option, so we have additional checks for atEOF. If we are at an EOF, we don't want to do anything. (We will have additional logic at the end of this function to determine what to do next if we do encounter EOF.) If we encounter EOF while fetching data, we record that in our instance variable, but only if we are not supposed to loop audio. If we are supposed to loop, we pretend we never hit EOF and just rewind the file.

The other minor thing is we now use alBufferDataStatic.

Still in updateQueue, let's handle the buffer underrun case.

ALenum current_playing_state;
    alGetSourcei(streaming_source, AL_SOURCE_STATE, &current_playing_state);
    // Handle buffer underrun case
    if(AL_PLAYING != current_playing_state)
    {
       ALint buffers_queued = 0;
       alGetSourcei(streaming_source, AL_BUFFERS_QUEUED, &buffers_queued);
       if(buffers_queued > 0 && NO == self.isStreamingPaused)
       {
          // Need to restart play
          alSourcePlay(streaming_source);
       }
     }
   }
}

This code is essentially identical to the previous example.

The part to handle EOF and finished playing is new but simple. To end the updateQueue method, for convenience, this function will return YES if we discover that the source has finished playing its sound. (The OpenALSoundController will use this information later.) This requires two conditions:

Once we detect those two conditions, we just record the state so we can return it. We also add a repeat check for processed buffers, since typically, once it is detected that the sound is finished playing, this method will no longer be invoked for that buffer instance. This is a paranoid check. In theory, there is a potential race condition between OpenAL processing buffers and us trying to remove them. It is possible that once we pass the processed buffers check at the top of this method, OpenAL ends up processing another buffer while we are in the middle of the function. We would like to leave everything in a pristine state when it is determined that that we are finished playing, so we run the processed buffers check one last time and remove the buffers as necessary.

if(YES == self.isAtEOF)
  {
     ALenum current_playing_state;

     alGetSourcei(streaming_source, AL_SOURCE_STATE, &current_playing_state);
     if(AL_STOPPED == current_playing_state)
     {
        finished_playing = YES;

        alGetSourcei(streaming_source, AL_BUFFERS_PROCESSED, &buffers_processed);

while(buffers_processed > 0)
        {
           ALuint unqueued_buffer;
           alSourceUnqueueBuffers(streaming_source, 1, &unqueued_buffer);
           [availableDataBuffersQueue insertObject:[queuedDataBuffersQueue lastObject]
             atIndex:0];
           [queuedDataBuffersQueue removeLastObject];

           buffers_processed--;
         }
     }
  }

  return finished_playing;
}

Whew! You made it through this section. I hope that wasn't too bad, as you've already seen most of it before. Now we need to make some changes to the OpenALSoundController.

Since the OpenALSoundController is our centerpiece for all our OpenAL code, we need to tie in the stuff we just wrote with this class. There are only a few new methods we are going to introduce in OpenALSoundController:

- (EWStreamBufferData*) streamBufferDataFromFileBaseName:(NSString*)sound_file_basename;
- (void) playStream:(ALuint)source_id streamBufferData:(EWStreamBufferData*)stream_buffer_data;
- (void) setSourceGain:(ALfloat)gain_level sourceID:(ALuint)source_id;

The method streamBufferDataFromFileBaseName: is just a pass-through to EWStreamBufferData's streamBufferDataFromFileBaseName:, which we add mostly for aesthetic reasons. This gives us symmetry with OpenALSoundController's soundBufferDataFromFileBaseName: method.

- (EWStreamBufferData*) streamBufferDataFromFileBaseName:(NSString*)sound_file_basename
{
   return [EWStreamBufferData streamBufferDataFromFileBaseName:sound_file_basename];
}

The setSourceGain: method is kind of a concession/hack. Prior to this, only our BBSceneObjects made sound. They all access their OpenAL source properties (such as gain) through EWSoundSourceObjects. But because we are focusing on background music, it doesn't make a lot of sense to have a BBSceneObject to play background music. So, this is a concession to let us set the volume level on our background music without needing the entire SceneObject infrastructure.

- (void) setSourceGain:(ALfloat)gain_level sourceID:(ALuint)source_id
{
   alSourcef(source_id, AL_GAIN, gain_level);
}

The playStream: method is the most important new method we add. This method will allow us to designate we are playing a stream instead of a preloaded sound. We could get really clever and try to unify everything into one play method, but this will keep things more straightforward for educational purposes.

We also need to go back and modify some existing methods to be aware of streams. Additionally, we will introduce some additional bookkeeping to manage our streams. Most important, we will introduce a new instance variable:

NSMutableDictionary* streamingSourcesDictionary;

In the initOpenAL method, allocate a new dictionary for this instance variable with the other collections. (And don't forget to release it in the tearDownOpenAL method.)

streamingSourcesDictionary = [[NSMutableDictionary alloc] initWithCapacity:MAX_NUMBER_OF_ALSOURCES];

We will use this to track which sources are currently playing a stream, similar to how we tracked currently playing (preloaded) sounds. You may notice this is a dictionary instead of a set, which is different from what we did with preloaded sources. In this case, we also want access to the buffer that the source is currently playing. So when we go through our OpenALSoundController's update loop, we can call the updateQueue: method we wrote in the previous section. In the preloaded case, we didn't need to do anything with the buffer, so we didn't need access to the data. Thus, we could use an NSSet containing only sources. But our streamingSourcesDictionary will be keyed by source IDs, and the values will be EWStreamBufferData objects.

Now that you know where we are going with streamingSourcesDictionary, let's look at our new methods. The only one with new material here worth talking about is playStream:.

// Must call reserveSource to get the source_id
- (void) playStream:(ALuint)source_id
    streamBufferData:(EWStreamBufferData*)stream_buffer_data
{
   // Trusting the source_id passed in is valid
   [streamingSourcesDictionary setObject:stream_buffer_data
          forKey:[NSNumber numberWithUnsignedInt:source_id]];
   // updateQueue will automatically start playing
   [stream_buffer_data updateQueue:source_id];
}

This is a simple method because all the logic is elsewhere. It is just bookkeeping. First, we add our source and EWStreamBufferData to our streamingSourcesDictionary. Then we just call the updateQueue: method we implemented in the previous section to start playing. Recall that one pass of updateQueue: will queue a new buffer and then do buffer underrun detection. Since there will be a new buffer in the queue but the source is stopped (because we never started playing it), the code will consider this to be a buffer underrun situation and automatically start playing for us. How nice and elegant, right?

Now it's important that we enhance our update method to handle streams. It needs to do all the same stuff that the loop does for preloaded sources—detect finished playback, invoke callbacks, and recycle the sources. But it also needs to update the buffer queues for each playing stream. To update the buffer queues, we need to invoke updateQueue: on each EWStreamBufferData object. This will be very easy to implement. In fact, it is almost a copy-and-paste of the existing code. At the bottom of the update method, add the following code:

NSMutableDictionary* streaming_items_to_be_purged_dictionary =
    [[NSMutableDictionary alloc] initWithCapacity:[streamingSourcesDictionary count]];
  for(NSNumber* current_number in streamingSourcesDictionary)
  {
      ALuint source_id = [current_number unsignedIntValue];
      EWStreamBufferData* stream_buffer_data =
        [streamingSourcesDictionary objectForKey:current_number];
      BOOL finished_playing = [stream_buffer_data updateQueue:source_id];
      if(YES == finished_playing)
      {
         [streaming_items_to_be_purged_dictionary setObject:stream_buffer_data
         forKey:current_number];
      }
  }
  for(NSNumber* current_number in streaming_items_to_be_purged_dictionary)
  {
      [streamingSourcesDictionary removeObjectForKey:current_number];
      [self recycleSource:[current_number unsignedIntValue]];
      if([self.soundCallbackDelegate
         respondsToSelector:@selector(soundDidFinishPlaying:)])
      {
          [self.soundCallbackDelegate soundDidFinishPlaying:current_number];
      }
  }
  [streaming_items_to_be_purged_dictionary release];

As before with preloaded sources, we iterate through all the items in the collection and look for sources that have stopped playing. Notice that we use the return value of updateQueue: to determine if we stopped playing. We use this instead of querying the source directly because we don't want to accidentally interpret a buffer underrun condition as a finished playing situation. We already wrote all that logic in updateQueue:, and now we get to benefit from it.

Also notice that we kill two birds with one stone. In calling updateQueue: to find out if a source has finished playing, we are also updating the OpenAL buffer queue for the stream if it is not finished playing. (Don't you love how this is all coming together?)

And finally, when we do encounter a sound finished situation, we do what we did before: recycle the source and invoke our soundDidFinishedPlaying: delegate callback. Again, this all wonderfully coming together in an elegant manner. Even though we are talking about buffer streaming, it doesn't need to affect the behavior of the OpenAL sources. So when a source finishes playing, regardless of whether it is preloaded or loaded, you get (the same) callback notification. This means all our existing BBSceneObject code that listens for these callbacks doesn't need to change at all.

Now you can see why trying to deal with callbacks in our AVFoundation background music example would require some serious design decisions and possible significant code changes: AVFoundation callbacks don't match up with how our BBSceneObjects are using them.

Now that we're on a roll with respect to integration, there is one more easy thing we can try. Even though we have separate playSound: and playStream: methods, we can reuse the stopSound: method for both cases.

- (void) stopSound:(ALuint)source_id
{
   // Trusting the source_id passed in is valid
   alSourceStop(source_id);
   alSourcei(source_id, AL_BUFFER, AL_NONE); // detach the buffer from the source
   // Remove from the playingSourcesCollection or streamingSourcesDictionary,
   //no callback will be fired. Just try removing from both collections.
   //As far as I know, there is no problem trying to remove if it doesn't exist.
   [playingSourcesCollection removeObject:[NSNumber numberWithUnsignedInt:source_id]];
   [streamingSourcesDictionary removeObjectForKey:[NSNumber
      numberWithUnsignedInt:source_id]];
   [self recycleSource:source_id];
}

We are employing a trick here. Instead of first checking to see which collection the source is in, we just try to remove it from both. According to Apple's documentation on NSMutableDictionary, removing a key that does not exist does nothing. So it sounds like a safe operation. The documentation doesn't say anything about NSMutableSet, but I am going to assume they are consistent.

Also, notice this line:

alSourcei(source_id, AL_BUFFER, AL_NONE);

For preloaded sources, I said it was optional to detach the buffer, since we were just attaching new buffers when we needed the source again. But with buffer queuing, if we stop prematurely, we may still have queued buffers, and there is no API to remove those. Officially, from the spec, this line of code is how you clear all the buffers from a source.

Finally, as we leave OpenALSoundController, remember to change the audio session mode to kAudioSessionCategory_SoloAmbientSound in the init method in our call to InitAudioSession().

We're finally here. Now we get to try playing our music. Again, we will be using the music from FatLab Music, just as in the AVFoundation/Space Rocks! example.

Remember to add the #define to BBConfiguration.h:

// Music by Michael Shaieb
// © Copyright 2009 FatLab Music
// From "Snowferno" for iPhone/iPod Touch
#define BACKGROUND_MUSIC @"D-ay-Z-ray_mix_090502"

In BBSceneController, we are going to add two new instance variables:

EWStreamBufferData* backgroundMusicStreamBufferData;
ALuint backgroundMusicSourceID;

In the init method, let's load the sound file and tell it to loop:

backgroundMusicStreamBufferData = [[EWStreamBufferData alloc] initFromFileBaseName:BACKGROUND_MUSIC];
backgroundMusicStreamBufferData.audioLooping = YES;

In the startScene method, let's reserve a source and start playing. (We also set the music's gain level here.)

BOOL source_available = [[OpenALSoundController sharedSoundController] reserveSource:&backgroundMusicSourceID];
if(YES == source_available)
{
   [[OpenALSoundController sharedSoundController] setSourceGain:0.50
       sourceID:backgroundMusicSourceID];
   [[OpenALSoundController sharedSoundController] playStream:backgroundMusicSourceID
       streamBufferData:backgroundMusicStreamBufferData];
}
else
{
   NSLog(@"Unexpected Error: No AL source available for background music");
}

To clean up, in the dealloc method, we will stop playing and release the buffer.

[[OpenALSoundController sharedSoundController] stopSound:backgroundMusicSourceID];
[backgroundMusicStreamBufferData release];

That's it. You're ready to build and play. You can now pat yourself on the back. You have a fully capable OpenAL engine that can play sound effects and streams.

I promised to explain why we used 32 buffers and 16KB buffer sizes. This was a number I experimentally found to work well enough with Space Rocks! I don't claim this is optimal, but it is sufficient to get over one of our dangerous bottleneck points.

In our game engine design, we use NSTimer to trigger our game update loop, which the audio engine is tied to. The problem is that NSTimer operates on the main thread, so it can be blocked if we are busy doing other things. When you die and touch the game restart button, the engine reloads a whole bunch of things. This creates a bottleneck point.

On a first-generation iPhone, the reload time was sufficiently long that the music suffered a buffer underrun condition. While the game is loading, we are not able to queue more buffers, and with too few buffers, the audio system hit an underrun. This may be exacerbated by the fact that we don't guarantee the queue is always filled with the maximum number of buffers and it is a best-effort approach. On a first-generation iPhone, all the visual particle effects can tax the device pretty hard. Usually, the particle effects increase when the player dies, which may subsequently lead to starving the buffer queue because the CPU can't keep up with our desired update rate. By the time the game gets to the reload routine, there may be very few buffers remaining in the queue to endure the length of the reload routine time. Experimentally, I found 32 buffers at 16KB to be sufficient to be able to reload the game without encountering an underrun condition.^[34]

Now 32 buffers at 16KB buffer sizes add up to arguably a lot of memory to use when the point of streaming was to minimize our memory footprint in a given moment.^[35] When you begin memory tuning for your game, this should be one of the first areas to revisit. As a reference point to consider and shoot for, Daniel Peacock says he personally likes to use four buffers of 25 milliseconds. Buffer size is scaled based on the sample rate to fill 25 milliseconds.

There are other remedies you can try instead of increasing the number of buffers or sizes of the buffers. One approach is to identify the long blocking parts of your application and sprinkle some calls to queue more buffers throughout the routines. For example, in our reloading routine, we could try something aggressive and queue a buffer after every new BBRock we allocate.

Another solution is to use threads to update our OpenAL audio engine so it is not blocked. You could explicitly write your own multithreaded code. Alternatively. if you have access to another timer system like CADisplayLink on the iPhone or Core Video on the Mac (a high-priority timer running on an alternative thread that is guaranteed to fire in unison with your display's refresh rate), you could tie the OpenAL buffer updates to that. This is obviously a general statement to all OpenAL developers, not just iPhone developers, as this blocking problem may be an issue on all single-threaded engine designs. You will need to find a good system that works for you and your target platform(s).

And at the risk of stating the obvious, a very easy solution is to reduce the sample rate of your audio. A 44 kHz sample takes twice as much memory as a 22 kHz sample of the same duration. Another way to think of this is that you can fit twice as much play time in a single buffer using a 22 kHz sample as opposed to a 44 kHz sample.

While we have just accomplished streaming (stereo) music, we've only scratched the surface of what the streaming part of our audio engine can do. So, we're not just going to leave it there. We will do one more short embellishment to give you an idea of what kind of audio power you now have.

Next, we should add some missing infrastructure to EWSoundSourceObject. We currently have a playSound: method in the class, but not a playStream: method. Let's add it.

- (BOOL) playStream:(EWStreamBufferData*)stream_buffer_data
{
   OpenALSoundController* sound_controller = [OpenALSoundController
      sharedSoundController];
   if(sound_controller.inInterruption)
   {
      NSInvocation* an_invocation =
         CreateAutoreleasedInvocation(self,@selector(playStream:),
         stream_buffer_data, nil);
      [sound_controller queueEvent:an_invocation];
      // Yes or No?
      return YES;
    }
    else
    {
       ALuint source_id;
       BOOL is_source_available = [sound_controller reserveSource:&source_id];
       if(NO == is_source_available)
       {
          return NO;
       }

       self.sourceID = source_id;
       self.hasSourceID = YES;
       [self applyState];
       [sound_controller playStream:source_id streamBufferData:stream_buffer_data];
    }
    return YES;
}

This code is essentially a copy-and-paste from playSound:, but is a little simpler, because we pass the buck and let the OpenALSoundController figure out how to attach and queue the buffers.

Now we're finished filling in the missing infrastructure. Wow, that was too easy.

Next, let's modify BBSceneController to create a taunting UFO on player death. We will change the gameOver method to add two new lines:

-(void)gameOver
{
   UFOCountDown = RANDOM_INT(500,800);
   [self addTauntUFO];
   [inputController gameOver];
}

First, we reset the UFOCountDown timer, because we want to minimize the chance of having two UFOs at the same time. Since the UFO has a noisy engine, we would like to increase the chances of hearing the speech clearly. (You might wait for the first UFO to pass before dying.) For simplicity, if there is a UFO already in the scene, we leave it alone. There is a small probability that the existing UFO could destroy our new taunting UFO with its missiles, but we're not going to worry about that. Consider it an Easter Egg.

Next, we call a new method called addTauntUFO to create the UFO.

-(void)addTauntUFO
{
   BBUFO * ufo = [[BBUFO alloc] init];
   // The UFO starts in the upper left and moves to the right
   ufo.translation = BBPointMake(-270.0, 60.0, 0.0);
   ufo.speed = BBPointMake(50.0, 0.0, 0.0);
   ufo.scale = BBPointMake(30, 30, 30.0);
   ufo.rotation = BBPointMake(-20.0, 0.0, 0.0);
   ufo.rotationalSpeed = BBPointMake(0.0, 0.0, 50.0);
   ufo.shouldTaunt = YES;
   [self addObjectToScene:ufo];
   [ufo release];
}

This is very similar to addUFO. We just change the speed and rotation for distinctiveness, and adjust the y position so it is lower on the screen. This is to avoid overlapping with the regular UFO if it is on the screen. We also set a new property called shouldTaunt to YES, placing it before addObjectToScene:. We want this property set before the UFO's awake method is called so the correct audio sample will be used.

And that's all we need to change in BBSceneController. This is way too easy. Now we only need to modify BBUFO to taunt.

We will add the new Boolean property, shouldTaunt, to BBUFO. While we're here, let's add an ALuint for the tauntID. We will use this to save the OpenAL source ID that we are playing the taunt on for later use with a callback.

@interface BBUFO : BBMobileObject {
   BBParticleSystem * particleEmitter;
   NSInteger missileCountDown;
   BOOL destroyed;
   BOOL shouldTaunt;
   ALuint tauntID;
}
@property(nonatomic, assign) BOOL shouldTaunt;

After synthesizing the shouldTaunt property in the implementation, we modify the awake method to conditionally set the proper sound effect. If we are taunting, we load the gameover file as a stream. If we are not taunting, we load the engine sound as usual. (We could add a second source to the UFO to play both, but the engine is noisy, so this is fine.)

I want to reiterate that we could do this without touching a single 3D audio property, but the existing directional cone is going to make hearing the speech kind of difficult. So as a special case, we will disable the directional cones for taunting. (Perhaps it would have been better to subclass the UFO, but this is meant to be just a quick hack for demonstration purposes.)

if(YES == self.shouldTaunt)
        {
           self.soundSourceObject.coneInnerAngle = 360.0;
           self.soundSourceObject.coneOuterAngle = 360.0;
           self.soundSourceObject.coneOuterGain = 0.0;
           self.soundSourceObject.rolloffFactor = 0.5;
           self.soundSourceObject.referenceDistance = 300.0;

           self.soundSourceObject.audioLooping = AL_FALSE;
           self.soundSourceObject.gainLevel = 1.0;

           EWStreamBufferData* game_over_speech =
             [[OpenALSoundController sharedSoundController]
             streamBufferDataFromFileBaseName:GAME_OVER_SPEECH];
           [self.soundSourceObject playStream:game_over_speech];
           tauntID = self.soundSourceObject.sourceID;
        }
        else
        {
           self.soundSourceObject.coneInnerAngle = 90.0;
           self.soundSourceObject.coneOuterAngle = 270.0;
           self.soundSourceObject.coneOuterGain = 0.50;
           self.soundSourceObject.rolloffFactor = 0.5;
           self.soundSourceObject.referenceDistance = 300.0;

           self.soundSourceObject.audioLooping = AL_TRUE;
           self.soundSourceObject.gainLevel = 0.3; // let's lower sound it's too loud
          [self.soundSourceObject playSound:
[[OpenALSoundController sharedSoundController]
             soundBufferDataFromFileBaseName:UFO_ENGINE]];
   }

With streaming, we need to remember to always turn off looping on the source, because we do looping in the buffer. In this case, we aren't looping at all, so both need to be off. We load the file and play it. And we save the source ID in tauntID. We release the newly created buffer because we don't need it anymore. (The system will retain it as long as it is playing.)

In the dealloc method, let's make sure we stop the taunt. We could let it play to completion, but it makes more sense to kill the taunt if the player starts a new game.

if(AL_TRUE == self.soundSourceObject.audioLooping)
  {
     [self.soundSourceObject stopSound];
  }
  // Additional check needed for taunt mode. Let's kill the speech if we are resetting
  else if(YES == self.shouldTaunt)
  {
     [self.soundSourceObject stopSound];
  }

And that's it. We have a UFO that will taunt us! And this streaming speech will have all the same 3D effects that we applied to the engine hum. You can even hear the Doppler effect mess up the singing.

But let's now put the cherry on top. We will use the callback feature that we spent so much effort building in Chapter 10. When the UFO is finished speaking, we will do something very explicit so we know everything is working. We will have the UFO fire off a four-missile salvo to show off (see Figure 12-9). This will prove the callback system works!

Figure 12.9. Using the callback system we implemented in Chapter 10, we instruct the UFO to fire a four-missile salvo after the taunt.

Let's implement the sound callback.

- (void) soundDidFinishPlaying:(NSNumber*)source_number
{
   [super soundDidFinishPlaying:source_number];

   if(YES == self.shouldTaunt && [source_number unsignedIntValue] == tauntID)
   {
      tauntID = 0;
      [self fireMissile];
      [self fireMissile];
      [self fireMissile];
      [self fireMissile];
   }
}

If we are taunting and get back the tauntID we saved, that means the source finished playing the taunt.

Build and run the game. With luck, everything will just work. Congratulations! You have completed the entire OpenAL engine. Yes, that was a lot of work, but you have everything! You have resource management, callbacks, 3D effects, and streaming—all working in tandem. In my opinion, this is much better than what we had in the AVFoundation music integration example. And the Audio Queue Services solution discussed in the next section will have the same impedance mismatch as AVFoundation.

We have been using Apple's alBufferDataStatic extension, introduced in Chapter 10, to make audio streamingmore efficient, but it has a nonobvious behavior that cancause your application to crash if you're not careful to work around it.

To refresh your memory, the alBufferDataStatic extension works by allowing you to provide a pointer to the raw PCM data for your audio for an OpenAL buffer. This is in contrast to the standard alBufferData command, which will copy the PCM data to the OpenAL buffer. The alBufferDataStatic extension gives an edge in performance because you avoid the memory copy. But as a trade-off, you must manage the memory for the PCM data yourself.

Managing the memory yourself entails two simple concepts:

Let's begin with a basic example. We start playing a sound in OpenAL using a buffer created from the alBufferDataStatic extension. If we try to free the raw PCM buffer while OpenAL is still playing this sound, we will likely crash our program. So obviously, we shouldn't free our PCM buffers while OpenAL is using them. This seems easy enough.

Let's extend the example. As we just did in the previous example with the taunting UFO, we start playing a buffer and poll OpenAL until it stops playing (or explicitly stop the sound ourselves). When OpenAL tells us the AL_SOURCE_STATE is AL_STOPPED, we detach the OpenAL buffer from the source, delete the OpenAL buffer using alDeleteBuffers(), and then free the raw PCM buffer. On the surface, this seems reasonable and should work. But there is an implementation detail that we must contend with. Apple's OpenAL implementation may still be using the raw PCM buffer under the hood, even though it seemingly appears the system is done with the buffer. And if Apple is still using the buffer, our program will likely crash. Unfortunately, the UFO taunting example has created the perfect storm for this race condition.

Why did this race condition appear only now, in the UFO taunting example? Well, this is the first time we delete buffers immediately after they finish playing. Please be aware that this is not a bug specific to streaming. This could happen with preloaded sounds, too. With our preloaded sounds from Chapter 10 and our streaming background music, we didn't delete the buffers until shutdown, and the race condition didn't occur in those situations.

Apple's response to dealing with this issue is to check for an OpenAL error after you call alDeleteBuffers(), but before you free the PCM data. Apple claims to have overloaded the error mechanism for this case. Officially, the only thing the spec says is that alDeleteBuffers returns the error AL_INVALID_NAME. So Apple says it added a special case to let you know that the deletion of the buffer has failed, which you should use as an indicator of whether it is actually safe to free the PCM data. Unfortunately, this doesn't seem to work for me.

So you can experience this yourself, let's add it to Space Rocks! (These code changes are included as part of SpaceRocksOpenALStreaming2.) In EWStreamBufferData.m, modify the destroyBuffers method to look like the following:

- (void) destroyOpenALBuffers
{
   ALenum al_error = alGetError(); // clear errors
   [availableDataBuffersQueue release];
   [queuedDataBuffersQueue release];

   alDeleteBuffers(EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS,
      openalDataBufferArray);
   al_error = alGetError();
   if(AL_NO_ERROR != al_error)
   {
      NSLog(@"EWStreamBufferData alDeleteBuffers error: %s", alGetString(al_error));
   }
#ifdef USE_BUFFER_DATA_STATIC_EXTENSION_FOR_STREAM
   for(NSUInteger i=0; i<EW_STREAM_BUFFER_DATA_MAX_OPENAL_QUEUE_BUFFERS; i++)
   {
       if(NULL != pcmDataBufferArray[i])
       {
          free(pcmDataBufferArray[i]);
          pcmDataBufferArray[i] = NULL;
       }
   }
#else
  if(NULL != pcmDataBuffer)
  {
     free(pcmDataBuffer);
     pcmDataBuffer = NULL;
        }
#endif
}

There are two things to notice:

To try to reproduce this problem, you should reactivate the line and recompile. Then run the game and destroy your ship to summon the taunting UFO. When the UFO stops speaking, the crashing bug has a chance of occurring. You may also tap the Try Again button while the UFO is in the middle of the taunt. This will immediately stop the playback and run the same cleanup code.^[37] For me, doing this creates a fairly reliable reproducible demonstration of the bug (i.e., over 50% crash rate).

In the HandleOutputBuffer callback, the critical change is to reset the mCurrentPacket to 0. Being even more explicit, the fifth parameter to AudioFileReadPackets tells the function which packet to start reading from. In the rewind case, it is 0.

// New rewind code
pAqData->mCurrentPacket = 0; // reset counter
UInt32 numPackets = pAqData->mNumPacketsToRead;
   result = AudioFileReadPackets(
   pAqData->mAudioFile,
   false,
   &numBytesReadFromFile,
   pAqData->mPacketDescs,
   0,  // start at the beginning (packet #0)
   &numPackets,
   inBuffer->mAudioData);

Apple talks about interruptions in Technical Q&A QA1558, found here:

http://developer.apple.com/iphone/library/qa/qa2008/qa1558.html

The crux is that starting in iPhone OS 3.0, audio queues are paused automatically in an interruption, and you just need to resume (if appropriate) on the endInterruption event. But there is a major caveat. If you are using the hardware decoder, all bets are off, because the hardware decoder may not be able to restore its state for ambiguous reasons. Since we are using the hardware decoder for our music, we can't rely on this feature.^[41] Thus, we must do it the pre-3.0 way by completely tearing down the audio queue on interruption and reloading it on endInterruption.

To accomplish the complete teardown and then restore the state, we must do the following:

Then on endInterruption, we create a new audio queue using the same file.^[42] We remember to set the mCurrentPacket to our saved value, and then start playing the queue. See the methods beginInterruption and endInterruption in AudioQueueServicesController.m.

A special change we need to make for Space Rocks! to handle interruptions is where we place the callback function. Originally, we had the MyInterruptionCallback function inside OpenALSoundController. The problem is that we need to handle interruptions for both OpenAL and audio queues, but they are in separate files that don't know about each other. So we move the callback and audio session initialization code to the BBSceneController.m file, since our scene controller needs to talk to both of these files to play sound and music. Note that we didn't have this problem with our AVFoundation/Space Rocks! integration because AVAudioPlayer has its own delegate callbacks that are invoked on interruption.

For more information about handling interruptions, see Apple's documentation entitled "Handling Audio Interruptions" which is part of the Audio Sessions Programming Guide, found here:

http://developer.apple.com/iphone/library/documentation/Audio/Conceptual/AudioSessionProgrammingGuide/HandlingAudioInterruptions/HandlingAudioInterruptions.html#//apple_ref/doc/uid/TP40007875-CH11-SW11

The OpenAL Capture API is pretty small. It consists of five functions:

ALCdevice* alcCaptureOpenDevice(const ALCchar* device_name, ALCuint sample_rate,
ALCenum al_format, ALCsizei buffer_size);
ALCboolean alcCaptureCloseDevice(ALCdevice* device_name);
void alcCaptureStart(ALCdevice* device_name);
void alcCaptureStop(ALCdevice* device_name);
void alcCaptureSamples(ALCdevice* device_name, ALCvoid* data_buffer, ALCsizei
  number_of_samples);

Open, close, start, and stop should be pretty obvious. The remaining function, alcCaptureSamples, is how you get PCM data back from the input device.

To start with, you might consider checking to see if a capture device actually exists. You can use Audio Session Services to tell you this on iPhone, as in the preceding example. With pure OpenAL, you could just try opening the device, which might be the best way of doing it. You can also try the ALC enumeration system, which lets you get a list of devices and find out the name of the default device. There are separate flags for getting input devices and output devices.

To be overzealous, you can check for the extension anyway (using the OpenAL extension mechanism). Then you can get the device name and list of devices. However, on iPhone OS as of this writing, the Enumeration extension seems to be broken. The extension returns true, but you get no device names back. (It fails on output devices, too.) So, you are better off just trying to open the device. But for interest, because dealing with a double NULL-terminated C array for string lists is probably something a lot of people aren't used to seeing, here is some sample code:

if(alcIsExtensionPresent(NULL, "ALC_ENUMERATION_EXT") == AL_TRUE)

{
    // Enumeration extension found
    printf("ALC_ENUMERATION_EXT available");

    const ALCchar* list_of_devices;
    const ALCchar* default_device_name;

    // Pass in NULL device handle to get list of devices
    default_device_name = alcGetString(NULL, ALC_CAPTURE_DEFAULT_DEVICE_SPECIFIER);

    // Devices contains the device names, separated by NULL
    // and terminated by two consecutive NULLs
    list_of_devices = alcGetString(NULL, ALC_CAPTURE_DEVICE_SPECIFIER);

    printf("Default capture device is %s
", default_device_name);
    const ALCchar* device_string_walk = list_of_devices;
    int device_num = 0;
    do
    {

       printf(" * Capture Device %d: [%s].
", device_num, device_string_walk);
       device_string_walk += strlen(device_string_walk)+1;
       device_num++;

    } while(device_string_walk[0] != ''),
}

To actually open the device, we do this:^[46]

alCaptureDevice = alcCaptureOpenDevice(NULL, 22050, AL_FORMAT_MONO16, 32768);

The first parameter is a C string containing the device name, which can be obtained from alcGetString(), as shown in the previous code snippet. Or we can just pass NULL to open the default capture device as we do here.

The second parameter is the sample rate. We use 22 kHz here. (Optionally, you could query the currentHardwareSampleRate, as discussed earlier.) If the rate is higher than what your hardware can support, OpenAL generally up-samples.

The third parameter is the data format. We are telling it we want to get back the data as 16-bit mono. Other options include 8-bit and stereo. You've seen these same data formats before when dealing with alBufferData.

The last parameter is how large of a buffer you want OpenAL to reserve for holding audio data. The larger the buffer, the larger the window of samples you get to see. Note that with higher sample rates and larger data formats, you need a larger buffer to hold the same number of samples as you would with lower rates and smaller formats. For simplicity, I somewhat arbitrarily pick 32KB here.

So, if we don't get back NULL from this call, we have an open device. Unlike with OpenAL output, we don't set up a context. OpenAL capture does not have a context. In fact, the three OpenAL object types—sources, buffers, and listeners—do not make an appearance in the OpenAL Capture API. You deal with the device and PCM buffers directly. (If you want to take the PCM buffer and play it, then you can give it to an OpenAL buffer and play it on a source.)

Once we are ready to start capturing samples, we call the following method:

alcCaptureStart(alCaptureDevice);

Once again, we need to remember that OpenAL has a polling-oriented design. That means we need to constantly check if OpenAL has gotten more input data. In our update loop, we query OpenAL to find out how many samples it has collected (see OpenALCaptureController.m's dataArray:maxArrayLength:getBytesPerSample). If we want the samples, we can retrieve them.

To find out how many samples OpenAL has collected, we do this:

ALCint number_of_samples = 0;
alcGetIntegerv(alCaptureDevice, ALC_CAPTURE_SAMPLES, 1, &number_of_samples);

Now we have some options. We can retrieve the PCM samples now, or we can wait until we accumulate more. In our demo program, we wait until we reach a certain threshold before we retrieve the data. This way, the rest of the code doesn't need to worry about always dealing with different amounts of data. One trade-off with this approach is latency. We have higher latency costs because we wait until we have a full buffer.

Finally, to retrieve the data, we do this:

alcCaptureSamples(alCaptureDevice, data_array, number_of_samples);

The variable data_array is the buffer where we want the OpenAL capture data to be copied, Once we retrieve the data, it will be removed from OpenAL's internal buffer. Also, keep in mind that the number of samples is not the same as the number of bytes. So we need to make sure the data_array we pass in is large enough to hold the number of samples we retrieve.

To compute the number of bytes needed for a given number of samples, the formula is as follows:

where:

or combined:

That's pretty much it. If you want to pause capturing, you can call this method:

alcCaptureStop(alCaptureDevice);

And to close the device, do this:

alcCloseDevice(alCaptureDevice);

So that is OpenAL capture in a nutshell. If you run the program on your Mac and your default input device is set to the internal microphone (go to System Preferences), you should just be able to make some noise and see the scope change.

This demo application does a few extracurricular activities that should be explained:

The OpenGL code itself demonstrates an OpenGL optimization technique, which is discussed in the next section.