It’s called The Amazing Audio Engine, and it represents the product of years of experience with iOS audio. It’s a sophisticated iOS audio engine that lets developers skip the Core Audio learning curve, and get on with writing great software.

The tech behind this is what drives Loopy and Loopy HD, as well as the in-development Audiobus app.

Subscribe at theamazingaudioengine.com to be kept in the loop as it approaches launch time.

Some of the features:

• Automatic mixing of multiple audio signals with per-channel volume and pan controls.
• Built-in support for audio filtering and effects, including the ability to form complex filter chains, constructing channel groups, or even whole trees of groups, and filtering them as one composite signal.
• Built-in support for audio input, including optional use of the Voice Processing IO unit, for automatic echo removal – great for VoIP.
• Record or monitor the output of the whole audio system, for in-app session recording, or get the output of one channel, or any group of channels in the processing tree.
• Support for any audio format (AudioStreamBasicDescription) that the hardware supports: Interleaved, non-interleaved, mono, stereo, 44.1kHz or any other supported sample rate, 16-bit, 8.24 fixed floating-point – whatever you need for your project.
• Very light, efficient engine, designed from the ground up for speed. All Core Audio code is pure C; no Objective- C or BSD calls, no locks, no memory allocation.
• Efficient mixing of input signals, using Apple’s MultiChannelMixer.
• Fast, lock-free synchronisation mechanism, enabling developers to send messages to the main thread from the Core Audio context, and vice versa, without locking or memory allocation from the Core Audio thread. Message sending from the main thread is two-way, and can be asynchronous, with a response block, or synchronous.

If you’re using TPCircularBuffer at all, I recommend updating!

This dramatically simplifies the use of a circular buffer — you can use chunks of the buffer without any need to worry about where the wrap point is.

See the new implementation, which is thread-safe with one consumer and one producer, with no need for locks, making it perfect for use with high-priority Core Audio threads, on GitHub: TPCircularBuffer.

There’s a basic example of its use over on the original post.

Hands up, backing away

Disclaimer: I am not at all an expert in realtime programming, or Mach, or iOS-device optimisation, so this is pretty much a fumble in the dark. I won’t be at all offended if anyone wishes to shoot me down and offer a more sensible solution — in fact, please do! Until then, watch as I stumble on…

Also note that there are often ways to eliminate the need for precise timing of this nature, by architecting code appropriately — when it comes to audio, for example, CoreAudio provides a very accurate time base in render callbacks. For things like metronomes or audio synthesizers, it’s always better to establish a starting time, and use the difference between the current time and the starting time in order to determine state, rather than using a timer to advance the state. Still, sometimes, you just need a timer…

What the blazes?

So, I’m working on an update to Loopy, which uses a shared clock object to synchronise tracks and a variety of events (like user interface updates or timed track manipulations). A tester noted that the mute/unmute quantisation feature that I’ve recently implemented, which will mute or unmute a loop at its starting point (rather than whenever you tap it), tends to overshoot a little, resulting in a small part of the beginning of the loop being audible.

Of course, there are other solutions to this particular problem (like stopping or starting playback from the audio render callback, and using Core Audio’s timestamps for exact timing), but I use timers in other places outside Core Audio’s domain, which makes Core Audio’s timing mechanism unavailable, and I wanted to see how accurate I could get the timing.

Our friend, mach_wait_until

I read in several places mention of the Mach API utility mach_wait_until (from mach/mach_time.h), which is very low-level and supposedly fairly accurate. So, based on that lead, I put together an Objective-C singleton class that launches a high-priority thread, and uses said thread to schedule events.

An NSArray of events are maintained, and a scheduleAction:target:inTimeInterval: routine creates and adds events to this array, then pokes the thread.

The thread grabs the next event in sequence, then uses mach_wait_until to sleep until the time of the next event arrives, then performs the specified action on the target. It’s kinda a DIY NSRunLoop.

Here’s a comparison between this technique, and just using performSelector:withObject:afterDelay: (which schedules a timer on the NSRunLoop), observed while performing various scheduled events within Loopy running on my iPhone 4 with the debugger, and derived by comparing the time of event execution with the event’s scheduled time:

That was attempt number 1: This seems to give us about 11.4ms better accuracy on average (three times more accurate).

Not bad, but it turns out mach_wait_until isn’t really that accurate, particularly if there’s a bunch of other stuff going on in other threads.

Spinning, for fun and profit

For my second attempt, the thread performs a mach_wait_until until just before the event is due, then performs a spin lock until the time arrives, using mach_absolute_time to compare the current time with the target time.

This gave further improved results — here’s that table again, but with the new scheme added, with a few different spin lock times:

It appears that the more stuff there is going on in other threads, the more likely the mach_absolute_time call is to overshoot. So, the more time spent in the spin lock, the more leeway mach_absolute_time has to wait too long. Of course, that’s at the cost of making the CPU twiddle its thumbs for the duration.

Better than a punch in the knee

The results weren’t quite as fantastic as I’d hoped — still within the same order of magnitude, that’s for sure — but the average case for the 200ms spinlock approach is 14ms, or 5.8 times, more accurate than the traditional approach, and the minimum case is dramatically better.

You know, I think if I was aware of the results in advance, I might not bother, but I’ll stick with my hard-won 14ms now that I’m here (that’s 617 audio samples, I’ll have you know).

If anyone’s curious about the implementation (or wants to take a stab at doing better), here it is, along with a wildly simplistic commandline test app: TPPreciseTimer.zip

Now to get back to some real work.

Addendum: GCD follow-up

Chris in the comments below suggested trying a GCD-based approach, using dispatch_after. Curious, I rigged it up, and these are the stats, collected the same way as above, added to the prior table:

So, they appear pretty much the same as the NSRunLoop stats.

Apple’s produced some sample code (iPhoneExtAudioFileConvertTest), which demonstrates how it’s done, but their implementation isn’t particularly easy to use in existing projects, as it requires some wrapping to make it play nice.

For my upcoming looper app Loopy, I’ve put together a simple Objective-C class that performs the conversion of any audio file to an AAC-encoded m4a, asynchronously with a delegate, or converts any audio provided by a data source class (which provides for recording straight to AAC) and I thought I’d share it.

Grab the code, and a sample project demonstrating its use at the GitHub repository for TPAACAudioConverter.

To use it:

• Include the class in your project, and make sure you’ve got the AudioToolbox framework added, too.
• Audio session setup:

If you already have an audio session set up in your app, make sure you disable mixing with other device audio for the duration of the copy operation, as this stops the hardware encoder from working (you’ll see funny errors like kAudioQueueErr_InvalidCodecAccess (Error 66672)). I know that AVAudioSessionCategoryPlayAndRecord, AVAudioSessionCategorySoloAmbient and AVAudioSessionCategoryAudioProcessing work for sure. TPAACAudioConverter will automatically disable kAudioSessionProperty_OverrideCategoryMixWithOthers, if it’s set.

If you’re not already setting up an audio session, you could do so just before you start the conversion process.

You’ll need to provide an interruption handler to be notified of audio session interruptions, which impact the encoding process. You’ll also need to create a member variable to store the converter instance, so you can tell it when interruptions begin and end (via interrupt and resume).

// Callback to be notified of audio session interruptions (which have an impact on the conversion process)
static void interruptionListener(void *inClientData, UInt32 inInterruption)
{
	AACConverterViewController *THIS = (AACConverterViewController *)inClientData;
 
	if (inInterruption == kAudioSessionEndInterruption) {
		// make sure we are again the active session
		checkResult(AudioSessionSetActive(true), "resume audio session");
        if ( THIS->audioConverter ) [THIS->audioConverter resume];
	}
 
	if (inInterruption == kAudioSessionBeginInterruption) {
        if ( THIS->audioConverter ) [THIS->audioConverter interrupt];
    }
}
 
/*snip*/
 
-(void)startConverting {
 
    /*snip*/
 
    // Initialise audio session, and register an interruption listener, important for AAC conversion
    if ( !checkResult(AudioSessionInitialize(NULL, NULL, interruptionListener, self), "initialise audio session") ) {
        [[[[UIAlertView alloc] initWithTitle:NSLocalizedString(@"Converting audio", @"")
                                     message:NSLocalizedString(@"Couldn't initialise audio session!", @"")
                                    delegate:nil
                           cancelButtonTitle:nil
                           otherButtonTitles:NSLocalizedString(@"OK", @""), nil] autorelease] show];
        return;
    }
 
 
    // Set up an audio session compatible with AAC conversion.  Note that AAC conversion is incompatible with any session that provides mixing with other device audio.
    UInt32 audioCategory = kAudioSessionCategory_MediaPlayback;
    if ( !checkResult(AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, sizeof(audioCategory), &audioCategory), "setup session category") ) {
        [[[[UIAlertView alloc] initWithTitle:NSLocalizedString(@"Converting audio", @"")
                                     message:NSLocalizedString(@"Couldn't setup audio category!", @"")
                                    delegate:nil
                           cancelButtonTitle:nil
                           otherButtonTitles:NSLocalizedString(@"OK", @""), nil] autorelease] show];
        return;
    } 
 
 
    /*snip*/
}

• Make the relevant view controller implement the TPAACAudioConverterDelegate protocol: That means implementing AACAudioConverterDidFinishConversion:, and AACAudioConverter:didFailWithError:, and optionally AACAudioConverter:didMakeProgress: to receive progress updates.
• Create an instance of the converter, pass it the view controller as the delegate, and call start:

audioConverter = [[[TPAACAudioConverter alloc] initWithDelegate:self 
                                                         source:mySourcePath
                                                    destination:myDestinationPath] autorelease];
 
[audioConverter start];

Alternatively, if you wish to encode live audio, or provide another source of audio data, you can implement the TPAACAudioConverterDataSource protocol, which defines AACAudioConverter:nextBytes:length:, which provides a buffer to copy at most “length” bytes of audio into, and then expects you to update “length” to the amount of bytes provided. For that you’ll need to use the second initialiser, initWithDelegate:dataSource:audioFormat:destination:.

I noted previously that you can’t encode AAC live, which is what Apple’s docs say, but Alex in the comments informed me that it wasn’t so. So, I added the datasource method, and sure enough, it does work live!

The one caveat is that it’s a relatively heavy process. As it turns out, because my app Loopy is busily mixing and displaying visualisations and such, it was too much to also encode straight to AAC, and I was getting glitches. But it would probably work fine for plain recording. Thanks, Alex!

They’re designed for FIFO (first-in-first-out) use, like storing audio coming in the microphone for later playback or processing.

Consider a naive alternative: You copy the incoming audio into an NSData you allocate, and then pass that NSData off. This means you’re allocating memory each time, and deallocating the memory later once you’re done processing. That allocation incurs a penalty, which can be a show-stopper when part of an audio pipeline – The Core Audio documentation advises against any allocations when within a render callback, for example.

Alternatively, you can allocate space in advance, and write to that, but that has problems too: Either you have a synchronisation nightmare, or you spend lots of time moving bytes around so that the unprocessed audio is always at the beginning of the array.

A better solution is to use a circular buffer, where data goes in at the head, and is read from the tail. When you produce data at the head, the head moves up the array, and wraps around at the end. When you consume at the tail, the tail moves up too, so the tail chases the head around the circle.

Here’s a simple C implementation I recently put together for my app Loopy: TPCircularBuffer

typedef struct {
    void             *buffer;
    int32_t           length;
    int32_t           tail;
    int32_t           head;
    volatile int32_t  fillCount;
} TPCircularBuffer;
 
bool  TPCircularBufferInit(TPCircularBuffer *buffer, int32_t length);
void  TPCircularBufferCleanup(TPCircularBuffer *buffer);
void  TPCircularBufferClear(TPCircularBuffer *buffer);
 
// Reading (consuming)
void* TPCircularBufferTail(TPCircularBuffer *buffer, int32_t* availableBytes);
void  TPCircularBufferConsume(TPCircularBuffer *buffer, int32_t amount);
 
// Writing (producing)
void* TPCircularBufferHead(TPCircularBuffer *buffer, int32_t* availableBytes);
void  TPCircularBufferProduce(TPCircularBuffer *buffer, int32_t amount);
int   TPCircularBufferProduceBytes(TPCircularBuffer *buffer, const void* src, int32_t len);

Update: The implementation has been recently amended to make use of a neat virtual memory mapping technique that inserts a virtual copy of the buffer memory directly after the buffer’s end, negating the need for any buffer wrap-around logic. Clients can simply use the returned memory address as if it were contiguous space.

The virtual memory technique was originally proposed by Philip Howard, and adapted to Darwin by Kurt Revis.

Use TPCircularBufferHead to get a pointer to write to the buffer, followed by TPCircularBufferProduce to submit the written data, then TPCircularBufferTail to get a pointer to the next data to read, followed by TPCircularBufferConsume to free up the space once processed.

TPCircularBufferProduceBytes is a convenience routine for writing data straight to the buffer.

The implementation is thread-safe (no need for locks) in the case of a single producer and single consumer.

Here’s an example, using circular buffers to implement a simple playthrough scheme that plays audio coming in the microphone:

@interface MyAudioController : NSObject {
  TPCircularBuffer buffer;
}
 
@end
 
#define kBufferLength 1024
 
@implementation MyAudioController
 
- (void)setup {
    // Initialise buffer
    TPCircularBufferInit(&buffer, kBufferLength);
 
    // Setup audio, etc
}
 
- (void)dealloc {
    // Release buffer resources
    TPCircularBufferCleanup(&buffer);
 
    [super dealloc];
}
 
static OSStatus audioInputCallback(void *inRefCon, 
                                   AudioUnitRenderActionFlags *ioActionFlags, 
                                   const AudioTimeStamp *inTimeStamp, 
                                   UInt32 inBusNumber, 
                                   UInt32 inNumberFrames, 
                                   AudioBufferList *ioData) {
    MyAudioController *THIS = (MyAudioController *)inRefCon;
 
    // Render audio into buffer
    AudioBufferList bufferList;
    bufferList.mNumberBuffers = 1;
    bufferList.mBuffers[0].mNumberChannels = 2;
    bufferList.mBuffers[0].mData = NULL;
    bufferList.mBuffers[0].mDataByteSize = inNumberFrames * sizeof(SInt16) * 2;
    OSStatus err = AudioUnitRender(THIS->ioAudioUnit, ioActionFlags, inTimeStamp, kInputBus, inNumberFrames, &bufferList);
    if ( !checkResultLite(err, "AudioUnitRender") ) { return err; }
 
    // Put audio into circular buffer
    TPCircularBufferProduceBytes(&THIS->buffer, bufferList.mBuffers[0].mData, inNumberFrames * 2 * sizeof(SInt16));
}
 
static OSStatus audioOutputCallback(void *inRefCon, 
                                    AudioUnitRenderActionFlags *ioActionFlags, 
                                    const AudioTimeStamp *inTimeStamp, 
                                    UInt32 inBusNumber, 
                                    UInt32 inNumberFrames, 
                                    AudioBufferList *ioData) {
    MyAudioController *THIS = (MyAudioController *)inRefCon;
 
    int bytesToCopy = ioData->mBuffers[0].mDataByteSize;
    SInt16 *targetBuffer = (SInt16*)ioData->mBuffers[0].mData;
 
    // Pull audio from playthrough buffer
    int32_t availableBytes;
    SInt16 *buffer = TPCircularBufferTail(&THIS->buffer, &availableBytes);
    memcpy(targetBuffer, buffer, MIN(bytesToCopy, availableBytes));
    TPCircularBufferConsume(&THIS->buffer, sampleCount);
 
    return noErr;
}
@end

With digital audio though, the volume doesn’t go up to oh-god-please-make-them-stop – it’s limited to a small dynamic range.

So, digital mixing actually requires a little thought in order to avoid overflowing these bounds and clipping. I recently came across this when writing some mixing routines for my upcoming app Loopy 2, and found a very useful discussion on mixing digital audio by software developer and author Viktor Toth.

The basic concept is to mix in such a way that we stay within the dynamic range of the target audio format, while representing the dynamics of the mixed signals as faithfully as possible.Note that a simple average of the samples (as in, (sample 1 + sample 2) / 2) won’t accomplish this – for example, if sample 1 is silent, while sample 2 is happily jamming away, sample 2 will be halved in volume.

Instead, we want to meet three goals – assuming signed audio samples, the standard format for Remote IO/audio units on the iPhone/iPad, which can range from negative, through to zero (silence), up to positive values.

1. If both samples are positive, we mix them so that the output value is somewhere between the maximum value of the two samples, and the maximum possible value
2. If both samples are negative, we mix them so that the output value is somewhere between the minimum value of the two samples, and the minimum possible value
3. If one sample is positive, and one is negative, we want them to cancel out somewhat

If we’re talking about signed samples, MIN…0…MAX, this does the trick:

This lets the volume level for both samples remain the same, while fitting within the available range.

Here’s how it’s done on iOS:

SInt16 *bufferA, SInt16 *bufferB;
NSInteger bufferLength;
SInt16 *outputBuffer;
 
for ( NSInteger i=0; i<bufferLength; i++ ) {
  if ( bufferA[i] < 0 && bufferB[i] < 0 ) {
    // If both samples are negative, mixed signal must have an amplitude between 
    // the lesser of A and B, and the minimum permissible negative amplitude
    outputBuffer[i] = (bufferA[i] + bufferB[i]) - ((bufferA[i] * bufferB[i])/INT16_MIN);
  } else if ( bufferA[i] > 0 && bufferB[i] > 0 ) {
    // If both samples are positive, mixed signal must have an amplitude between the greater of
    // A and B, and the maximum permissible positive amplitude
    outputBuffer[i] = (bufferA[i] + bufferB[i]) - ((bufferA[i] * bufferB[i])/INT16_MAX);
  } else {
    // If samples are on opposite sides of the 0-crossing, mixed signal should reflect 
    // that samples cancel each other out somewhat
    outputBuffer[i] = bufferA[i] + bufferB[i];
  }
}

Update: A reader recently demonstrated that this technique can introduce some unpleasant distortion with certain kinds of input — as the algorithm is nonlinear, some distortion is inevitable (see the sharp points on the waveform where the condition switches over). For the kind of audio I’m mixing, the results seem to be perfectly adequate, but this may not be generally true.

Update 2: Here’s an inline function I put together for neatness:

inline SInt16 TPMixSamples(SInt16 a, SInt16 b) {
    return  
            // If both samples are negative, mixed signal must have an amplitude between the lesser of A and B, and the minimum permissible negative amplitude
            a < 0 && b < 0 ?
                ((int)a + (int)b) - (((int)a * (int)b)/INT16_MIN) :
 
            // If both samples are positive, mixed signal must have an amplitude between the greater of A and B, and the maximum permissible positive amplitude
            ( a > 0 && b > 0 ?
                ((int)a + (int)b) - (((int)a * (int)b)/INT16_MAX)
 
            // If samples are on opposite sides of the 0-crossing, mixed signal should reflect that samples cancel each other out somewhat
            :
                a + b);
}

He was quite optimistic, and as it turns out, he was right: It’s rather capable.

I modified the iPhoneMultichannelMixerTest sample project to add a bunch of channels, and measured how my iPhone 4 performed. It looks pretty linear: there’s pretty much a 1:1 relationship between number of channels, and the CPU usage, actually.

Of course, this is on the newest-most powerful iPhone, but there was no stuttering, and the interface (admittedly simple as it is) was fully responsive, including setting output volume, even with 100 channels. You’d probably want to stick with a maximum number of channels around the 75-100 mark, less for targeting lesser devices, but that’s a pretty generous limit.

Not bad.

Update: Not such great news for the iPhone 3G I just tested this on, though — it freaks at anything more than 20 channels, and isn’t too responsive with 20. The 3Gs seems to behave almost as well as the iPhone 4, but the CPU:channels relationship is more like 2:1.

Any app that makes use of the Remote IO audio unit framework (which is generally necessary for the kind of responsiveness required in a realtime musical app) provides audio to the hardware via a callback, which is periodically called when the hardware is ready for more.

The trick here is to provide the right chunk of samples in this callback for the current time position.

Loopy achieves this by:

1. Keeping track of where in the timeline we are at the time the callback is called

This is easily accomplished by keeping a record of the time the clock was started, subtracting this from the current time, and possibly performing a modulus with the tempo. For example:

• (now - startTime) % timePerBar gives the number of time units into the current bar (lets call it timeIntoBar).
• timeIntoBar / (timePerBar/beatsPerBar) gives the number of beats into the current bar, and
• timeIntoBar % (timePerBar/beatsPerBar) gives us the time into the current beat.

2. Determining first if we should be playing audio at this time, and if so, which samples should be playing

This involves first converting our time units from step 1 into samples. For instance, you can convert microseconds to samples by dividing your time by 1000000/yourSampleRate. Aside: Of course, you can convert back from samples to time by multiplying instead of dividing.

Next, in the case of Loopy’s metronome, for example, we test for whether samplesIntoBeat < sound.lengthInSamples. If so, that means we should be playing audio. If the sound was a loop, of course, we could be always playing.

The offset into the sound, in samples, is just samplesIntoBeat, in the case of the simple metronome. In the case of a loop, you probably will be more interested in the number of samples into your loop — so instead of determining (now - startTime) % timePerBar, you may be interested in (now - startTime) % timePerLoop.

So, we want to return the requested number of samples starting from this offset into the sample array representing our audio.

3. Returning smooth audio in time

Note that if you just go returning any old set of samples, willy-nilly, you’re going to get nasty clicks and pops from discontinuities you get by not matching the start of your next buffer to the last one.

To ensure smoothness, Loopy keeps track of the offset of the last samples we returned, and just return the immediately following bunch of samples — unless we’re more than some threshold number of samples out of time, in which case we’ll suffer the pop in order to stay synchronised. Actually, you can even generally avoid the pop if you smoothly blend buffers over a short time, removing any discontinuity.

Final words

The example above was a relatively trivial one, for a metronome sound. For longer audio that may span multiple bars, you’ll probably want to perform a modulus by the length of your audio clip, possibly quantised to your time signature, and possibly using a per-loop time base, so you can start the loop at any point in your timeline and have it begin from the start. This is something Loopy doesn’t currently do — Loopy will keep your loops synchronised so when you start a loop playing, it’ll play whatever part corresponds to the current timeline, not from the start of the loop. Maybe it’ll be an option in the future?

I wrote a little about the timing of loops in my second article on Loopy’s implementation.

Some googling turned up this thread on Stack Overflow, which presented a solution for the AVAudioSession framework: It turns out, on recent SDKs (somewhere equal to or above 3.0), the audio system will refuse to start up if the session is set to a recording mode, with no input device available, spitting out this error. Alas, the error isn’t listed anywhere that I could find.

So, the way to fix it is to check whether there’s an input device available, then choose an audio category accordingly:

OSStatus status;
 
UInt32 inputAvailable=0;
UInt32 size = sizeof(inputAvailable);
AudioSessionGetProperty(kAudioSessionProperty_AudioInputAvailable, 
                        &size, 
                        &inputAvailable);
UInt32 sessionCategory;
if ( inputAvailable ) {
    // Set the audio session category for simultaneous play and record
    sessionCategory = kAudioSessionCategory_PlayAndRecord;
} else {
    // Just playback
    sessionCategory = kAudioSessionCategory_MediaPlayback;
}
 
status = AudioSessionSetProperty (kAudioSessionProperty_AudioCategory,
                                  sizeof (sessionCategory),
                                  &sessionCategory);    
checkStatus(status);

It’s probably a good idea to respond when input becomes available or goes away, so add a property listener too:

void inputAvailablePropertyListener (void                      *inClientData,
                                     AudioSessionPropertyID    inID,
                                     UInt32                    inDataSize,
                                     const void                *inData) {
    if ( inID = kAudioSessionProperty_AudioInputAvailable ) {
        UInt32 *inputAvailable = (UInt32*)inData;
        UInt32 sessionCategory;
        if ( *inputAvailable ) {
            // Set the audio session category for simultaneous play and record
            sessionCategory = kAudioSessionCategory_PlayAndRecord;
        } else {
            // Just playback
            sessionCategory = kAudioSessionCategory_MediaPlayback;
        }
 
        OSStatus status = AudioSessionSetProperty (kAudioSessionProperty_AudioCategory,
                                                   sizeof (sessionCategory),
                                                   &sessionCategory);    
        checkStatus(status);
    }
}

then:

// Listen for audio input availability
status = AudioSessionAddPropertyListener(kAudioSessionProperty_AudioInputAvailable,
                                         inputAvailablePropertyListener, 
                                         NULL);

Wanna skip the Core Audio learning curve and start writing code straight away? Check out my new project:

Update: Thanks to Joel Reymont, we now have an explanation for the “CrashIfClientProvidedBogusAudioBufferList” iPhone simulator bug: The simulator doesn’t like mono audio. Thanks, Joel!

Update: Happily, Apple have now created some excellent documentation on Remote IO, with some good sample projects. I recommend using that as a resource, now that it’s there, as that will continue to be updated.

Update: Tom Zicarelli has created a very extensive sample app that demonstrates the use of AUGraph, with all sorts of goodies.

So, we need to obtain an instance of the RemoteIO audio unit, configure it, and hook it up to a recording callback, which is used to notify you that there is data ready to be grabbed, and where you pull the data from the audio unit.

Overview

1. Identify the audio component (kAudioUnitType_Output/ kAudioUnitSubType_RemoteIO/ kAudioUnitManufacturerApple)
2. Use AudioComponentFindNext(NULL, &descriptionOfAudioComponent) to obtain the AudioComponent, which is like the factory with which you obtain the audio unit
   
3. Use AudioComponentInstanceNew(ourComponent, &audioUnit) to make an instance of the audio unit
   
4. Enable IO for recording and possibly playback with AudioUnitSetProperty
   
5. Describe the audio format in an AudioStreamBasicDescription structure, and apply the format using AudioUnitSetProperty
   
6. Provide a callback for recording, and possibly playback, again using AudioUnitSetProperty
   
7. Allocate some buffers
   
8. Initialise the audio unit
   
9. Start the audio unit
   
10. Rejoice
    

Here’s my code: I’m using both recording and playback. Use what applies to you!

Initialisation

Initialisation looks like this. We have a member variable of type AudioComponentInstance which will contain our audio unit.

The audio format described below uses SInt16 for samples (i.e. signed, 16 bits per sample)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

#define kOutputBus 0
#define kInputBus 1
 
// ...
 
 
OSStatus status;
AudioComponentInstance audioUnit;
 
// Describe audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
 
// Get component
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
 
// Get audio units
status = AudioComponentInstanceNew(inputComponent, &audioUnit);
checkStatus(status);
 
// Enable IO for recording
UInt32 flag = 1;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioOutputUnitProperty_EnableIO, 
                              kAudioUnitScope_Input, 
                              kInputBus,
                              &flag, 
                              sizeof(flag));
checkStatus(status);
 
// Enable IO for playback
status = AudioUnitSetProperty(audioUnit, 
                              kAudioOutputUnitProperty_EnableIO, 
                              kAudioUnitScope_Output, 
                              kOutputBus,
                              &flag, 
                              sizeof(flag));
checkStatus(status);
 
// Describe format
audioFormat.mSampleRate			= 44100.00;
audioFormat.mFormatID			= kAudioFormatLinearPCM;
audioFormat.mFormatFlags		= kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
audioFormat.mFramesPerPacket	= 1;
audioFormat.mChannelsPerFrame	= 1;
audioFormat.mBitsPerChannel		= 16;
audioFormat.mBytesPerPacket		= 2;
audioFormat.mBytesPerFrame		= 2;
 
// Apply format
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_StreamFormat, 
                              kAudioUnitScope_Output, 
                              kInputBus, 
                              &audioFormat, 
                              sizeof(audioFormat));
checkStatus(status);
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_StreamFormat, 
                              kAudioUnitScope_Input, 
                              kOutputBus, 
                              &audioFormat, 
                              sizeof(audioFormat));
checkStatus(status);
 
 
// Set input callback
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = recordingCallback;
callbackStruct.inputProcRefCon = self;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioOutputUnitProperty_SetInputCallback, 
                              kAudioUnitScope_Global, 
                              kInputBus, 
                              &callbackStruct, 
                              sizeof(callbackStruct));
checkStatus(status);
 
// Set output callback
callbackStruct.inputProc = playbackCallback;
callbackStruct.inputProcRefCon = self;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_SetRenderCallback, 
                              kAudioUnitScope_Global, 
                              kOutputBus,
                              &callbackStruct, 
                              sizeof(callbackStruct));
checkStatus(status);
 
// Disable buffer allocation for the recorder (optional - do this if we want to pass in our own)
flag = 0;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_ShouldAllocateBuffer,
                              kAudioUnitScope_Output, 
                              kInputBus,
                              &flag, 
                              sizeof(flag));
 
// TODO: Allocate our own buffers if we want
 
// Initialise
status = AudioUnitInitialize(audioUnit);
checkStatus(status);

Then, when you’re ready to start:

1
2

OSStatus status = AudioOutputUnitStart(audioUnit);
checkStatus(status);

And to stop:

1
2

OSStatus status = AudioOutputUnitStop(audioUnit);
checkStatus(status);

Then, when we’re finished:

1

AudioUnitUninitialize(audioUnit);

And now for our callbacks.

Recording

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

static OSStatus recordingCallback(void *inRefCon, 
                                  AudioUnitRenderActionFlags *ioActionFlags, 
                                  const AudioTimeStamp *inTimeStamp, 
                                  UInt32 inBusNumber, 
                                  UInt32 inNumberFrames, 
                                  AudioBufferList *ioData) {
 
    // TODO: Use inRefCon to access our interface object to do stuff
    // Then, use inNumberFrames to figure out how much data is available, and make
    // that much space available in buffers in an AudioBufferList.
 
    AudioBufferList *bufferList; // <- Fill this up with buffers (you will want to malloc it, as it's a dynamic-length list)
 
    // Then:
    // Obtain recorded samples
 
    OSStatus status;
 
    status = AudioUnitRender([audioInterface audioUnit], 
                             ioActionFlags, 
                             inTimeStamp, 
                             inBusNumber, 
                             inNumberFrames, 
                             bufferList);
    checkStatus(status);
 
    // Now, we have the samples we just read sitting in buffers in bufferList
    DoStuffWithTheRecordedAudio(bufferList);
    return noErr;
}

Playback

1
2
3
4
5
6
7
8
9
10
11

static OSStatus playbackCallback(void *inRefCon, 
                                  AudioUnitRenderActionFlags *ioActionFlags, 
                                  const AudioTimeStamp *inTimeStamp, 
                                  UInt32 inBusNumber, 
                                  UInt32 inNumberFrames, 
                                  AudioBufferList *ioData) {    
    // Notes: ioData contains buffers (may be more than one!)
    // Fill them up as much as you can. Remember to set the size value in each buffer to match how
    // much data is in the buffer.
    return noErr;
}

Finally, rejoice with me in this discovery ;)

Resources that helped

• http://pastie.org/pastes/219616
• http://developer.apple.com/samplecode/CAPlayThrough/listing8.html
• http://listas.apesol.org/pipermail/svn-libsdl.org/2008-July/000797.html

No thanks at all to Apple for their lack of accessible documentation on this topic – They really have a long way to go here! Also boo to them with their lack of search engine, and refusal to open up their docs to Google. It’s a jungle out there!

Update: You can adjust the latency of RemoteIO (and, in fact, any other audio framework) by setting the kAudioSessionProperty_PreferredHardwareIOBufferDuration property:

float aBufferLength = 0.005; // In seconds
AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration, 
                        sizeof(aBufferLength), &aBufferLength);

This adjusts the length of buffers that’re passed to you – if buffer length was originally, say, 1024 samples, then halving the number of samples halves the amount of time taken to process them.

The error occurs when using AudioQueueEnqueueBuffer. It happens when one tries to enqueue a buffer when the queue in question is no longer running.

Wrap the AudioQueueEnqueueBuffer in a statement that checks to see if the queue is actually running, as in the SpeakHere/SpeakHear (depending on where you look) example. Something like:

if ( [track recording] ) {

  // Re-enqueue this buffer

  status = AudioQueueEnqueueBuffer (inAudioQueue,

                     inBuffer,

                     0,

                     NULL);

  checkStatus(status);

}

..Should do the trick.