November 4, 2008

Using RemoteIO audio unit

by Michael Tyson in Geekspeak

I’ve had nasty old time trying to get some audio stuff going on the iPhone, no thanks to Apple’s lack of documentation. If you’re an iPhone developer interested in getting RemoteIO/IO Remote/whatever it’s called working on the iPhone… Do I have good news for you. Read on.

Wanna skip the Core Audio learning curve and start writing code straight away? Check out my new project:

The Amazing Audio Engine: Core Audio, Cordially

Update: Thanks to Joel Reymont, we now have an explanation for the “CrashIfClientProvidedBogusAudioBufferList” iPhone simulator bug: The simulator doesn’t like mono audio. Thanks, Joel!

Update: Happily, Apple have now created some excellent documentation on Remote IO, with some good sample projects. I recommend using that as a resource, now that it’s there, as that will continue to be updated.

Update: Tom Zicarelli has created a very extensive sample app that demonstrates the use of AUGraph, with all sorts of goodies.

So, we need to obtain an instance of the RemoteIO audio unit, configure it, and hook it up to a recording callback, which is used to notify you that there is data ready to be grabbed, and where you pull the data from the audio unit.


Overview

  1. Identify the audio component (kAudioUnitType_Output/ kAudioUnitSubType_RemoteIO/ kAudioUnitManufacturerApple)
  2. Use AudioComponentFindNext(NULL, &descriptionOfAudioComponent) to obtain the AudioComponent, which is like the factory with which you obtain the audio unit
  3. Use AudioComponentInstanceNew(ourComponent, &audioUnit) to make an instance of the audio unit
  4. Enable IO for recording and possibly playback with AudioUnitSetProperty
  5. Describe the audio format in an AudioStreamBasicDescription structure, and apply the format using AudioUnitSetProperty
  6. Provide a callback for recording, and possibly playback, again using AudioUnitSetProperty
  7. Allocate some buffers
  8. Initialise the audio unit
  9. Start the audio unit
  10. Rejoice

Here’s my code: I’m using both recording and playback. Use what applies to you!

Initialisation

Initialisation looks like this. We have a member variable of type AudioComponentInstance which will contain our audio unit.

The audio format described below uses SInt16 for samples (i.e. signed, 16 bits per sample)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

#define kOutputBus 0
#define kInputBus 1
 
// ...
 
 
OSStatus status;
AudioComponentInstance audioUnit;
 
// Describe audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
 
// Get component
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
 
// Get audio units
status = AudioComponentInstanceNew(inputComponent, &audioUnit);
checkStatus(status);
 
// Enable IO for recording
UInt32 flag = 1;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioOutputUnitProperty_EnableIO, 
                              kAudioUnitScope_Input, 
                              kInputBus,
                              &flag, 
                              sizeof(flag));
checkStatus(status);
 
// Enable IO for playback
status = AudioUnitSetProperty(audioUnit, 
                              kAudioOutputUnitProperty_EnableIO, 
                              kAudioUnitScope_Output, 
                              kOutputBus,
                              &flag, 
                              sizeof(flag));
checkStatus(status);
 
// Describe format
audioFormat.mSampleRate			= 44100.00;
audioFormat.mFormatID			= kAudioFormatLinearPCM;
audioFormat.mFormatFlags		= kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
audioFormat.mFramesPerPacket	= 1;
audioFormat.mChannelsPerFrame	= 1;
audioFormat.mBitsPerChannel		= 16;
audioFormat.mBytesPerPacket		= 2;
audioFormat.mBytesPerFrame		= 2;
 
// Apply format
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_StreamFormat, 
                              kAudioUnitScope_Output, 
                              kInputBus, 
                              &audioFormat, 
                              sizeof(audioFormat));
checkStatus(status);
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_StreamFormat, 
                              kAudioUnitScope_Input, 
                              kOutputBus, 
                              &audioFormat, 
                              sizeof(audioFormat));
checkStatus(status);
 
 
// Set input callback
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = recordingCallback;
callbackStruct.inputProcRefCon = self;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioOutputUnitProperty_SetInputCallback, 
                              kAudioUnitScope_Global, 
                              kInputBus, 
                              &callbackStruct, 
                              sizeof(callbackStruct));
checkStatus(status);
 
// Set output callback
callbackStruct.inputProc = playbackCallback;
callbackStruct.inputProcRefCon = self;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_SetRenderCallback, 
                              kAudioUnitScope_Global, 
                              kOutputBus,
                              &callbackStruct, 
                              sizeof(callbackStruct));
checkStatus(status);
 
// Disable buffer allocation for the recorder (optional - do this if we want to pass in our own)
flag = 0;
status = AudioUnitSetProperty(audioUnit, 
                              kAudioUnitProperty_ShouldAllocateBuffer,
                              kAudioUnitScope_Output, 
                              kInputBus,
                              &flag, 
                              sizeof(flag));
 
// TODO: Allocate our own buffers if we want
 
// Initialise
status = AudioUnitInitialize(audioUnit);
checkStatus(status);

Then, when you’re ready to start:

1
2

OSStatus status = AudioOutputUnitStart(audioUnit);
checkStatus(status);

And to stop:

1
2

OSStatus status = AudioOutputUnitStop(audioUnit);
checkStatus(status);

Then, when we’re finished:

1

AudioComponentInstanceDispose(audioUnit);

And now for our callbacks.

Recording

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

static OSStatus recordingCallback(void *inRefCon, 
                                  AudioUnitRenderActionFlags *ioActionFlags, 
                                  const AudioTimeStamp *inTimeStamp, 
                                  UInt32 inBusNumber, 
                                  UInt32 inNumberFrames, 
                                  AudioBufferList *ioData) {
 
    // TODO: Use inRefCon to access our interface object to do stuff
    // Then, use inNumberFrames to figure out how much data is available, and make
    // that much space available in buffers in an AudioBufferList.
 
    AudioBufferList *bufferList; // <- Fill this up with buffers (you will want to malloc it, as it's a dynamic-length list)
 
    // Then:
    // Obtain recorded samples
 
    OSStatus status;
 
    status = AudioUnitRender([audioInterface audioUnit], 
                             ioActionFlags, 
                             inTimeStamp, 
                             inBusNumber, 
                             inNumberFrames, 
                             bufferList);
    checkStatus(status);
 
    // Now, we have the samples we just read sitting in buffers in bufferList
    DoStuffWithTheRecordedAudio(bufferList);
    return noErr;
}

Playback

1
2
3
4
5
6
7
8
9
10
11

static OSStatus playbackCallback(void *inRefCon, 
                                  AudioUnitRenderActionFlags *ioActionFlags, 
                                  const AudioTimeStamp *inTimeStamp, 
                                  UInt32 inBusNumber, 
                                  UInt32 inNumberFrames, 
                                  AudioBufferList *ioData) {    
    // Notes: ioData contains buffers (may be more than one!)
    // Fill them up as much as you can. Remember to set the size value in each buffer to match how
    // much data is in the buffer.
    return noErr;
}

Finally, rejoice with me in this discovery ;)

Resources that helped

No thanks at all to Apple for their lack of accessible documentation on this topic – They really have a long way to go here! Also boo to them with their lack of search engine, and refusal to open up their docs to Google. It’s a jungle out there!

Update: You can adjust the latency of RemoteIO (and, in fact, any other audio framework) by setting the kAudioSessionProperty_PreferredHardwareIOBufferDuration property:

float aBufferLength = 0.005; // In seconds
AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration, 
                        sizeof(aBufferLength), &aBufferLength);

This adjusts the length of buffers that’re passed to you – if buffer length was originally, say, 1024 samples, then halving the number of samples halves the amount of time taken to process them.

Another Update: In the comments below, Florian Bomers pointed out that I was incorrectly using the AudioUnitUninitialize to clean up the Audio Unit. This is incorrect, and should in fact be AudioComponentInstanceDispose. Further [discussion here](http://stackoverflow.com/questions/12688883/ios-5-6-low-volume-after-first-usage-of-coreaudio). Cheers Florian!

One Comment

  1. Vincent
    March 7, 2013 at 8:43 am

    Hi Michael, It’s great to visit your site. Regarding the audio unit development. Is it possible put a typeEffect audio node with subTypeReverb2 between the rioInput and rioOutput, which to make the play through sound with the Reverb?

    rioInput->Reverb2->rioOutput

    Best

    Vincent

  2. ksam
    March 20, 2013 at 6:49 am

    Hi Michael,

    I implemented audio capturing and playback functions using audio unit for VOIP application. When I am making my application go into the background it is not capturing or playing the data, while coming into the foreground it is displaying the message ” AURemoteIO::ChangeHardwareFormats: error -10875″.

    Then I did changes to play and record the audio when my app is going into the background mode, here only playback is happening and capture call back function is getting stopped automatically. I observed that when the application is in foreground only (not coming from background, after launching itself) it is displaying the same above message.

    I am capturing and playing tha data at 16K sampling rate with mono channel. Any input on these problem will greatly helpful.

    Waiting for your reply,

    Thanking you, ksam.

  3. Maarten
    April 11, 2013 at 2:57 pm

    Hi Michael,

    Are you aware of any way to captururing audio output of other applications? I other words, can I capture audio playing in te background so I can for example visualize it?

    Regards, Maarten.

  5. Titus Abraham
    May 15, 2013 at 6:11 am

    Hi,

    I noticed that When i record from MIC and give it to AU to play on speaker, the volume i hear is not that loud. Any means on how to set it high. I have noticed that the current hard ware volume is .5 (by getting property value for kAudioSessionProperty_CurrentHardwareOutputVolume).

    Any suggestions?

    Thanks, Titus

    1. iBanzai
      May 16, 2013 at 7:38 pm

      I am also looking for suggestions about this.

  6. William Power
    May 30, 2013 at 2:13 am

    I have a RemoteIO AudioUnit set up for play-through. No graph. Only one callback. (Similar to the setup in the book “Learning Core Audio”, chapter 10)

    This is great when there are headphones attached, but there’s tremendous feedback when there are no headphones attached. I’ve learned that I cannot simply disable the playback using the AudioUnitSetProperty call while recording is taking place, and I can’t seem to find the proper invocation to get recording going WITHOUT play-through.

    Can you provide any feedback / guidance?

    Thanks

    1. Brennon Bortz
      May 30, 2013 at 3:50 am

      Once you’ve done whatever you need to do with your samples (persist them to a file, etc.), just zero them out. Boom–no more playback.

      1. William Power
        May 31, 2013 at 4:43 am

        Yes, absolutely. I’m using extended audio file services to write to file async. AFTER that call, zeroing out ioData totally works. Thanks!

    2. Brennon Bortz
      May 30, 2013 at 3:53 am

      Or, just use the kAudioSessionCategory_RecordAudio session category.

      1. William Power
        May 31, 2013 at 4:42 am

        Tried it. I’m using the render callback to save to file using extended audio file services’ write async. The category session stuff on a pass-through remote io setup (without using an augraph) causes nothing to be recorded.

        1. Brennon Bortz
          June 3, 2013 at 3:22 am

          So, everything working now, or is something still funky?

          1. William Power
            June 3, 2013 at 11:59 pm

            The Record-only audio session wasn’t working, but zeroing out the buffer after firing ExtAudioFileWriteAsync did.

          2. Puneet Sharma
            June 6, 2013 at 5:14 am

            Hi

            I am facing a problem related to Audio recording. What i want to achieve is that suppose an Audio track is being played and at the same time recording is done, the playing track sound should not be recorded. But presently i am not able to achieve it. Always when i play and record simultaneous the played track is also merged in the recorded voice. Please help in this, it has been more than a week trying to sort this.

            Thanks, Puneet

  7. Puneet Sharma
    June 5, 2013 at 12:39 pm

    Hi

    I am facing a problem related to Audio recording. What i want to achieve is that suppose an Audio track is being played and at the same time recording is done, the playing track sound should not be recorded. But presently i am not able to achieve it. Always when i play and record simultaneous the played track is also merged in the recorded voice. Please help in this, it has been more than a week trying to sort this.

    Thanks, Puneet

  8. realloxer
    July 3, 2013 at 4:27 pm

    Hi, how can i set the number of Buffers? I have a VOIP app,I want to play continuously from socket bytes, can anyone help me?

  9. Lucas
    August 5, 2013 at 9:44 pm

    I copied and pasted ‘OSStatus status’ and ‘AudioComponentInstance audioUnit’ into my header file; put the rest of the initialization code in a method called ‘testMathod’ in my implementation file; and I’ve put ‘static OSStatus playbackCallback’ and ‘static OSStatus recordingCallback’ in my implementation file, but not inside a (objective -C) method. I want to be able to save the recording file to a URL I create and call it later to be played by an AVAudioPlayer in another method. I am currently getting the following failures: ‘audioFormat’ undeclared, ‘recordingCallback’ undeclared, ‘playbackCallback’ undeclared, ‘audioInterface’ undeclared, and nested functions are disabled, use -fnested-functions to reenable. <— those are the failures I am getting. The last thing I need to know is how to specify the URL to have the recording file saved to it.

  10. ksam
    August 8, 2013 at 6:38 am

    Hi all,

    I am trying to capture and play the audio simultaneously in iOS device using Remote IO audio unit, and it is working fine. What I am facing is, in iPhone 5 device when I am trying to capture the audio the captured audio is getting clipped and it is not smooth. How to over come this.

    When I am using the Voice processing audio unit then both the capture and play is happening but the captured data is not coming fine. It is looking like half duplex and VPIO is producing the zero sampled signals when I am using Voice chat to cancel the echo. So how to make use of inbuilt AEC effectively.

    Waiting for suggestions. Thank you in advance. ksam.

    1. Puneet Sharma
      August 8, 2013 at 7:44 am

      Hi Kasam

      I also faced the same issue and after weeks of googling i came to know that, Voice processing is use full in case of VOIP as in that case the Voice latency is considerable. But the same approach is not useful if you are developing some Musical app or when the Output and Receiver both are same. So while recording you need to pause the playback of other sound. Hope this helps.

      Puneet

      1. ksam
        August 8, 2013 at 8:07 am

        Punith,

        Thank you for your feedback. I am not concentrating on musical application, I want to use for VOIP in which low latency matters most. I am creating the separate Audio Units for both capturing and playback using kAudioVoiceProcessing unit. Here the captured audio is coming like half duplex one where as with RemoteIO it is coming fine. How to improve the captured audio quality when we are using VPIO.

  11. rahul parashar
    October 11, 2013 at 1:07 pm

    Hi Sir,

    i want to implement the functionality in my app. Like i have one bluetooth handsfree and one iPad device, now when i speak from bluetooth handsfree speaker my voice will transfer to iPad device via bluetooth connection establish between two ,and voice must come out through iPad speakers. Please suggest appropriate answer.

    Thanks

  12. Pingback: OpenAl, Core Audio, ddcli | Tercer día en el Big Nerd Ranch | Cocoa Mental

Comments are closed.