I’ve had nasty old time trying to get some audio stuff going on the iPhone, no thanks to Apple’s lack of documentation. If you’re an iPhone developer interested in getting RemoteIO/IO Remote/whatever it’s called working on the iPhone… Do I have good news for you. Read on.
Wanna skip the Core Audio learning curve and start writing code straight away? Check out my new project:
Update: Thanks to Joel Reymont, we now have an explanation for the “CrashIfClientProvidedBogusAudioBufferList” iPhone simulator bug: The simulator doesn’t like mono audio. Thanks, Joel!
Update: Happily, Apple have now created some excellent documentation on Remote IO, with some good sample projects. I recommend using that as a resource, now that it’s there, as that will continue to be updated.
Update: Tom Zicarelli has created a very extensive sample app that demonstrates the use of AUGraph, with all sorts of goodies.
So, we need to obtain an instance of the RemoteIO audio unit, configure it, and hook it up to a recording callback, which is used to notify you that there is data ready to be grabbed, and where you pull the data from the audio unit.
Overview
- Identify the audio component (kAudioUnitType_Output/ kAudioUnitSubType_RemoteIO/ kAudioUnitManufacturerApple)
- Use AudioComponentFindNext(NULL, &descriptionOfAudioComponent) to obtain the AudioComponent, which is like the factory with which you obtain the audio unit
- Use AudioComponentInstanceNew(ourComponent, &audioUnit) to make an instance of the audio unit
- Enable IO for recording and possibly playback with AudioUnitSetProperty
- Describe the audio format in an AudioStreamBasicDescription structure, and apply the format using AudioUnitSetProperty
- Provide a callback for recording, and possibly playback, again using AudioUnitSetProperty
- Allocate some buffers
- Initialise the audio unit
- Start the audio unit
- Rejoice
Here’s my code: I’m using both recording and playback. Use what applies to you!
Initialisation
Initialisation looks like this. We have a member variable of type AudioComponentInstance which will contain our audio unit.
The audio format described below uses SInt16 for samples (i.e. signed, 16 bits per sample)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | #define kOutputBus 0 #define kInputBus 1 // ... OSStatus status; AudioComponentInstance audioUnit; // Describe audio component AudioComponentDescription desc; desc.componentType = kAudioUnitType_Output; desc.componentSubType = kAudioUnitSubType_RemoteIO; desc.componentFlags = 0; desc.componentFlagsMask = 0; desc.componentManufacturer = kAudioUnitManufacturer_Apple; // Get component AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc); // Get audio units status = AudioComponentInstanceNew(inputComponent, &audioUnit); checkStatus(status); // Enable IO for recording UInt32 flag = 1; status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, kInputBus, &flag, sizeof(flag)); checkStatus(status); // Enable IO for playback status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Output, kOutputBus, &flag, sizeof(flag)); checkStatus(status); // Describe format audioFormat.mSampleRate = 44100.00; audioFormat.mFormatID = kAudioFormatLinearPCM; audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked; audioFormat.mFramesPerPacket = 1; audioFormat.mChannelsPerFrame = 1; audioFormat.mBitsPerChannel = 16; audioFormat.mBytesPerPacket = 2; audioFormat.mBytesPerFrame = 2; // Apply format status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, kInputBus, &audioFormat, sizeof(audioFormat)); checkStatus(status); status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, kOutputBus, &audioFormat, sizeof(audioFormat)); checkStatus(status); // Set input callback AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = recordingCallback; callbackStruct.inputProcRefCon = self; status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Global, kInputBus, &callbackStruct, sizeof(callbackStruct)); checkStatus(status); // Set output callback callbackStruct.inputProc = playbackCallback; callbackStruct.inputProcRefCon = self; status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Global, kOutputBus, &callbackStruct, sizeof(callbackStruct)); checkStatus(status); // Disable buffer allocation for the recorder (optional - do this if we want to pass in our own) flag = 0; status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_ShouldAllocateBuffer, kAudioUnitScope_Output, kInputBus, &flag, sizeof(flag)); // TODO: Allocate our own buffers if we want // Initialise status = AudioUnitInitialize(audioUnit); checkStatus(status); |
Then, when you’re ready to start:
1 2 | OSStatus status = AudioOutputUnitStart(audioUnit); checkStatus(status); |
And to stop:
1 2 | OSStatus status = AudioOutputUnitStop(audioUnit); checkStatus(status); |
Then, when we’re finished:
1 | AudioComponentInstanceDispose(audioUnit); |
And now for our callbacks.
Recording
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | static OSStatus recordingCallback(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData) { // TODO: Use inRefCon to access our interface object to do stuff // Then, use inNumberFrames to figure out how much data is available, and make // that much space available in buffers in an AudioBufferList. AudioBufferList *bufferList; // <- Fill this up with buffers (you will want to malloc it, as it's a dynamic-length list) // Then: // Obtain recorded samples OSStatus status; status = AudioUnitRender([audioInterface audioUnit], ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, bufferList); checkStatus(status); // Now, we have the samples we just read sitting in buffers in bufferList DoStuffWithTheRecordedAudio(bufferList); return noErr; } |
Playback
1 2 3 4 5 6 7 8 9 10 11 | static OSStatus playbackCallback(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData) { // Notes: ioData contains buffers (may be more than one!) // Fill them up as much as you can. Remember to set the size value in each buffer to match how // much data is in the buffer. return noErr; } |
Finally, rejoice with me in this discovery ;)
Resources that helped
- http://pastie.org/pastes/219616
- http://developer.apple.com/samplecode/CAPlayThrough/listing8.html
- http://listas.apesol.org/pipermail/svn-libsdl.org/2008-July/000797.html
No thanks at all to Apple for their lack of accessible documentation on this topic – They really have a long way to go here! Also boo to them with their lack of search engine, and refusal to open up their docs to Google. It’s a jungle out there!
Update: You can adjust the latency of RemoteIO (and, in fact, any other audio framework) by setting the kAudioSessionProperty_PreferredHardwareIOBufferDuration
property:
float aBufferLength = 0.005; // In seconds AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration, sizeof(aBufferLength), &aBufferLength); |
This adjusts the length of buffers that’re passed to you – if buffer length was originally, say, 1024 samples, then halving the number of samples halves the amount of time taken to process them.
Another Update: In the comments below, Florian Bomers pointed out that I was incorrectly using the AudioUnitUninitialize
to clean up the Audio Unit. This is incorrect, and should in fact be AudioComponentInstanceDispose
. Further [discussion here](http://stackoverflow.com/questions/12688883/ios-5-6-low-volume-after-first-usage-of-coreaudio). Cheers Florian!
Hi Michael. First, thank you very much for sharing ur knowledge about the RemoteIO and ur experiences with loopy. It was very inspiring. At the moment I am developing an application that uses Audio Queue. Its a kind of step sequencer drum machine. The thing is that everything is working ok, the buffers are small enough to keep a good realtime response, I am doing the mixing by software so the timing is good aswell. Every two minutes I ran out of buffers at the audiocallback but what i can hear is a little "pop" that well, i will try to fix soon because I have a bigger problem :) The thing is that when I want to write something and the touch keyboard appears on screen I get a big delay that takes 2 seconds to recover. Really bad. Sound stops cos my Callback function processes all the buffers (because this keyboard thing is on the same thread I guess and blocks the buffer generation). I have been trying to put the Callback is another thread with no success using NSThread. Did u have the same problem ? Any ideas please ? Thank you !
Excuse my english :)
I'm afraid I have no idea what's happening there, Esteban – I just tried it with Loopy, to see if there was any delay when bringing up the keyboard, but there's nothing. Perhaps it's an Audio Queue thing?
I suggest taking this question to the developer forums (devforums.apple.com), if you haven't already
Best of luck!
This code was so appreciated, michael, you are the real deal.
I have coded a little playback example based on this code, which opens a wav file and uses it to fill a buffer, you can download it from here:
http://sites.google.com/site/iphoneappcoder/iphone-wav-file-playback
Very nice work, Aran – I've been meaning to put together a sample project that plays an audio file, very grateful that you have done so and are sharing it!
Thanks for the help with this great audio file playing code. I know this is a silly question, but is there an easy way to get this to simply play the audio file once instead of looping it?
Thats cool, the code only works on the device however. I'd like to get it going on the simulator. If anyone has any ideas that would be most appreciated.
Hi Sound experts, I am refering Speak here sample app to create sound recorder. Speak here stores audio in .caf file. But i want to keep audio in .m4a file. What changes i need to do? Can i directly record audio in .m4a file?
I'm afraid the iPhone SDK can't record to mp4 or other such complex compressed formats, as it's too CPU-intensive.
You may be able to incorporate another tool like the LAME encoder, but I don't know about the legalities.
It's damn right!
I hate Apple, they really did some awesome stuffs, but they forget to mark which is which.
I Damn HATE PublicUtility files. That is the real jungle!
Thank you so much for sharing this. Michael, you did a great thing!
Hey Michael, Thanks for sharing. I'm working on playing through the MultiChannelMixer. I'm actually doing it but I'm having some trouble and I know it's because I don't know the properties/parameters for the Multichannel mixer. Where did you find these for the remoteIO unit? For example, how did you know about kAudioOutputUnitProperty_EnableIO and kAudioUnitScope_Input in the following call? Where are these listed for each audio unit?
status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, kInputBus, &flag, sizeof(flag));
Hi RP; I found my references through fluked Google searches and nothing else (they're linked at the bottom of the post). As I rant at the end, there's very little documentation available for much of this stuff, so often we're on our own. Best of luck
Hey Michael, thanks a lot for sharing your great discovery. I downloaded Aran's sample project, introduced a few posts earlier and tried it. It repeats the sound after the "start" method is called(which calls AudioOutputUnitStart(audioUnit) ). I don't have a good understanding of digital audio(buffers,etc.). Could you please address me on how to prevent this from happening and play the sound only once? Thanks for your time.
Has anyone tested this code out on beta 3.0 firmware? I have a report of a crash. Probably a bit early to be concerned though.
Thread 3 Crashed: 0 AudioToolbox 0x31e7a3e0 SampleRateConverter::RenderOutput(CABufferList, unsigned long, unsigned long&, AudioStreamPacketDescription) + 8 1 AudioToolbox 0x31e79f34 BufferedAudioConverter::FillBuffer(unsigned long&, AudioBufferList&, AudioStreamPacketDescription) + 252 2 AudioToolbox 0x31e7a5c8 AudioConverterChain::RenderOutput(CABufferList, unsigned long, unsigned long&, AudioStreamPacketDescription) + 136 3 AudioToolbox 0x31e79f34 BufferedAudioConverter::FillBuffer(unsigned long&, AudioBufferList&, AudioStreamPacketDescription) + 252 4 AudioToolbox 0x31e79148 AudioConverterFillComplexBuffer + 456 5 AudioToolbox 0x31f7e234 AUIOHelper::NotifyInputAvailable(AudioTimeStamp const&, unsigned long, AudioBufferList const&) + 316 6 AudioToolbox 0x31f70edc AURemoteIO::PerformIO(int, unsigned int, unsigned int, AQTimeStamp const&, AQTimeStamp const&) + 432 7 AudioToolbox 0x31f71204 AURIOCallbackReceiver_PerformIOSync + 40 8 AudioToolbox 0x31f66908 _XPerformIOSync + 232 9 AudioToolbox 0x31f2cc94 mshMIGPerform + 224 10 AudioToolbox 0x31f2cef0 MSHMIGDispatchMessage + 44 11 AudioToolbox 0x31f6eeb4 AURemoteIO::IOThread::Run() + 140 12 AudioToolbox 0x31f73628 AURemoteIO::IOThread::Entry(void) + 4 13 AudioToolbox 0x31ecef40 CAPThread::Entry(CAPThread) + 208 14 libSystem.B.dylib 0x31d7a66c _pthread_body + 20
re: Justin Brady
I was able to get Arans code working quite easily on 3.0 firmware, at least in the simulator. I have yet to try it on the actual device.
re: Aran thanks so much for this sample code, it is really helping me out quite a bit! on wards and upwards!
re: folks with questions about digital audio / dsp, a great resource is http://musicdsp.org/ which is not exactly a primer, but has plenty of helpful functions for interpolation / synthesis / filters / etc.
Hi Michael, thank you for sharing this. It's very helpful.
I have coded a playback callback function, which synthesize/render audio samples in real-time. But it doesn't work on the device because the rendering process is CPU-intensive and I can't render enough audio samples for ioData->mBuffers[i].mData. So, my question is "Do I have to fill ALL the buffers?" Can I fill the ioData->mBuffers[i].mData and set ioData->mBuffers[i].mDataByteSize with only what I have rendered so far?
Thank you.
A question, i would like to get a handle on the thread that has the audio callback that fills the buffers. anyone know of a way to query the audio unit to find out?
I know i could find out the thread once the audio callback is called, but is there a way to ask the audio unit itself.
Thanks a lot, Michael. Your example was very informative for me. Good luck with Loopy.
I still can't get anything working in the iPhone simulator 2.2. I always get the CrashIfClientProvidedBogusAudioBufferList error despite the fact I have set mChannelsPerFrame = 2.
Any ideas would be greatly appreciated.
Thanks
Hi Toby,
I'm afraid this is a bug in the simulator, not your code – it's not well understood under what circumstances the bug appears, but Apple have been made aware of the problem. Have you tried the 3.0 SDK?
Thanks for replying,
Haven't tried it in 3.0 as I haven't paid to join the developer programme yet. Was hoping to get an app at near completion before having to stump up the $99 fee. Am I right in thinking I can't test in on device until I have paid up either?
Ah, fair enough. Yep, that's right, gotta pay to test on device. Best of luck!
Forgot to mention, it is on AudioOutputUnitStart that the error occurs
Hey Michael, I've gotten RemoteIO working great. Thanks for the great help here. Do you have any idea how to allow the iPod music currently playing to not stop once the app is opened. I know how to do this with Audio Sessions, but it doesn't seem to work with RemoteIO. Let me know if you have any tips. Thanks!
Hi RP,
From my understanding, it's just the same! RemoteIO still operates underneath the same envelope, which is overseen by the audio session stuff.
All the best
I used the audioqueue for the midi player (http://www.interactiveblueprints.com) I'll look into the remoteio. Thx for the info! G
Hi, I need some help, on iPhone 3.0 GM seed but also seen on 3.0b5, I am not able to re-route the audio to the speaker while I am recording. The following code is starting the audio engine (setup as Play&Record in another function) and re-routing audio from headset to speaker but it does not work anymore on 3.0b5 or 3.0 GM. -(OSStatus)start{ // Routing default audio AudioSessionInitialize(NULL, NULL, NULL, NULL); // Set the audio session category for simultaneous play and record UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; OSStatus status = AudioSessionSetProperty (kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); float aBufferLength = 1024.0 / 48000.0; // In seconds AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration, sizeof(aBufferLength), &aBufferLength); status = AudioSessionSetActive(true); status = AudioOutputUnitStart(audioUnit); UInt32 newAudioRoute = kAudioSessionOverrideAudioRoute_Speaker; status = AudioSessionSetProperty(kAudioSessionProperty_OverrideAudioRoute, sizeof(newAudioRoute), &newAudioRoute); return status; } Can someone help me on this? Frederic
iPhone OS 3.0 GM seed seems to have broken my RemoteIO program. Every other time I press play (begin the callback) the processor maxes out and I get stuttering and jittering in the audio. Weird thing is, it’s exactly every 2nd time I start and stop the player. The other times it works exactly right. No processor max out. Anyone else having problems with 3.0?
GM 3.0 seems to be completely rooted. I’ve got problems with Audio Queues, offline Audio Queues, and RemoteIO.
I’m talking about the simplest programs possible.
Didn’t know the NDA was still on. Is there somewhere we* can talk about this?
*We = those who have signed their rights away
Hi RP,
I don’t feel comfortable getting into specifics about my experiences with OS3.0b5 and OS3.0 GM Seed because of the non-disclosure agreement. But I will say that no, you are not going crazy. Some of the default behavior of audiosessions has changed with the new OS. Nothing insurmountable, but it is different.
Marcus, Thanks for the reply. I was able to solve some problems, but not all. I’d be interested in discussing this further once the NDA is lifted, which I presume will be soon since OS 3.0 is to be officially released today.
I can’t for the life of me get this working. Has anyone produced a working Xcode project from this?
rfistman, I think the NDA is off now. I believe anyone can download the 3.0 SDK from Apple without signing anything. I was able to get much better performance by compiling with this SDK for 2.2.1 rather than an earlier OS.
Hi All, I am desperately seeking help with getting this working correctly. Right now all I care about is playback (recording to come later though).
I think the problem might be related to the bit rate of the WAVs I am using. The bit rate (as iTunes reports) is 705kbps sample size 16 bits and mono. But I don’t see how this can be configured anywhere.
What happens is the WAV is played, followed by an equal amount of silence. My current incantation of the callback looks like this (and is based off of Aran Mulholland’s sample): [code] static OSStatus playbackCallback(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData) { LoopPlayer *loopPlayer = (LoopPlayer *)inRefCon;
}
//TODO return a "real" return code instead of always okay :) return noErr;
}
[/code]
also the format is set up with this: [code] audioFormat.mSampleRate = 44100.00; audioFormat.mFormatID = kAudioFormatLinearPCM; audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked; audioFormat.mFramesPerPacket = 1; audioFormat.mChannelsPerFrame = 1; audioFormat.mBitsPerChannel = 16; audioFormat.mBytesPerPacket = 2; audioFormat.mBytesPerFrame = 2; [/code]
Any one have any ideas?
Hi, Michael. Thank you so much for your great work! I just started learning I/O remote by reading document from apple, and I had no idea what to use it. this is much easier to understand than Apple’s explanation. At least The chord above is working for me, but I still having problem with very basic step. (1) I do not have much idea how I can open mp3 file instead of wav. (2) I just want to call(or open) the audio file only once for each time I set the OS status to start play and do not need call back right now, but I dont have much idea how to disable the callback function. Sorry for asking you silly question. I am very appreciate your help. Thanks!
This thread may be long in the tooth but it as well as Aran’s sample code were IMMENSELY helpful- thanks!
And now the obligatory question (sorry)- I’m trying to get simultaneous playback and recording going at the same time, and I’m still shaky on my grasp of how to get at the i/o data in one place. Can this be done in one callback? Or are the two callbacks run simultaneously?
Thanks once again; you guys should get merit badges from apple!
Hi Benett,
I’ve been trying to do the same and finally I succeeded. You do not need two callbacks (it will make your code toggle between them and it will slow things down). You need only one callback. In the callback the ioData points to the output buffer (for the speakers). You just need to place this pointer in the AudioUnitRender() as input parameter and connect this function the the output of bus1. This function will automatically copy everything into ioData buffer. The ioData buffer is the output buffer (for the speakers). This way only one buffer is used for pass-through. You can also have two buffers if you want to record something and play a different thing. Just define another buffer like ioData (let’s call it inData) in the callback and fill this buffer with the input samples using AudioUnitRender(). That way you will have one input buffer inData with the input samples from the mic and one output buffer ioData for the speakers.
I hope this helps, Regards, StefanS
Hi Michael, Thanks for your great Post! Can you possibly tell me if i can Connect an AudioUnit to the Output of the Mediaplayer So That i can add effects to †he outputted audio of the ipod app? Any help would be appreciated
Thanks Andy
i’m going to ask this question here and on the dev forums:
how does one go about scheduling future events? ie. for a basic step sequencer
information on this is sparse or perhaps i’m searching for the wrong keywords.
any help appreciated.
Hi Michael,
Is the trimming of audio file is possible by iPhone sdk?
Thanks a lot!
It helped me a lot.
Hi….nice work……. Im a new developer, I want to record audio from audio unit to a file….. I did this with audio queue…. but I don’t know how to do with audio unit…Is it possible to change volume of the audio data which is to be recorded to a file… Thank you
hi…. am able to record to a file. One thing i want to know is can we change the volume of the audio data which is to be recorded to a file
Can you share (code) how you recorded audio unit to file? I’m trying to do the same, but am very confused.