I’ve had nasty old time trying to get some audio stuff going on the iPhone, no thanks to Apple’s lack of documentation. If you’re an iPhone developer interested in getting RemoteIO/IO Remote/whatever it’s called working on the iPhone… Do I have good news for you. Read on.
Wanna skip the Core Audio learning curve and start writing code straight away? Check out my new project:
Update: Thanks to Joel Reymont, we now have an explanation for the “CrashIfClientProvidedBogusAudioBufferList” iPhone simulator bug: The simulator doesn’t like mono audio. Thanks, Joel!
Update: Happily, Apple have now created some excellent documentation on Remote IO, with some good sample projects. I recommend using that as a resource, now that it’s there, as that will continue to be updated.
Update: Tom Zicarelli has created a very extensive sample app that demonstrates the use of AUGraph, with all sorts of goodies.
So, we need to obtain an instance of the RemoteIO audio unit, configure it, and hook it up to a recording callback, which is used to notify you that there is data ready to be grabbed, and where you pull the data from the audio unit.
Overview
- Identify the audio component (kAudioUnitType_Output/ kAudioUnitSubType_RemoteIO/ kAudioUnitManufacturerApple)
- Use AudioComponentFindNext(NULL, &descriptionOfAudioComponent) to obtain the AudioComponent, which is like the factory with which you obtain the audio unit
- Use AudioComponentInstanceNew(ourComponent, &audioUnit) to make an instance of the audio unit
- Enable IO for recording and possibly playback with AudioUnitSetProperty
- Describe the audio format in an AudioStreamBasicDescription structure, and apply the format using AudioUnitSetProperty
- Provide a callback for recording, and possibly playback, again using AudioUnitSetProperty
- Allocate some buffers
- Initialise the audio unit
- Start the audio unit
- Rejoice
Here’s my code: I’m using both recording and playback. Use what applies to you!
Initialisation
Initialisation looks like this. We have a member variable of type AudioComponentInstance which will contain our audio unit.
The audio format described below uses SInt16 for samples (i.e. signed, 16 bits per sample)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | #define kOutputBus 0 #define kInputBus 1 // ... OSStatus status; AudioComponentInstance audioUnit; // Describe audio component AudioComponentDescription desc; desc.componentType = kAudioUnitType_Output; desc.componentSubType = kAudioUnitSubType_RemoteIO; desc.componentFlags = 0; desc.componentFlagsMask = 0; desc.componentManufacturer = kAudioUnitManufacturer_Apple; // Get component AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc); // Get audio units status = AudioComponentInstanceNew(inputComponent, &audioUnit); checkStatus(status); // Enable IO for recording UInt32 flag = 1; status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, kInputBus, &flag, sizeof(flag)); checkStatus(status); // Enable IO for playback status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Output, kOutputBus, &flag, sizeof(flag)); checkStatus(status); // Describe format audioFormat.mSampleRate = 44100.00; audioFormat.mFormatID = kAudioFormatLinearPCM; audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked; audioFormat.mFramesPerPacket = 1; audioFormat.mChannelsPerFrame = 1; audioFormat.mBitsPerChannel = 16; audioFormat.mBytesPerPacket = 2; audioFormat.mBytesPerFrame = 2; // Apply format status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, kInputBus, &audioFormat, sizeof(audioFormat)); checkStatus(status); status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, kOutputBus, &audioFormat, sizeof(audioFormat)); checkStatus(status); // Set input callback AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = recordingCallback; callbackStruct.inputProcRefCon = self; status = AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Global, kInputBus, &callbackStruct, sizeof(callbackStruct)); checkStatus(status); // Set output callback callbackStruct.inputProc = playbackCallback; callbackStruct.inputProcRefCon = self; status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Global, kOutputBus, &callbackStruct, sizeof(callbackStruct)); checkStatus(status); // Disable buffer allocation for the recorder (optional - do this if we want to pass in our own) flag = 0; status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_ShouldAllocateBuffer, kAudioUnitScope_Output, kInputBus, &flag, sizeof(flag)); // TODO: Allocate our own buffers if we want // Initialise status = AudioUnitInitialize(audioUnit); checkStatus(status); |
Then, when you’re ready to start:
1 2 | OSStatus status = AudioOutputUnitStart(audioUnit); checkStatus(status); |
And to stop:
1 2 | OSStatus status = AudioOutputUnitStop(audioUnit); checkStatus(status); |
Then, when we’re finished:
1 | AudioComponentInstanceDispose(audioUnit); |
And now for our callbacks.
Recording
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | static OSStatus recordingCallback(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData) { // TODO: Use inRefCon to access our interface object to do stuff // Then, use inNumberFrames to figure out how much data is available, and make // that much space available in buffers in an AudioBufferList. AudioBufferList *bufferList; // <- Fill this up with buffers (you will want to malloc it, as it's a dynamic-length list) // Then: // Obtain recorded samples OSStatus status; status = AudioUnitRender([audioInterface audioUnit], ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, bufferList); checkStatus(status); // Now, we have the samples we just read sitting in buffers in bufferList DoStuffWithTheRecordedAudio(bufferList); return noErr; } |
Playback
1 2 3 4 5 6 7 8 9 10 11 | static OSStatus playbackCallback(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData) { // Notes: ioData contains buffers (may be more than one!) // Fill them up as much as you can. Remember to set the size value in each buffer to match how // much data is in the buffer. return noErr; } |
Finally, rejoice with me in this discovery ;)
Resources that helped
- http://pastie.org/pastes/219616
- http://developer.apple.com/samplecode/CAPlayThrough/listing8.html
- http://listas.apesol.org/pipermail/svn-libsdl.org/2008-July/000797.html
No thanks at all to Apple for their lack of accessible documentation on this topic – They really have a long way to go here! Also boo to them with their lack of search engine, and refusal to open up their docs to Google. It’s a jungle out there!
Update: You can adjust the latency of RemoteIO (and, in fact, any other audio framework) by setting the kAudioSessionProperty_PreferredHardwareIOBufferDuration
property:
float aBufferLength = 0.005; // In seconds AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration, sizeof(aBufferLength), &aBufferLength); |
This adjusts the length of buffers that’re passed to you – if buffer length was originally, say, 1024 samples, then halving the number of samples halves the amount of time taken to process them.
Another Update: In the comments below, Florian Bomers pointed out that I was incorrectly using the AudioUnitUninitialize
to clean up the Audio Unit. This is incorrect, and should in fact be AudioComponentInstanceDispose
. Further [discussion here](http://stackoverflow.com/questions/12688883/ios-5-6-low-volume-after-first-usage-of-coreaudio). Cheers Florian!
Aran, your code was immensely helpful, I don’t know how else to contact you. I just wanted to say thanks.
Nice work on putting together this example. Yours and Aran’s(http://sites.google.com/site/iphonecoreaudiodevelopment/remoteio-playback) hard work have helped greatly in getting a jumpstart with core audio on the that popular mobile device…
Hi,
Code incredibly helpful indeed, both yours and Aran’s, so many thanks for these. Quick question: I’m trying to lower the volume with a slider. Works fine when I go up (multiplying the Uint32 of the signal by integer) BUT fails completely (sound distorted, yet recognizable) when I divide. So obviously I’m doing it wrong, but I don’t know why. I’ve tried shifting instead of dividing, same thing. Clues welcome :)
Hi Guillaume, Hmm, try casting to float/double first.. e.g. sample = (float)sample * volume;
It may be doing integer division, which would result in distortion. Good luck!
You & Aran were right, it is a cast problem. Casting to signed int did the trick. Heartfelt thanks for the advice, and for your outstanding contribution!
Thanks for all the information shared here, it’s been really useful.
I know the post is quite old, but I’d be really grateful if you could expand on the cast problem and what you did to fix it, if possible?
I’m having a very similar problem – everything working fine but I can’t work with the buffer data in my playback callback without introducing distortion/’static’.
When attempting to control volume, If I multiply the current UInt32 of the frameBuffer by 1 or above, it responds as I expect (I think!), but dividing (or trying to lower the volume) mangles things up. I’ve tried every type of cast/manipulation I can think of, but (hopefully) not the one you used successfully…
For information, I’m using an array of stereo PCM files and mixing in or supplying zero, as required, on the fly in the callback. At the moment I’m simply adding the current frame UInt32 for each PCM in the mix, so shared frequencies sound bad. I’m working toward some limiting/clipping reduction (any ideas?) but until I can reliably manipulate volume in the callback, I obviously can’t get much further.
I’ve spent a couple of days on this distortion problem alone, so any ideas or pointers on how to get past it would be very helpful. Thanks for your time.
I finally fixed it.
For anyone coming across this in the future, the solution for me was casting and, very important in my case: unpacking 32 bit frames to 2 x 16 frames, perform manipulations, then recombine for filling the buffer. No amount of casting combinations was going to work until I had that ‘DOH!’ moment of realisation re: unpacking the stereo :)
Thanks again guys.
Hi Michael,
I have got an app working using your example above having the audio from the mic going straight through to playback callback. However I am having problems getting the audio to come out of the speaker rather than the earpiece.
I have tried setting the route to be kAudioSessionOverrideAudioRoute_Speaker but it doesn’t seem to do anything. I was reading in your blog about Loopy that you had a problem with feedback because of audio coming out of the speaker, but how did you do it?
Thanks
Hey Toby can you explain how you got audio passing from mic to headphones? I am trying to do the same thing, but I am not sure what to put in the recording callback to do this. I assume it involves using the function AudioUnitRender, then copying over the buffer? Any suggestions / sample code? Thanks
Hi Chris – as far as I know, as long as the headphones are plugged in, that’s the route the audio should take, unless you’ve manually overridden the path (or perhaps if you’re using a session category that has that behaviour?).
My problem was as Toby expressed above (and I totally, inexcusably missed his question at the time!), getting the audio to come out of the speaker rather than the receiver; What I’m doing is listening for changes in the audio route, which happens when headphones are plugged in, etc:
// Listen for changes in audio route AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange, routeChangeCallback, self);
Then in the callback, checking the route, and overriding it if necessary:
... if ( [(NSString*)route isEqualToString:@"ReceiverAndMicrophone"] ) { ... // Re-route audio to the speaker (not the receiver) UInt32 newRoute = kAudioSessionOverrideAudioRoute_Speaker; status = AudioSessionSetProperty(kAudioSessionProperty_OverrideAudioRoute, sizeof(route), &newRoute); checkStatus(status);
Hi Chris,
I can’t remember off the top of my head now I did it now, so long ago :)
I can take a look when I get home, but I believe I used this example as a basis which is slightly different from Michael’s example.
http://pastie.org/pastes/445149
It’s useful. But I still do not understand how to copy input samples to output. Is there a complete example of a playthru with, in the middle, a callback that perform some sort of modification (e.g. a volume)?
Thank you for this simple walkthrough! I’m going to use this as reference while doing some experimentation with Pandora.
Pingback: iPhone Core Audio Part 3 – Audio Callback « Tim Bolstad
Your name is Mike Tyson… That’s awesome!
Kinda like Michael Bolton from Office Space?
“…I told those fudge-packers I liked Michael Bolton’s music…“
Yeah, I can relate….
Thanks for posting this wonderful example.
I am having trouble getting the example to work. Is there a place I can download your code?
Hey, great code. In recordingCallback how would you process the ioData to get the amplitude of each sample. I have everything else apart form this sorted and im so stumped. Any help would be appreciated. Thanks.
That would be very interesting to me too!! Thanks!
Great looking bit of code…
I have a question. Would something like this code allow me to record multiple samples to an audio file?
I have a musical instrument app and I am desperately trying to record the output to an audio file. I have been trying everything and I think Audio Units is the answer but Apples Docs are far from helpful.
Any advice would be greatly appreciated.
Thanks
Ben
Hi Ben – Sorry about the respond delay.
This code doesn’t actually include any file operations, but you would probably put something in the record callback. The Audio File services are what you’re after.
Pingback: Decibel metering from an iPhone audio unit | Politepix
Hi guys,
We have remoteio set up mostly from info in this article and working perfectly on the iPhone and iPad.
The problem we have is that on the iPod (1st gen tested) the record playback is never called. We get no errors from any of the setup calls.
Anyone any idea on this issue?
Thanks for any help
Andy
To Andy Capon, You do have the mic inputed in the iPod touch right? Because it has to mic of it’s own…
Hi UnrealZ,
Thanks for the reply.
We found the problem, the Gen 1 touch has not got a mic connection on the headphone socket while the gen 2 and 3 do.
The Gen 1 needs a mic input via the docking connector to get it to work.
Cheers
Andy
Hi Andy Capon,
In case you read this, and are the Andy Capon I am looking for, please contact me via samuelbuckleATgmxDOTnet
Cheers !
Hi, I am working with aurio touch sample code from apple. I want to use vibration but it doesn’t work. I think that remoteio setup makes vibration stop. I have tried several things, but it doesn’t work.
How can I make vibration using remoteio setup? thanks for reading.
I’m not entirely sure what it is, but have you tried some different audio session types?
How can seek to a particular time in the audio recording in the example you have provided.?
I figured out the answer to the question that others have been asking about how to make Aran’s code stop looping: just replace packetIndex = 0; with return -1; in the getNextPacket method.
It looks like Aran had already implemented the method to do as such when the file is out of packets (as you can see in the method comment from the header file)… I’m guessing the looping was an advanced feature he threw in there just to show how looping can be done.
Simple :-)
By the way Aran, I’ve been working with your code extension to Michael’s code for a few days now and hacking away to make it more robust and add some things like a currentTime property, duration property, etc… to make it a little more AVAudioPlayer’esque. Would you mind if I sent you the code to look at? I’m new to core audio as well but i’m a full-time iphone programmer in my day job and I’d like to try to utilize this code for some upcoming personal apps.
Michael… great job man! I’ve been digging through the Core Audio docs for a few weeks now and they are the shittiest i’ve seen. I finally understand how this all comes together with your post.
Ok.. nevermind about returning -1.. The playback buffer is still looking for samples… it just doesn’t freeze up when you stop.
On to my question… I’ve spent about 8+ hours now trying to get Aran’s code to stop at the end of the inMemoryFile (when I’ve noticed that i’ve exhausted all the packets of the file). I’ve tried just terminating the audioUnit using AudioOutputUnitStop(audiounit) but it seems to screw something up in the runloop and freeze up the app.
I’m assuming that in Aran’s code, the audio playback buffer continues until infinity and it’s only job is to collect more samples from the inMemoryAudioFile. If I try to place an AudioOutputUnitStop during the loop through the frames, it freezes. How do I kill the playbackCallback loop when the frames in the inMemoryAudioFile are exhausted?
Even more broad a question… is it possible to just tell the audioUnit from the beginning exactly how many frames I have to play? I like the the fact that looping is an option… but not being able to interrupt the audio playbackCallback when I’m done is quite annoying.
Thanks!
Actually, as far as I know, the general procedure when you don’t actually want to play anything, is to give the audio unit silence instead – fill up the buffer with a bunch of zeroes.
I would avoid doing any kind of management stuff (like starting/stopping) from within the callback. Rather, just provide silence in the callback. If you want to actually stop the audio unit (which, remember, will incur delays next time you start it, as the hardware startup time is quite long), do it from your main thread, not within the callback.
Is it possible to have the remoteio unit initialized and running and determine whether or not we want to record or playaback a file at runtime?
In other words, could I specify some way of accepting input (from the microphone) only when I’m in recording mode and reading the samples from a file only when I’m in playback mode?
Can I do this without stopping the audio unit?
Absolutely. This is what I do, too.
Just keep both the recording and playback units running all the time, and only save incoming samples when you actually want to record, and only provide non-zero audio samples when you actually want to play (provide silence – zeroes – otherwise)
So, i’ve been messing around with this for several weeks now and I’m starting to get a good objective-c class hierarchy together to wrap up the C stuff…
If I’m writing an app where the user has the ability to select the output format of the file they want to save AND also give them the ability to play files of multiple formats, I should only need to convert the file’s data into the PCM when I need to grab the sample(s) from the file/dump the samples to a file right?
How do I deal with the case where the file i’m reading has multiple channels but I only have my RemoteIO instance set up and running with 1 channel. Would I stop the audio unit, change the audioFormat struct and then restart the audio unit or can I hotswap the audio format while the audioUnit is initialized and running? \ Thanks!
Hi Corey,
That’s right, you only need to decode when you actually want to use the audio data. And as far as I know, you’ll need to re-create the audio unit with the new format settings when starting on a new file with a different format.
Michael,
Im still having trouble trying to alert the app when i’m done injecting audio data from a wave file into the audio unit’s output buffer. Basically i’m utilizing the same pattern from the InMemoryAudioFile and setting a STATE enumeration on the RemoteIO when i’m trying to trigger the playing of the audio data.. so the playbackCallback works as follows:
If state is kRecording, accept the rendered samples that were passed from the recordingCallback function… otherwise, if state is kPlaying, go ahead and request the next packet from the injected inMemoryAudioFile. Upon the completion of the packets from the inMemoryAudioFile, the playbackCallback just keeps asking for the next packet… I know that I can’t change the state in the inMemoryAudioFile and expect the RemoteIO loop to pick up on it, because it just calls the getNextPacket() function indefinitely. I can’t seem to find a good pattern for alerting the client controller that i’m done playing and being able to set the state on the remoteIO class so that it stops trying to seek the next packet from the inMemoryAudioFile.
I’ll change my pattern if I need to. Another way that I found works is by setting a isDonePlaying flag on the inMemoryAudioFile and just using the client controller’s timer (that’s updating the sound meters/waveform) to set it’s state to an idle mode- however the RemoteIO is STILL asking for more packets from the inMemoryAudioFile at this point… I can’t seem to find a natural way for the RemoteIO to just let the buffer die out and go back to it’s default state.
Any thoughts? I know there’s been a lot of other posts on this problem in this thread and I know a lot of people would love to have an answer to this. Maybe I should a different method of injecting the audio file packets into the playback buffer where it can just stop? I’m assuming this is a problem with unsynchronized threads somewhere in the chain.
Thanks man!
Thanks a bunch for your code, I can’t believe this is from 2008 and apple’s documentation is STILL not any more helpful.
I used your code to create a very simple application that takes the microphone audio data and reroutes it to the speakers.
It can be found here: http://ios-coreaudio-example.googlecode.com/files/Aruts.zip
Hi Michael,
Thank you very much for the tip to set kAudioSessionProperty_PreferredHardwareIOBufferDuration to adjust the latency.
I stumbled upon your page looking for a solution to a problem that was latency related with the OpenAL framework. My first thoughts when reading your post were: maybe I should start using RemoteIO as well. But I quickly realized this would become very difficult for a newcomer to the iPhone platform. But when I read your final update at the bottom of the post, I quickly tried it in the project I was working on and it did the trick.
Best wishes, Theo
Hi,
I am writing a beatbox sequencer for the iphone using CoreAudio.
I have 4 sounds that play on a pattern. I have a AUGraph with IORemote and MultiChannelMixer together. The 4 inputs of the mixer have a callback reading the sound buffers attached to it. It works fine as long as long as sequencer pattern is simple and does not play too many sounds at the same time. If I turn on multiple sounds at the same time, glitches appear and it sound like the CPU is overloaded (I am using an iPhone 1st gen for testing). I am careful not to alloc memory, call UI functions, or NSLog during the callbacks for reading the buffers. The Mixer buffers are filled with datas that are preloaded in a “Soundfile” class. This call has a Queue system that allow multiple overlapping instances of the (same) sound to played – for example 2 hits of the same Kick sound (that is rather long) where the 2nd kick does not stop the 1st one.
Here is the code:
http://pastie.org/1282268
Is there something I am missing or that I should know that could improve my program? Is there a CPU limit on 1st gen iPhone that I should know of?
Thanks.
Pascal
I got a passthrough to work fine.. input from mic to output to speaker .. BUT i want to input from iPhone mic and play back on bluetooth speaker.. I can’t seem to find a solution to this.. is this not allowed?
Jake asked back on November 13, 2010 if there was a way to get
“…input from iPhone mic and play back on bluetooth speaker…”
Is there? If not, why not?
Hi Michael. You seem to have the remoteIO Kung Fu going on, you might be able to help me with a related problem.
Basically I’m mixing 16 different audio files. It all works great on the simulator but on a device I can only successfully mix 4 files. Anything more than that and I either get degraded audio or no audio even though the callback is running.
I can refer you to a more detailed description here.
http://stackoverflow.com/questions/4325248/remoteio-audio-problem-simulator-good-device-bad
Have you any thoughts on what the issue may be? I’m finding it very difficult to find the problem.
Mmm… is there any way to process planar samples instead of interleaved one ? I tried
audioFormat.mSampleRate = 44100.0; audioFormat.mFormatID = kAudioFormatLinearPCM; audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsNonInterleaved; // A Verifier si le kAudioFormatFlagIsNonInterleaved est supporté sur iOS audioFormat.mFramesPerPacket = 1; audioFormat.mChannelsPerFrame = 2; audioFormat.mBitsPerChannel = 32; audioFormat.mBytesPerPacket = 4; audioFormat.mBytesPerFrame = 4;
Then my callback is given int values into 2 buffers, but I cannot make it work… in stereo. Do I have to change something in the output too ?
…well, don’t laugh but my headphone’s jack was not plugged completely and thus got a mono sound… (^ö^) Btw the code I posted is wrong for the “audioFormat.mBitsPerChannel” parameter. It has to be “audioFormat.mBitsPerChannel = 16;” not 32…