Blog

Four common mistakes in audio development

This is a discussion of four common mistakes that audio developers make, how to do better, and how to detect whether there’s a problem. It’s written primarily for developers, but should be accessible to non-developers too. I introduce Realtime Watchdog, a diagnostic tool for developers, and provide a brief survey of popular audio libraries.

Making audio apps is enormous fun — it’s rewarding, there’s huge scope for creativity, and then when you’re done, other people use it to be creative too! There aren’t many fields that are like that, and I consider myself very fortunate to be able to work in this area.

But there’s also a serious side to working with audio. As audio developers we have a responsibility to our users to, basically, not embarrass them in public. A DJ whose equipment emits an ear-piercing crunch mid set will not thank us (well, it depends on the club. Maybe they will?). Nor will a performer whose backing drum machine clicks and crunches distractingly, throwing the performance. Same goes for in private — if the user just nailed a take, only to discover that there’s a giant click in the middle of the recording, they’re going to be cursing our name.

Now we’re living in a post-Audiobus/IAA world, where our users’ setups often span multiple apps, one bad actor can mess everything up, and it’s often impossible to tell from where the problem originates.

Imagine if Loopy HD had glitched in the middle of that?

The audio engineer on The Tonight Show told me the main reason that they chose Loopy for the segment above was because he had been a Loopy user for years, and it has always been solid and reliable.

Even if there’s just a one-in-ten-thousand chance that an app will glitch during a typical session, well, that’s one glitch a day if your app sees ten thousand sessions per day, which is not uncommon. Two glitches a day if it has twenty thousand sessions a day. And I’ll bet most music apps have a higher glitch rate than that.

It can take just one glitch during a live performance for a musician to completely lose faith in their whole setup. The one thing they cannot troubleshoot in their setup is their apps, because it’s an opaque system. And so every app they’re using is indicted. They’ll stop using all of them. It’s an angry Facebook post to all of their musician friends waiting to happen; the exact opposite of what anyone reading this would want.

So, it’s this duty of care that we audio developers have that I want to focus on in this article, because our music apps have to be solid and reliable. All of the time.

First of all, a quick aside: I’m not a perfect software developer — not by a long shot. I make mistakes, I let bugs slip through, although I try my darnedest to keep it to a minimum. So, although there is a high horse present in this article, consider me standing beside it pointing at it, rather than sitting on top of it.

My work with Audiobus and The Amazing Audio Engine tends to bring me up close and personal with a lot of other developers’ code, both newcomers to the field, and developers working for big-name companies.

Unfortunately, much more often than I’d like, the code I’m seeing from both groups breaks one or more of the cardinal rules of working with audio. In fact, I see it all the time. And it’s driving me crazy.

More alarmingly, a number of high-profile libraries exist out there that break the rules too. You can find a brief survey of some popular audio engine libraries at the bottom of this article. I recently had to scramble to fix some glitches in Loopy which, as it turns out, were being caused by a third-party library (not an audio engine, but something else) doing things it shouldn’t.

So, those cardinal rules I mentioned? There are four obvious ones.

  1. Don’t hold locks on the audio thread. Like pthread_mutex_lock or @synchronized.
  2. Don’t use Objective-C/Swift on the audio thread. Like [myInstance doAThing] or myInstance.something.
  3. Don’t allocate memory on the audio thread. Like malloc(), or new Abcd or [MyClass alloc].
  4. Don’t do file or network IO on the audio thread. Like read, write or sendto.

They may seem arbitrary, but violating any of these four can cause your hard-won audio code to devolve to a crackling, clicking mess when you least expect it. Worse, using a third-party library in your app that does this can do the same.

Note that there are plenty more reasons why an app’s audio engine could glitch — logic errors, sub-optimal routines, or just asking too much of the device’s capabilities. But the above four are easy ones to spot and address, and I see them all the time.

The result of violating these rules may be a deafening crunch, or it may be a tiny little click. You won’t know, but lets avoid it.

So, what causes these glitches?

Basically, running out of time. Here’s how that can happen.

Any app that does audio has at least two threads of execution: the main thread, and the audio thread. Often there’ll be other stuff going on too, like networking threads or threads to handle UI rendering.

Threads

The CPU, being a limited resource, is shared between these threads and those of all the other apps currently running:

Scheduling

Rendering live audio is quite demanding: the system has to deliver n seconds of audio data every n seconds to the audio hardware. If it doesn’t, the buffer runs dry, and the user hears a nasty glitch or crackle: that’s the hard transition from audio, to silence.

Audio Deadlines

The audio thread needs to be serviced at regular intervals, and it needs to complete that task within very tight time constraints: just a couple of milliseconds, or less. It’s a realtime thread, which gives it certain privileges. If something’s going on in the UI (blue above), or a network operation is running (orange thread), and it’s time to render some audio, the CPU drops everything to service the audio thread. It gets top priority, ‘cos it needs it.

With me so far? Here’s where it gets sticky.

None of the popular operating systems we use today are true “realtime” OSs. They take a “best effort” approach, which means they try, but provide no hard guarantees. That means something could happen, entirely beyond our control, that could cause the audio thread to be interrupted and run out of time.

So what we aim for is to minimise the risk from our end that something goes wrong.

If you break one of the aforementioned rules within code which is running on the audio thread, something awkward happens. Say we’ve got some code that uses a data structure that’s shared with the main thread. Maybe it’s a list of playing notes, and we want to add to and remove notes from that list in response to the user pressing keys:

// Define some types
struct Note {
  int noteId;
  float frequency;
  float velocity;
  uint64_t startTime;
};
struct NoteList {
  int noteCount;
  struct Note notes[1]; // noteCount-1 Notes follow
};
 
struct NoteList * __noteList;
 
...
 
// Functions to add and remove notes
- (int)addNoteWithFrequency:(float)frequency velocity:(float)velocity atTime:(uint64_t)startTime;
- (void)removeNoteWithId:(int)noteId;

Now, we’ve got some audio rendering code that takes that list, and produces audio based on what’s in it. But we’re working with a resource that’s shared between the main thread, and the audio thread. Because these can be interrupted at any time, or even run simultaneously, we could run into the situation where the audio thread is reading from the data at the same time as it’s being edited by the main thread, resulting in crashes or data corruption.

The usual way to address these concurrency issues is to use a lock (also known as a mutex, or Mutual Exclusion object). This allows only one thread through at a time: when we’re about to interact with the shared data structure, we look to see if it’s already locked. If it is, we wait until it becomes unlocked. Then we lock it ourselves. When we’re done, we unlock it. That way, we avoid the concurrent access situation.

So, continuing our example, we’re going to protect ourselves with a mutex, both here and in the functions that manipulate the list:

pthread_mutex_t __noteListMutex;
 
void MyAudioRenderFunction() {
  // Lock it up
  pthread_mutex_lock(__noteListMutex);
 
  // Make noise
  for ( int i=0; i<__noteList->noteCount; i++ ) {
    ProduceAudioForNote(&__noteList->notes[i]);
  }
 
  // Okay, we're done, unlock
  pthread_mutex_unlock(__noteListMutex);
}

Sorted, right? Well, hold onto your butts.

What happens when we hit the pthread_mutex_lock on the audio thread, if the main thread’s currently updating the list? The CPU’s going to block in the audio thread, and it’s going to ditch the thread in favour of another one that’s not blocking. If we take too long to finish updating the list on the main thread, well…

Preempt

Yep, we ran over time, and the audio system glitched. Here’s what that looks and sounds like:

Glitch

Okay, so what about if we replace our list update code with something that runs super-fast? As long as we only hold the lock for a short time, we’re fine, right?

No: far from it. Some devs think they’re fine as long as they don’t hold locks too long. They’re wrong. Here’s why.

Remember that other, yellow thread in the diagrams above? Well, say it’s got a slightly higher priority than the main thread. Maybe our app is doing some MIDI stuff; maybe it’s doing some time-critical offline processing, or some network communication: all of these may require a higher priority.

The thing about multi-threading is that it’s beyond our control. The scheduler — that mysterious beast which guides the CPU’s attention — can interrupt threads at any point, and give their CPU time to another thread that needs it more. And that’s just within one process; the scheduler also needs to farm out the CPU to other threads, in other running apps. It’s a busy little guy.

So now this happens:

Priority Inversion

Because our secondary thread (yellow) is higher-priority than our main thread (blue), the scheduler steals CPU time from the main thread, upon which the audio thread is waiting. This is called priority inversion.

By minimising the amount of time we’re holding a lock on the main thread, we’re lowering the chances of this happening somewhat, but we’re not eliminating them.

And I’d argue that’s simply not good enough.

Sure, it might work just fine for us and our small group of testers right now. But our app could be seeing thousands of sessions a day across our user base. Combine that with an Audiobus or IAA multi-app environment, and the risks go up, because there’s more going on. Even if there’s a one-in-ten-thousand chance that there’ll be a glitch during a typical session, it’s going to be happening to one of our users every day if our app gets ten thousand sessions a day.

So, no locks. What’s the problem with Objective-C or Swift?

It’s mighty convenient just using Obj-C/Swift to render audio – you can just pass around objects, you can have inheritance, polymorphism, all the good stuff. Plenty of third-party audio libraries out there do this. But guess what?

Objective-C and Swift hold locks as part of their normal operations.

Behind Objective-C’s message sending system (i.e. calling Obj-C methods) is some code that does a whole lot of stuff — including holding locks. Don’t believe me? Take a look for yourself — the source code is available over on opensource.apple.com: objc_msgSend calls __class_lookupMethodAndLoadCache3 calls lookUpImpOrForward calls — yep — lock(runtimeLock), which is a global rwlock_t, and lock(cacheUpdateLock), another.

By the way, accessing a property via the dot syntax (myInstance.property) counts as an Objective-C method call, too, so that is also off-limits.

In fact, you can’t even allow ARC to retain Objective-C or Swift objects because — you guessed it — the retain mechanism holds a lock, too. Check it out: sidetable_retain calls table.trylock(), and if it fails, calls sidetable_retain_slow which calls table.lock().

How about allocating memory?

The thing about malloc and friends — the functions that allocate and give you a block of memory to work with — is that their execution time is unbounded. That means it can take longer than you expect, when you least expect it, which is a problem very similar to priority inversion.

Annoyingly, I struggled to find any direct references here, aside from passing mentions like the above paragraph. I decided not to mount an expedition into the source code because I didn’t know what to look for, or where to look for it, and I wasn’t even sure I’d even find it given that iOS and OS X are both pretty closed-off systems. So, we’ll have to take the word of those more knowledgeable than us, at least for now.

Along with the unbounded execution time, malloc also uses a lock.

The same goes for file and network IO

All of the IO functions — read, fread, fgets, write, send, sendto, recv, recvfrom and friends — also have an unbounded execution time, just like malloc. They need to happen on a secondary thread.

What about libdispatch, and using blocks?

Unfortunately, these are off-limits, too. While you can safely call a block on the audio thread, as long as you don’t retain or release it there, creating a block on the audio thread causes some memory allocation as well as some object retains, both of which will hold locks. But I have good news on that front; if you’re impatient, skip down to the bit about AEMessageQueue.

So, what’s to be done?

As I said earlier, none of the widely-used operating systems are true realtime OSs, which means there are no guarantees. So what we’re after is minimising the chance of getting into trouble.

And there’s good news: there are lots of tools out there to help you out with this, and some pretty easy patterns to follow.

First of all, Objective-C is built around C, and Objective-C instances can actually be accessed just like C structs, from C functions within the @implementation block:

FFCrewMember * jayne;
...
jayne->location = FFLocationBunk;

That’s a plain old pointer dereference, with no Objective-C shenanigans to hold locks or anything, so it’s perfectly safe for use on the realtime thread.

So you can still pass around and use objects; but like I said earlier, you need to avoid any retains. Again, we’re covered: when declaring an Objective-C instance variable, we just need to use the __unsafe_unretained attribute to bypass any ARC stuff:

void MyCFunction(__unsafe_unretained FFFertileLand * thisLand) {
	__unsafe_unretained FFFertileLand * yourGrave = thisLand;
}

Piece of cake, huh?

A note: While offering some feedback on this article prior to publishing, Rolf Wöhrmann of Tempo Rubato (NLog, Nave, iSEM) suggested forbidding any reference to Objective-C or Swift objects from within audio code, even with the __unsafe_unretained attribute, and instead just passing in C or C++ variables. He advocates a total separation of the two. This is certainly the safest bet; it’s a very defensive strategy. If in doubt, this is the way to go, for sure.

What about cross-thread synchronisation? How do we replace locks?

There are lots of possibilities, here. Apple offer a number of very useful facilities within the libkern/OSAtomic.h header, like OSAtomicEnqueue and OSAtomicDequeue, OSAtomicAdd32Barrier and OSMemoryBarrier. It’s also considered quite acceptable to use try locks (like pthread_mutex_trylock), as long as you have a decent idea what you’ll do if you don’t get the lock.

It can be helpful to know that, on all modern processors, you can safety assign a value to a int, double, float, bool, BOOL or pointer variable on one thread, and read it on a different thread without worrying about tearing, where only a portion of the value has been assigned by the time you read it. That’s because byte, halfword and word-length assignment is atomic (ARM® Architecture Reference Manual ARMv7-A and ARMv7-R edition), as long as the variable is naturally aligned — which it will be, if it’s an Objective-C instance variable or in an un-packed struct, for instance. Note that this doesn’t necessarily apply to variables of other types; if you’re on a 32-bit processor and you assign a uint64_t variable, you could get into trouble, because the processor needs two separate instructions to store the value, and so another thread could read the value halfway through.

But if you don’t want to get your hands dirty — and I don’t blame you; thinking through concurrent and preemptable operations hurts my brain too — there are plenty of tools available to help you out.

Here’re the ones I use myself.

TPCircularBuffer is a widely-used circular buffer library that I wrote years back, and still use every single day; you can stick data in one end from one thread, and pull it out from another thread without holding any locks, and with a virtual memory trick it lets you totally ignore the fact that you’re using a circular buffer with a wrap point. It also lets you read and write AudioBufferLists, both interleaved and non-interleaved, and carries AudioTimestamp values too, all of which makes it convenient for use with Core Audio. It’s built right into The Amazing Audio Engine 2, as AECircularBuffer.

AEManagedValue from The Amazing Audio Engine 2 provides a pointer variable which is carefully managed so that assignments are atomic and release occurs only once the audio thread has finished with the value. That is, you can use it to point to any data structure you like, or an Objective-C class, and when you change the value, the old value is only released when it’s not going to mess with the audio thread.

To continue our note-list example from earlier, we could use AEManagedValue to maintain a reference to a NoteList pointer, and simply reallocate the list when we change it:

@interface MyClass ()
@property (nonatomic, strong) AEManagedValue * noteList;
@end
 
@implementation MyClass
- (instancetype)init {
  ...
  self.noteList = [AEManagedValue new];
  ...
}
 
- (int)addNoteWithFrequency:(float)frequency velocity:(float)velocity atTime:(uint64_t)startTime {
  // Get old list, and copy it to new one
  struct NoteList * oldNoteList = self.noteList.pointerValue
  struct NoteList * newNoteList = malloc([self sizeOfNoteListWithCount:oldNoteList->count + 1]);
  memcpy(newNoteList, oldNoteList, [self sizeOfNoteListWithCount:oldNoteList->count]);
 
  // Update
  newNoteList->count++;
  newNoteList->notes[newNoteList->count-1] = ...;
 
  // Assign new list - old value will be automatically freed at a safe time
  self.noteList.pointerValue = newNoteList;
}
 
void MyAudioRenderFunction(__unsafe_unretained MyClass * self) {
  // Get latest value
  struct NoteList * noteList = AEManagedValueGetValue(self->_noteList);
 
  // Make noise
  for ( int i=0; i<noteList->noteCount; i++ ) {
    ProduceAudioForNote(&noteList->notes[i]);
  }
}

Or, to make things even easier: AEArray, also from The Amazing Audio Engine 2, builds upon AEManagedValue to implement a map between an NSArray and a C array which you can access safely on the audio thread. You can either access the Objective-C instances directly on the audio thread, or you can provide a block which will map between those Objective-C objects and a C structure.

So we can revisit our example once more; say we have a MyNote Objective-C class, in an NSArray:

@interface MyClass ()
@property (nonatomic, strong) NSMutableArray * playingNotes;
@property (nonatomic, strong) AEArray * noteArray;
@end
 
@implementation MyClass
 
- (instancetype)init {
  ...
  self.playingNotes = [NSMutableArray array];
 
  self.noteArray = [[AEArray alloc] initWithCustomMapping:^void *(id item) {
    // We'll provide a map between the Objective-C MyNote instance, the properties of
    // which we cannot safely access on the audio thread; and our C struct, which we
    // *can* safely access.
    // This happens on the main thread during a call to "updateWithContentsOfArray", 
    // and the pointer we return will be freed automatically when the original 
    // Objective-C object is removed from the array.
    struct Note * note = malloc(sizeof(struct Note));
    note->frequency = ((MyNote*)item).frequency;
    note->velocity = ((MyNote*)item).velocity;
    note->startTime = ((MyNote*)item).startTime;
    return note;
  }];
  ...
}
 
- (int)addNote:(MyNote *)note {
  // Update our array
  [self.playingNotes addObject:note];
  [self.noteArray updateWithContentsOfArray:self.playingNotes];
}
 
void MyAudioRenderFunction(__unsafe_unretained MyClass * self) {
  // Enumerate the pointers in the array
  AEArrayEnumeratePointers(self->_noteArray, struct Note *, note, {
    ProduceAudioForNote(note);
  }
}

Finally, AEMessageQueue, again from The Amazing Audio Engine 2, lets you schedule blocks to be performed on the audio thread:

[self.messageQueue performBlockOnAudioThread:^{
  _state = newState;
}];

…and in the other direction allows you to safely schedule a target/selector on the main thread:

AEMessageQueuePerformSelectorOnMainThread(
  self->_messageQueue, 
  self, 
  @selector(doSomethingWithTrack:),
  AEArgumentScalar(track),
  AEArgumentNone);

This works a bit like libdispatch, but is completely audio thread safe.

So, how do you know if you have a problem?

I’ve created a tool to make this diagnostic chore a bit easier, the idea for which came from Taylor Holliday (of Audulus fame).

It’s a small library called Realtime Watchdog (also now built right into The Amazing Audio Engine 2, as well as version 1). You add it to your project, and it’ll keep an eye on any unsafe activity on the audio thread, and warn you if it spots anything.

It won’t catch everything, and it won’t catch anything in Apple’s own system code, but it’ll catch some locks, some memory allocation, all Objective-C use (but not Swift), all object retains, and some common IO tasks, in your code and that of any static libraries you’re using.

To use it, just add “RealtimeWatchdog” to your Cocoapods Podfile, (“pod 'RealtimeWatchdog'“) and run pod install. Then you’re done. It’ll automatically inform you about all infractions for your debug builds, and do absolutely nothing in your release builds. For debug builds it will slow down Objective-C message sends a little bit, so you can disable it at any time by commenting out the REALTIME_WATCHDOG_ENABLED define in AERealtimeWatchdog.h.

If you’re not using Cocoapods, check the instructions on the GitHub repository.

If you’re using The Amazing Audio Engine 1 or 2, then just uncomment that REALTIME_WATCHDOG_ENABLED define in AERealtimeWatchdog.h to turn it on.

Some final thoughts

More and more musicians are selling their hardware and turning to iOS, for the portability, convenience, affordability and the power of the platform. I’m often hearing from users about how both Loopy and Audiobus are opening up worlds of creative possibilities for them, and I find that very exciting and rewarding.

But often, too, I hear of disappointment and frustration, apps glitching out and not working right, requiring workarounds and causing anxiety. On those occasions when I’ve gotten involved and worked with developers of misbehaving apps, often it’s because they’re violating the rules of the audio thread: Don’t hold locks, don’t use Objective-C/Swift, don’t allocate memory, don’t do IO.

It is worth mentioning (and here, I’m paraphrasing Rolf Wöhrmann’s wise words) that this advice is based on assumptions about what’s going on at the system level. But iOS and Mac OS X are closed systems, and we can only get a peek under the hood, via opensource.apple.com, and the pickings are very slim. Even if we follow these rules, we could still be getting preempted. So, we make educated guesses. Ideally, we test and experiment, although this can be difficult and technical.

Anyway, my advice? Keep your ear to the ground. Maybe join the Core Audio API mailing list and ask questions. For now, open Xcode and take a look at what you’re doing in your audio code. Try Realtime Watchdog, and use it to check your code and that of any third-party libraries you’re using. You can find a list of popular libraries and their audio thread safety below.

Also, consider checking out C++, which tends to be safer than Objective-C for audio and gives you a lot more features than C alone.

If you’re seeing crackling and glitching and you’ve taken care of the above, then you might want to look at optimising your code; in particular, replacing any scalar operations with vector ones, using something like the Accelerate framework.

By paying more attention to how we code, we can make iOS audio a much better experience for our users. In turn, that’s going to attract more serious users and influencers to the platform, which is going to be great for everyone.

Happy coding!


Some more reading


Acknowledgements

I want to say a big thank-you to the developers who checked my work and gave me some valuable feedback:

And, of course, my partner in crime Sebastian Dittmann of Audiobus and Audanika.


A survey of popular audio engine libraries and their audio thread safety

Note that a “Caution” designation here doesn’t necessarily mean a total indictment of the library; it merely means the library is doing things that raise the chance of a glitch, possibly only fractionally. It’s up to you to decide whether it’s worth it or not. I’ve done my best to make an assessment in each case, but I may be off the mark, and any comments are welcome.

Last updated June 28 2016.

Name Rating Comments
AudioKit 3.1 Safe AudioKit itself is safe (uses AVAudioEngine and the C libraries Soundpipe and Sporth, with good isolation from Swift/Obj-C). It does include EZAudio, which is unsafe, but only uses its plotting features for optional main thread UI, which is fine.
AVAudioEngine (by Apple) Safe Part of Core Audio, so closed source and beyond the reach of Realtime Watchdog’s analysis so evaluation is difficult, but I think we can assume the Apple guys know what they’re doing.
EZAudio 1.1.3
(Deprecated)
Caution Uses Objective-C method calls from main output callback; based heavily around Objective-C
JUCE 4.2.2 Caution Uses a mutex on the output callback which may be contended on main thread by adding/removing callbacks or changing default MIDI output. Also uses a mutex within AudioSourcePlayer which can be contended on main thread by setting the source. Some comments by Julian on the matter, and why they’re not too concerned.
libpd 0.9.1 Caution Uses Objective-C method calls from main output callback; wraps calls in one big @synchronized mutex which may be frequently held by main thread. This is currently being investigated by the libpd team.
Novocaine Caution Uses Objective-C method calls and property accesses from main output callback
Superpowered as of Jun 28 2016 Safe Closed source, but no problems discovered by Realtime Watchdog diagnostic tool. Gábor Szántó also reports it is now lock-free.
The Amazing Audio Engine 1 1.5.7 Safe Full disclosure: this library was written and is maintained by me.
The Amazing Audio Engine 2 Beta Safe Full disclosure: this library was written and is maintained by me.

Note: If you are using a library that’s not listed here, or listed incorrectly, and you’d like me to check it out, leave a comment below or email me. If you’re the developer of one of the libraries marked unsafe and have updated since this was written, please do the same and I’ll update the list.


Appendix

Comments by Julian Storer on JUCE’s use of Mutexes:

Locking AudioDeviceManager when a client is attached or released, and locking AudioSource classes when their sources are added or removed are very similar use-cases whose characteristics are:
1. The lock is held by the audio thread for most of the time it’s active.
2. Other threads will grab it very infrequently – often as rarely as once or twice in the app’s entire runtime. When they do grab it, they hold the lock for only a few nanoseconds while they shuffle a handful of bytes around.

The way these particular classes use mutexes is a really simple solution that’ll work flawlessly in the vast majority of use-cases. Apps that aren’t constantly messing with the audio playback graph topology are extremely unlikely to ever cause an interruption to the audio stream. Apps that do hammer the edge-cases will very occasionally cause a minor glitch. End-users may or may not notice this. On balance, we don’t think that’s too bad a trade-off, given the extra complexity and bug-risk of a more complex lock-free implementation.

, , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

14 Comments

  1. David
    Posted June 15, 2016 at 9:08 am | Permalink

    I’d be keen to know your thoughts on how suitable Cinder is for iOS audio.

    https://libcinder.org/docs/guides/audio/index.html https://libcinder.org/docs/reference/audio.html

    • Posted June 15, 2016 at 11:15 am | Permalink

      Does Cinder actually support iOS? It seems to be just Windows and OS X (sorry…macOS =))

      • David
        Posted June 15, 2016 at 11:31 am | Permalink

        I’m not sure exactly when it was added, but it most certainly does.

        From https://libcinder.org/about: Cinder is a C++ library for programming with aesthetic intent – the sort of development often called creative coding. This includes domains like graphics, audio, video, and computational geometry. Cinder is cross-platform, with official support for OS X, Windows, iOS, and WinRT.

        Android and Linux support is also fairly close (https://github.com/cinder/Cinder/tree/android_linux).

        They have a number of audio demos on their Github: https://github.com/cinder/Cinder/tree/master/samples/_audio

        I’ve only had a cursory look, but I’ve been impressed with the performance.

        • Posted June 15, 2016 at 1:34 pm | Permalink

          Ah! Maybe the website hasn’t been updated. Or I misread something. I’ll check it out! (Or, you can do it yourself with Realtime Watchdog)

          • David
            Posted June 17, 2016 at 7:06 am | Permalink

            Usage is as simple as a ‘pod install’? If so, that is brilliant!

  2. Max
    Posted June 15, 2016 at 11:32 am | Permalink

    In case you’re wondering about why mallocs execution time is unbounded, consider the case of what happens when your app requests a chunk of memory, but both the current heap and system memory are full: the OS will have to swap something out to disk\* before it can service your request for a new page of memory. Now your malloc call is implicitly waiting for disk IO, which is unbounded.

    \* This is of course only true for macOS, as iOS doesn’t have swap space (last time I checked at least). In the case of iOS however, the OS will start going door-to-door with didReceiveMemoryWarnings. I’m not sure what the timeout is on these before apps get jetsam‘d (I can’t imagine there not being one), but I’m sure it’s not one that’s friendly for realtime audio processing.

  3. Posted June 15, 2016 at 12:41 pm | Permalink

    libsoundio is real-time safe.

  4. Derek Jones
    Posted June 15, 2016 at 2:09 pm | Permalink

    Excellent article. Great explanation. Thanks Michael!

  5. Posted June 15, 2016 at 3:03 pm | Permalink

    Thanks Michael! In one of my apps , all the relevant ObjC instance variables use make possibly transformed copies of themselves in a C structure using their - (void) setVar:(type) val and - (type) Var methods. This was working fine until 64 bit came along: the ObjC side laid out the structure differently than the C side! (mostly due to 64bit pointers I think). So I set the C side to compile with the ObjC compiler and all was well again. Boy, that was hard to figure out!

  6. Posted June 15, 2016 at 5:58 pm | Permalink

    Hi Michael,

    I recommend taking a look at Audio Weaver. It’s a framework for real-time audio signal processing. There’s also a graphical tool for designing your signal flow graph. Our primary use case is for embedded audio but we have done iphone apps in the past. You can email me if you have any questions.

  7. Jonathan Mackenzie
    Posted June 16, 2016 at 10:54 am | Permalink

    thanks for a well-informed and helpful article. i wanted to give AEMessageQueue a go but can’t find the function AEMessageQueuePerformSelectorOnMainThread anywhere in TAAE

  8. Jonathan Mackenzie
    Posted June 16, 2016 at 11:05 am | Permalink

    scrub that – was looking in v1 of taae!

  9. Brian Brylow
    Posted June 18, 2016 at 2:42 am | Permalink

    Excellent article Michael! Very thorough dive into the issues and the resolutions. Thanks for sharing.