Blog

Developing Loopy, Part 2: Implementation

LoopyThis is part 2 of a series following the development of Loopy, my iPhone app.

In part 1, I wrote about Loopy’s interface. Part 2 will be more technical, and will cover some challenges encountered during the evolution of Loopy from concept and mockup to working software. Or, more specifically, the stupid things I did along the way.

The Long Path to Audio

Loopy’s audio implementation went through many revisions, as I both gained insight into the workings of the iPhone’s audio subsystem, and chased down performance.

Take One

Audio implementation #1

Audio implementation #1

The audio implementation that Loopy’s first prototype had was based on the common ‘Audio Queue’ services: Each track had an audio queue for playback, and an audio queue for recording. Each queue was just started and stopped as needed.

The problem here was the long delays: Whenever one started a recording queue, for example, there was a half-second-or-so delay before audio started coming in. Obviously, for a music app, this was crazy – you’d lose the first beat!

Take Two

Audio implementation #2

Audio implementation #2

The second attempt involved a single recording audio queue that was always running – when we weren’t recording, we’d just throw away the sound. When a track started recording, the recorded sound from the queue would be pushed into the recording track.

This did away with any startup delays – but the new problem was the time taken to start and stop tracks playing. Again, there was a half-second delay when tracks were started.

Take Three

Audio implementation #3

Audio implementation #3

The third attempt used a single playback queue, to avoid the play/stop delays. When no track was playing, silence would be output; when one or more tracks were playing, their audio would be manually mixed and given to the queue.

This nearly had it, but latency was just too long – if one tapped out a beat, the sound coming out of the speakers would be often half a beat behind, which really messed with one’s head.

Take Four

Audio implementation #4

Audio implementation #4

The final implementation, which turned out fairly well, uses a system that is shrouded in mystery: Remote IO (or IO Remote, or The Great Audio Interface Of Doom depending on where you look)

The Remote IO system gives you near-direct access to the audio equipment, which means you can essentially pick the audio latency you want, at the cost of convenience and sanity.

Learning enough about this system to use it was a long process, and it ended up being a pastie.org snippet, random bits and pieces from a libSDL source code commit notification, and some obscure sample code from Apple that led me in the right direction. The lack of proper documentation here was quite absurd, and not at all helped by the stranglehold that Apple had placed on all development chatter at the time. Thankfully, things are moving in the right direction now. (I even got asked recently to write some documentation for Apple on the Remote IO framework, which was very cool, if mystifying)

Anyway, the new audio engine was fast and responsive, and I breathed a sigh of relief.

Echo Folly

200902141957.jpgThe other aspect of Loopy’s audio worth mentioning was my foray into echo cancellation – the original intention for Loopy was as a ‘performance’ device, able to be used without any necessary bits and pieces – like headphones – plugged in. This, of course, was complicated by the fact that the iPhone’s mic is right next to the speaker.

Consequently I decided to have a go at removing the echo signal from incoming audio.

Echo removal is actually a fairly interesting technology – at least, it is if you’re a geek like me. The general idea is that you have some audio playing, which you remember, and you have some audio recording. The recorded audio, because the speaker is nearby, consists of both the desired signal (singing, etc), plus a version of the audio coming out of the speaker.

200902142005.jpg

Because we remembered what we last put out the speaker, in theory we can then subtract the known audio from the recorded sound, to single out the original desired sound.

200902142011.jpg

As with many such things, it’s a lot more complicated: We have to find the speaker’s sound in the recorded audio before we can subtract it. Even more tricky, it won’t be exactly the same, because it’s distorted by the speaker – the audio will be a different volume, and in the case of the iPhone’s speaker, the bass parts of the audio will be gone, for example. It will also be sampled differently to the original signal.

Testing correlation

Testing correlation

Anyway, I gave it a go. Enthusiastic for the challenge, I started an implementation myself, which included a cross-correlation procedure to find the speaker’s audio in the recorded sound, and a routine to perform a subtraction, with a mechanism to ‘tune’ the procedure by determining how much audio was removed, and tuning parameters accordingly.

And, you know, I got close…but not close enough – not enough sound was removed to make it usable. The main issue was the lack of sophistication of my actual signal removal routine. There are algorithms that do a better job out there, but I didn’t have the time to spend researching them all. Maybe another day!

I tried a pre-built solution, built into the great Speex engine, but the requirements of the Speex echo cancellation library were much too great for the poor iPhone, and sound lag was huge.

So, in liu of having echo removal, Loopy now drops the speaker volume whenever it is recording (The U.S. spent $11 million developing a pen that works in space…The Russians used a pencil).

Seven Different Interfaces

The second major challenge was Loopy’s interface – the six rotating platters. The problem was update rate: The display had to appear smooth with all six tracks playing, which means a framerate of at least 20 frames per second, for all six tracks – at least 120 renders per second.

It sounds easy enough, but Loopy’s interface was re-implemented no less than seven times before I got it right. Some notable stages along this journey:

  • 200902142057.jpgThe first implementation used a sequence of images to draw the rotating ring. The drawing routine (drawInRect for each of the six UIViews) would simply draw each image in turn.

    This turned out to be too slow, and would completely block the interface with more than four tracks playing.

  • Next, I tried putting the drawing routines into a thread, and then drawing to an off-screen buffer (actually, drawing into a UIImage). Then the UIView‘s drawing routine would only have to draw a single image, instead of compositing the background with the ring and other elements

    My plan failed, however – at the time, I couldn’t for the life of me figure out how to draw into an image in a thread (from what I remember after I found out, I think one has to use a CGImage, or something, as the entire UIKit framework isn’t thread-safe), and even drawing a single image appeared to be too much to keep the framerate up, anyway.

  • 200902142048.jpgI discarded the pre-rendered images, and decided to go with an approach which didn’t require sending images to the iPhone’s video card each frame. Instead, I draw a mask and used that to mask out the ring image, with the mask rotated for each frame.

    Surprisingly, even this technique was too slow, even after trying an implementation that consolidated the drawing for all six tracks into a single thread.

200902142101.jpgAfter some consultation with other developers, I realised that even Core Animation just wasn’t cut out to do this kind of rendering (I get the impression that Core Animation, too, creates and uploads image data to the video card for every frame), and threw it all out the window.

I re-implemented the whole thing in OpenGL, with the same ‘masking’ concept (but drawing rotating triangles with texture co-ordinates bound to screen co-ordinates), and got what I needed.

This new implementation was much simpler and easier to maintain – always a good sign that it’s the right one!

Start, Stop, Pad, Truncate

Timing of loop recording was quite tricky to get right – deceptively so.

Right from the start, Loopy’s timing mechanism has been based upon a ‘clock’, which defined a length of time representing a base loop length – in musical terms, a bar.

200902151243.jpg

Tracks could then be multiples of the bar length, or a half, a quarter or an eighth of the bar. They could also be offset by a certain amount of time, meaning that recordings could start any time and would be kept in sync.

The original implementation forced the recording to extend to multiples of the ‘clock’ length:

200902151239.jpg

This meant that if you tapped to stop recording, the track would continue recording anyway, until the next clock tick (or half-tick, etc.), plus offset, was reached. This was the “don’t trust the user” approach, assuming that users wouldn’t keep time properly and would need to be guided.

I tried several variations on this theme, testing different logic, such as if the recording length is more than X time than the clock length, keep recording until the next tick, otherwise stop and delete the last X of the track.

That whole concept was generally a bad idea, though, and resulted in people (including myself, once or twice) thinking the interface wasn’t working properly.

The final implementation stops as soon as directed, which feels much better. It will either pad the rest of the track, so that it is up to multiples of the clock length (or a full quarter/half/etc.):

200902151309.jpg

…or it will truncate if it’s within some threshold of a beat:

200902151309.jpg

Something I realised as I was using it was that it was impossible to record things like anacruses (AKA upbeats), where a riff starts just before a beat. With some experimentation, I decided to replace the straight truncation with a mix-then-truncate approach:

200902151305.jpg

This is how Loopy operates now, and I think it works fairly well.

A Sense of Rhythm

The other timing issue was related to synchronisation and recorder latency – that is, if you record one track, then record another one, you want both recordings to be in time with each other. This was, and still is, tricky.

One has to track the latency for both the ‘play’ path – the time taken for audio in the memory buffer to get pushed out the speaker and heard – and the ‘record’ path – the time taken for sound to be digitised and stored in a buffer.

200902151328.jpg

In Loopy’s current implementation, this latency estimation is a hard-coded number worked out during testing: This number is used to offset the time associated with a recording, so that it stays in sync. And on my iPhone, it works great. However, I hear reports that the timing is off for other users, and so this is still an outstanding issue.

Does this mean that latency varies between devices – manufacturing variations? Or, is it software-based – perhaps if the device is busy doing some other things, like checking email, latency will increase?

Time will tell, I suppose.

The Way Forward

Good software is a constantly-evolving thing, which grows and is guided by its users, and this is certainly the plan for Loopy. I recently wrote a status update on Loopy, which outlines a little of the planned path.

While the original idea for Loopy was very constrained, many users are already seeing a much greater potential, which is exciting and gratifying, and I’m looking forward to taking it in new directions.

At some stage, I would like to write a third article for this series, covering promotion. This will be a while in coming, though, as this is a learning curve I have yet to climb! I will say this: My next experiment will be the creation of a free, “Lite” version of Loopy, which advertises the full version. This seems to have worked ridiculously well for some, so it’s well worth a shot!
For those who made it this far, thanks for reading – I’d love to hear your comments on these articles, and on Loopy itself.

Tagged , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

21 Comments

  1. Posted February 16, 2009 at 2:03 am | Permalink

    Thanks so much for this extremely interesting and beautifully designed blog post. The development of my “Wordle” toy saw many analogous “garden path” designs, but I don’t know if I could ever recapture and relate them in as organized and thorough a form as you’ve achieved here. Kudos!

  2. Charley Lanusse
    Posted February 17, 2009 at 10:36 am | Permalink

    Fantastic post, Michael! It’s rare to find someone so technically adept who can also communicate so elegantly. This post furthers really furthers my understanding of iPhone audio units. Thanks for sharing your struggles and epiphanies.

  3. Posted February 17, 2009 at 2:04 pm | Permalink

    Awesome post Michael. Very enlightening and encouraging to read about your creative and technical process. It was also fascinating to hear about the compromises along the way.

  4. Ray Valdes
    Posted February 21, 2009 at 3:21 am | Permalink

    Great post, especially the informative diagrams. Must have taken some time to create.

  5. Hamish Allan
    Posted March 4, 2009 at 9:07 am | Permalink

    Hi Michael,

    Many thanks for all your efforts in putting all of this information together. Your writing is clear and your presentation is marvelous.

    In the interests of full disclosure, I should start by saying that I came across your blog during a little preliminary research into writing a "loop pedal" app for the iPhone. So if you'd rather not reply to my comment because I might be a potential competitor, I'll understand. That said, I envisaged the focus of my app as being rather different to yours — the primary function of mine would be the equivalent of a single track in mix mode for Loopy:

    http://www.youtube.com/watch?v=LxbfBRNY4M8

    The interface I have in mind is completely different to yours, designed to facilitate jamming over a single loop, just as a traditional loop pedal does. Loopy is more powerful, but I want to make that common case more accessible (though I have considered multiple tracks).

    But because I want to focus on the "jam", echo cancellation is quite important to me. You mention that you attempted a cross-correlation technique, but I'd have expected the latency to be pretty much constant, with the overwhelming component of "echo" being direct from speaker to microphone rather than via the walls of the room or whatever. You hard-coded the hardware latency for syncing record and playback clocks; is it not possible to do the same for the outputbuffer-to-speaker-to-microphone-to-inputbuffer round-trip latency?

    I appreciate you spent some time on this, so there's probably something I'm missing. Obviously there's still the disparity between the original sound and the speaker's tinny reproduction to deal with, but it just struck me as odd that you needed anything more than a static technique to do the lining up of the signals.

    I don't really expect an answer, though I'd appreciate a reply even if just to say you don't want to talk to me As I say, I just wanted to introduce myself and state my intentions, as I'm bound to get involved in conversation with you in threads about RemoteIO and I don't want to be disingenuous. And thanks again for all the information you've made available.

    Best wishes, Hamish

    • Posted March 5, 2009 at 6:27 pm | Permalink

      Hi Hamish! Please excuse the delay in responding to your comment, Thanks for the kind words! I'm glad I could help.

      That's very thoughtful of you, with your disclaimer; that's fine, though.

      Those youtube videos were fantastic – that guy is brilliant!

      As far as the echo cancellation stuff goes, the main difficulty isn't so much the identification of the echo delay as much as the actual removal process, which is why my efforts failed; it's not just a subtraction of one signal from the other, it's actually a process that involves iterative refinement and more academic papers than I felt like reading =) I recommend taking a peek at the Speex source for some inspiration (the comments also refer some papers on the subject) – I'd love to hear how you go. If you succeed, maybe there's hope for the rest of us!

      It may be a large amount of work – it would be a very helpful piece of functionality though. Maybe it's something we could collaborate on somehow, since it would definitely benefit us both.

      Anyway, let me know how you go (if you feel like doing so).

      All the best!

      • Hamish Allan
        Posted March 6, 2009 at 11:27 am | Permalink

        Hi Michael,

        The more I consider it, the more I think that echo cancellation might be at odds with the very purpose of the app.

        For example, in "Just For Now" after Imogen has built up the main vocal loop, she claps a rhythm twice round the loop. In the second iteration, how is an adaptive filter to differentiate between the clapping coming from her hands and that coming from the monitor speakers? (Presumably, in the studio, they actually use a gated short-range mic to work around the problem.)

        If her timing and the audio reproduction are both perfect, the filter should theoretically remove the entire input signal. If her timing is not perfect, how can a filter be expected to converge? One could perhaps try to capitalise on the imperfect fidelity in the iPhone, but that doesn't really get you away from the bottom line: echo is rhythmic in nature.

        I think perhaps your approach is not just the easiest but actually the only way to go (maybe with a little adaptive gating?) Ho hum, I guess that rules out performance characteristics of the kind I'd hoped for. (ThePETEBOX is great, isn't he? He's damn good without the loop pedal too!)

        Cheers, Hamish

        • Posted March 6, 2009 at 11:51 am | Permalink

          He's pretty unbelievable – makes me want to learn to beatbox =)

          You make some good points – particularly with the clapping example. Echo removal is a fairly imprecise science, even when it's done well, and it can certainly interfere with incoming audio. Whether such interference is the exception or the rule when it comes to a musical looping application is something I'm not sure of.

          The primary limitation of the whole thing is, as you mention, the hardware – having the mic right next to the speaker on the iPhone isn't particularly conducive to a live performance-based app, unless users bring their own external hardware and connect it up to the iPhone.

          What would be perfect is to somehow gain access to the echo cancellation that's in place for speakerphone mode, but given that it's probably just embedded right into the hardware's audio pipeline, that's extremely unlikely to occur =)

  6. Bzit
    Posted April 2, 2009 at 10:48 pm | Permalink

    Hi Michael !

    Thanks for sharing your knowledge, that's not so frequent.

    As programmer and user of Loopy, I was wondering how you manage the memory part of the audio.

    Do you write the audio data directly to files, or is it retained in RAM partially or completely ?

    User side, the question becomes : Can we record almost infinite length loops within loopy ^^ ?

    Looking forward for the community features on Loopy !! By the way, will it be an update of Loopy or another app ?

    Cheers

    Raphaël

  7. kaustubh vibhute
    Posted April 14, 2010 at 8:18 pm | Permalink

    Hi Michael,

    I am really stuck on this one and need some opinion….. I am trying to design a step sequencer as the heart of a music app where beats can be played from a beat pad. The default quantization I have is 1/64 for one bar and maximum of 27 sound files for each time slot of the bar (64). Right now I am creating 1 AUGraph, 1 Audio Unit. I have 27 input buses for each of the time slots ( up to 64 ). But this approach does not work for me since 27*64 buses in all and this is probably creating a lot of noise while trying to play back the recorded sounds. Can you suggest me a good design approach towards creating a sequencer in terms of how many AUGraphs, Audio Units and input buses to be used?

    by the way…really appreciate the knowledge you have shared on the dev forums and over here. Thanks!

  8. Jason
    Posted May 3, 2012 at 8:51 pm | Permalink

    Hey, I was looking into echo cancellation, and I downloaded loopy. It looks like you were able to do it. Did you use the VoiceProcessingIO? Do you have any thoughts or info on using that audio unit?

    Thanks! Jason

    • Posted May 3, 2012 at 9:11 pm | Permalink

      Hey Jason,

      Yep, you got it – it’s all VPIO. It’s brilliant, a great tool. Not great for music, though – if you’ve played with it in Loopy much, you’ll notice it pretty much massacres anything repetitive. But it’s better than nothing. There’s really nothing to using it – you just use it in place of the normal Remote IO unit. One caveat is that it won’t play nicely with anything less than a 10ms (0.01s) hardware IO buffer duration – it’s pretty CPU intensive, and at lower buffer durations it’s just doing too much work, particularly for the iPhone 4 and below.

      • Jason
        Posted May 3, 2012 at 9:14 pm | Permalink

        Awesome thanks! So i’m wondering in your settings menu->noise reduction toggle, is that something totally different?

        • Posted May 3, 2012 at 9:21 pm | Permalink

          Nope, that just toggles VPIO use (actually, in the next version, i’ve removed the option, and just use VPIO all the time, when headphones aren’t in use)

      • Jason
        Posted May 11, 2012 at 2:58 pm | Permalink

        Hey Michael,

        Tried using VPIO. The only problem I’m having with it now is that it seems to automatically lower the gain for all output sounds. When I switch back to RemoteIO, sound levels are back to normal.

        I tried disabling kAUVoiceIOProperty_VoiceProcessingEnableAGC and also kAUVoiceIOProperty_DuckNonVoiceAudio, but no dice. Did you find this problem when you were doing Loopy?

        • Mike
          Posted November 14, 2012 at 7:56 pm | Permalink

          Hi! We’re having this exact same problem, and are very curious to hear if anyone else has encountered it or found a solution. It seems very strange that the VPIO unit is able to lower volume for all sounds including e.g. AVPlayer output, but that’s what it seems to be doing.

          • Jason
            Posted November 15, 2012 at 12:58 am | Permalink

            Hey Mike, have you noticed that this is especially true for iOS 6? I found a bug report about it, but wanted to know from first hand sources if it’s true. Here’s the thread on apple dev forums:

            https://devforums.apple.com/thread/162542?tstart=0

  9. Pier
    Posted November 22, 2012 at 8:04 am | Permalink

    Hi Michael,

    It’s me again. Could you share a bit more on how you addressed the issues of play latency and record latency? Do you do any quantisation on the first loop? If you move the recording backwards in time to address recording latency, wouldn’t you run the risk of losing some recording data in the front part of the loop? Am also interested in what you actually do with the play latency. Thank you.

    Pier.