Three thousand reboots later, I decided enough was enough, and set about seeing if I could run a virtual machine with a small Linux installation which operated the card on behalf of my Mac, and shared the resulting Internet connection. It was a long learning curve, but I managed to get it all working quite satisfactorily, along with an Internet Sharing setup on my mac to share the connection onwards to other machines on the local network.

What’s more, the drivers I’m using under Linux — actually, they’re Windows XP drivers, running under the fantastic ndiswrapper utility — are brilliant, and I get much better signal strength, stability and throughput, to my great surprise.

If you’re in a similar situation, and either have a crappy driver to contend with, or no driver at all, or even just want a more solid firewall between you and the dangerous open WiFi world, here’re some instructions on how to get it set up, along with a virtual appliance I put together to make it all happen.

A warning in advance: This is Linux, so it aint for the faint of heart. If you’re not familiar with Linux, by all means give it a go, but be forewarned that if everything doesn’t work for you out-of-the-box, it is a steep learning curve.

The virtual appliance is set up by default to run the RTL8187 device using ndiswrapper, but the standard wireless drivers package is installed, and should automatically work with another adapter. If not, let me know and I might be able to assist.

Step 1: Download

First, download and install VirtualBox, the fantastic and totally free VM software. You’ll also want to grab the “VirtualBox 4.1.2 Oracle VM VirtualBox Extension Pack”, which brings USB 2.0 support.

Next, download the Microcore Router virtual appliance that I put together (it’s 45Mb). This is set up for use with Microcore Linux, and contains:

• Wireless device drivers
• Ndiswrapper
• The Windows XP version of the RTL8187 device driver
• iptables, with a script to set up IP masquerading
• dnsmasq, set up to provide a DHCP address to your Mac host
• Wireless-tools (iwconfig, etc), and WICD

Finally, download the Microcore live CD image (the virtual appliance was set up with Microcore 3.8.4 – if you wish to use newer versions, you’ll also need to update the extensions, but you’re on your own there.).

Step 2: Installation and Setup

Install VirtualBox, then double-click on the extension pack to install that. Then, double-click on the Microcore-Router.ova virtual appliance to import it.

Next, we need to create a virtual network interface to link the guest virtual machine, and your Mac. Open VirtualBox preferences, and click the “Network” tab. Click the “+” button to create a new host-only network, and enter the settings below (192.168.22.2, netmask 255.255.255.0). Click the “DHCP Server” tab and make sure it’s disabled, then click OK. This virtual interface is already selected for use within the VM’s settings.

Now attach the Microcore ISO to the virtual appliance by clicking the “Storage” section of the VirtualBox VM settings, then selecting the CD item underneath “IDE Controller”, selecting “Choose a virtual CD/DVD disk file…”, and choosing the Microcore ISO. This lets us boot the VM into Microcore.

Finally, if you’re using a USB network device, plug it in now. Attach the network device to the VM by opening the “Ports” (or “USB”) settings, then clicking the USB “+” symbol and selecting the device.

Note: I’ve already added the RTL8187 device to the VM setup, so if you’ve got one too, you shouldn’t have to do anything here.

Step 3: Boot VM

Now, you’re ready to boot. Close the settings window and hit “Start”. You should see the Microcore boot screen appear, with a prompt for boot options.

Type the following in, to make Microcore boot with the setup contained within the virtual appliance:

mc opt=sda1 restore=sda1

If you’re using an RTL8187 too, then you’ll want to blacklist the buggy Linux device driver here, so that ndiswrapper can run the device with the improved Windows drivers:

mc opt=sda1 restore=sda1 blacklist=rtl8187

Hit enter to start booting, and within a few seconds you should be faced with an empty screen and a prompt.

With a bit of luck, your network device has been detected and started up; type wicd-curses and hit enter to launch the WiFi control panel.

If you don’t see any networks here, and you expect to, then it could be that your network device hasn’t been picked up. Otherwise, breathe a sigh of relief and press on.

Step 4: Configure the Mac

Now that we’re booted and running the VM, we need to set up your Mac to access the internet via the VM’s connection.

Minimise the VM window — we’re done for now — and open up a Terminal window on your Mac. Ping 192.168.22.1, which is the VM’s configured address. Do you see ping replies? Then you’re almost there.

The final step is to make your Mac use the guest’s DHCP server, to obtain the correct settings. Type the following in Terminal to make it happen:

sudo ipconfig set vboxnet0 DHCP

To make sure it’s done the right thing, type ifconfig vboxnet0. You should see something like:

vboxnet0: flags=8843 mtu 1500
    ether 0a:00:27:00:00:00 
    inet 192.168.22.2 netmask 0xffffff00 broadcast 192.168.22.255
    inet 192.168.22.72 netmask 0xffffff00 broadcast 192.168.22.255

That is, the default address we specified when setting up the VirtualBox host adapter, and a new alias that has an address assigned by the DHCP server running within the VM. If you see this, then you should be good to go: Try opening up a web browser or pinging google.com.

If it’s working, then congratulations! Mission accomplished.

Optional Step 5: Internet Sharing

For bonus points, you can set up your Mac so that other computers on your network can share the connection. It’s not as simple as just starting up Internet Sharing from System Preferences, however, because System Preferences doesn’t know about the virtual network device we’re using.

Instead, we have to do it the old-fashioned manual way: Set up IP forwarding, NAT, and set up a DHCP server. I’ve created a service (loosely based upon Schley Stauffer Andrew Kutz’s akin service) that monitors the VM’s network state, and automatically does all this, sharing the VM’s connection via Ethernet and Wi-Fi.

Download the service resources here:

vmboxnat.zip

Extract, then go to the folder in Terminal and type:

sudo ./install

This will install to /usr/local/vmboxnat, and register the service with launchctl. This is a script which runs in the background and starts up internet sharing when the VM is detected, then stops it when the VM goes away.

If you open up Console now, you should see something like:

…au.id.tyson.michael.vmboxnat[13099]: Starting vmboxnat
…au.id.tyson.michael.vmboxnat[13099]: assigned DHCP address
…au.id.tyson.michael.vmboxnat[13099]: created alias interfaces
…au.id.tyson.michael.vmboxnat[13099]: enabled ip forwarding
…au.id.tyson.michael.vmboxnat[13099]: enabled natd
…au.id.tyson.michael.vmboxnat[13099]: established ipfw rules
…au.id.tyson.michael.vmboxnat[13099]: enabled bootpd

That means you’re all set.

Note that you’ll need to manually create an ad-hoc WiFi network, as we can’t automatically create a network the way the official Internet Sharing system does. Once you’ve created it, though, other hosts accessing that network should automatically receive an address via DHCP, and be good to go.

I’ve just discovered a critical issue with Sparrow that I thought others should be warned of. It just bit me, big-time, when in the middle of an important negotiation with a third party, an earlier unpolished draft was sent out instead of the final email, when I experienced a momentary Internet connection dropout.

Sparrow has an issue where hitting “Send”, in unreliable network conditions, on a draft message that has been modified, results in a prior copy of the message being sent, rather than the latest version.

To replicate this issue reliably:

1. Create a new message (Cmd-N)
2. Specify a recipient (oneself), subject, and some body content (say, the letter “A”)
3. Hit “Save”, to save the draft
4. Change “A” to “B”, then disconnect the network connection (in my case, turning off the radio on my external WiFi adapter), then hit “Send”. Sparrow will report a failed connection, and will keep the outgoing message with state “Pending”
5. Restore network connection. Upon detecting the restored connection, Sparrow will proceed to send the message
6. Check email. Received test email will have “A” in the email body.

Scary stuff. I hope they’ll fix this soon, but it’s going to make the app difficult to trust in future!

Update: I should mention, a probable workaround for this is to never, ever hit “Send” until you’ve saved the draft, and verified that the save operation has completed (perhaps even restart the app first).

They’re quite useful for doing various quick things with clipboard contents, like, say, writing it to a file:

pbpaste > Desktop/output.txt

Or altering them:

pbpaste | sed s/bacon/bea-con/g | pbcopy

Fun!

A little utility I just put together does that for you: Post Grabber detects POST data and generates an equivalent “curl” command that can be used in shell or Automator scripts.

Post Grabber works with its own internal browser, so it can intercept POST submissions directly. That means it works with HTTPS, unlike the traditional web sniffer approach.

Download the app, or see the source on GitHub.

Features:

• Detects your Internet connection automatically, or you can start and stop the timer manually.
• Keep a log of your usage, including weekly, monthly, and all-time totals, and daily average, with the ability to reset these.
• Time in configurable blocks of time, if your carrier charges in blocks (e.g., 15 minutes)

Grab it here:

Internet Timer

NSTimer is great for that purpose, as well as repeatedly performing actions, but it’s most convenient utility method, scheduledTimerWithTimeInterval:target:selector:userInfo:repeats: only takes one ‘id‘ argument. Using the NSInvocation equivalent, scheduledTimerWithTimeInterval:invocation:repeats: requires creating and setting up the NSInvocation itself, which is always verbose and a pain.

NSProxy is a wonderful little construct that lets us interact with it like we were talking to the original object. I first learned how it works from a post by Shaun Wexler which shows an easier way to send a message on the main thread, by using the NSInvocation given via the NSProxy’s forwardInvocation: method. The same technique can be used to easily create a configured NSTimer.

So, instead of some awful thing like this:

NSInvocation *i = [NSInvocation invocationWithMethodSignature:[mapView methodSignatureForSelector:@selector(selectAnnotation:animated:)]];
[i setTarget:mapView];
MKAnnotation *annotation = view.annotation;
[i setArgument:&annotation atIndex:3];
BOOL flag=YES;
[i setArgument:&flag atIndex:4];
[NSTimer scheduledTimerWithTimeInterval:0.5 invocation:i repeats:NO];

We can do this:

[(MKMapView*)[TPTimerProxy scheduledTimerProxyWithTarget:mapView timeInterval:0.5 repeats:NO] selectAnnotation:view.annotation animated:YES];

Here’s the juice:

@interface TPTimerProxy : NSProxy {
    id target;
    NSTimeInterval timeInterval;
    BOOL repeats;
 
    NSTimer *timer;
}
+ (TPTimerProxy*)scheduledTimerProxyWithTarget:(id)target timeInterval:(NSTimeInterval)timeInterval repeats:(BOOL)repeats;
- (void)invalidate;
@end
 
 
@implementation TPTimerProxy
 
- (id)initWithTarget:(id)aTarget timeInterval:(NSTimeInterval)aTimeInterval repeats:(BOOL)repeatFlag {
    target = aTarget;
    timeInterval = aTimeInterval;
    repeats = repeatFlag;
    return self;
}
 
+ (TPTimerProxy*)scheduledTimerProxyWithTarget:(id)target timeInterval:(NSTimeInterval)timeInterval repeats:(BOOL)repeats {
    return [[[TPTimerProxy alloc] initWithTarget:target timeInterval:timeInterval repeats:repeats] autorelease];
}
 
- (void)invalidate {
    if ( timer ) {
        [timer invalidate];
        timer = nil;
    }
}
 
- (NSMethodSignature *)methodSignatureForSelector:(SEL)selector {
    return [target methodSignatureForSelector:selector];
}
 
- (void)forwardInvocation:(NSInvocation *)invocation {
    [invocation setTarget:target];
    timer = [NSTimer scheduledTimerWithTimeInterval:timeInterval invocation:invocation repeats:repeats];
}
 
@end

This can be done by adding -[NSException raise] and objc_exception_throw to your breakpoints list.

Once an exception happens, you can then check out the exception itself to see what went wrong. The approach varies between platforms. If you’re in the simulator (or any Mac OS X app running on Intel), the exception will be stored in the $eax register. Take a look by typing:

po $eax

If you’re on the iPhone, it’ll be $r0, so:

po $r0

So, I wrote a class that provides similar functionality to CABasicAnimation, but works on any object. I thought I’d share it.

Features:

• From/to value settings (currently only supports NSNumber and scalar numeric types, but easily extendable)
• Duration, delay settings
• Timing functions: Linear, ease out, ease in, and ease in/ease out
• Animation chaining (specify another configured animation for chainedAnimation, and it’ll be fired once the first animation completes)
• Delegate notification of animation completion
• Uses a single timer for smooth lock-step animation
• Uses CADisplayLink if available, to update in sync with screen updates

Use it like this:

- (void)startMyAnimation {
  TPPropertyAnimation *animation = [TPPropertyAnimation propertyAnimationWithKeyPath:@"state"];
  animation.toValue = [NSNumber numberWithFloat:1.0]; // fromValue is taken from current value if not specified
  animation.duration = 1.0;
  animation.timing = TPPropertyAnimationTimingEaseIn;
  [animation beginWithTarget:self];
}

Make sure you also include the QuartzCore framework, used to access CADisplayLink, if it’s available.

It’s BSD-licensed.

Grab it here: TPPropertyAnimation.zip

Reginald is a kindly old gentleman devoted to assisting you with those tricky regular expressions.

Provide some sample input, and your regular expression, and Reginald will provide you with colour-coded output and a list of all your matches and the corresponding capture groups for your exploration. Select a match or capture group in the list to the right, and the corresponding text will be selected in the panel to the left.

Reginald is built on RegexKitLite, and so uses the ICU syntax.

It will run on Mac OS X 10.6 and above.

Download Reginald here, or access the source on GitHub.

This can be handy for performing quick operations, like replacing text with regular expressions, sorting lists or swapping fields around.

When triggered, the service requests a command to use for filtering, then runs the command and replaces the selected text with the result.

Some sample operations:

• Sort lines alphabetically/numerically: sort or sort -n
• Change to lowercase: tr "[:upper:]" "[:lower:]"
• Replace a spelling mistake, taking care of case: sed -E 's/([tT])eh/\1he/g'
• Re-order elements in a tab- or comma-separated list: awk '{print $2 $1}' or awk -F, '{print $2 "," $1}'

Put it in Library/Services, and it should appear in the ‘Services’ menu.

Filter through Shell Command.zip

Talkie is my newest product, a Walkie Talkie for iPhone and Mac.

In Part 1 of this series, I wrote about basic broadcasting. This works fine with one network device, but it’s worth discussing how to send through all devices, so you can communicate with others connected via, say, Ethernet and WiFi simultaneously.

So, in Part 2 I’ll write about the approach I took in Talkie for broadcasting from all network devices (a.k.a. network interfaces), so that one can communicate with others connected via WiFi, Ethernet (on a Mac), and any other network devices simultaneously.

Bind them

From Part 1, we have a working broadcast mechanism, but it will only send through the default interface — whatever you’re connected to the network via. This is often sufficient, but if you have more than one device that you communicate through, like Ethernet and WiFi, then you will find that it only works with one.

In order to send through all your connected network interfaces, we need to create one socket for each interface, and bind the socket to its corresponding interface.

Here’s how:

First, we need to obtain a list of all network interfaces with getifaddrs.

#include <ifaddrs.h>
...
struct ifaddrs *addrs;
int result = getifaddrs(&addrs);
if ( result < 0 ) {
  // Error occurred
  return 0;
}

Now, addrs is a list of interfaces that we can iterate over. We now do so, picking out those devices that support broadcasting, and that aren’t loopback or point-to-point devices — loopback is an internal interface that is provided for your computer’s inner dialogue, and point-to-point (ppp) devices include dialup interfaces, 3G modems and the like. We can exclude those guys.

const struct ifaddrs *cursor = addrs;
while ( cursor != NULL ) {
  if ( cursor->ifa_addr->sa_family == AF_INET 
          && !(cursor->ifa_flags & IFF_LOOPBACK) 
          && !(cursor->ifa_flags & IFF_POINTOPOINT) 
          &&  (cursor->ifa_flags & IFF_BROADCAST) ) {
 
    // We will do some stuff in here
 
  }
  cursor = cursor->ifa_next;
}

Now, for each interface that meets our criteria, we create a socket (which we covered in Part 1), then bind the socket to the network interface, to force transmission from that particular device. Finally, as we did in Part 1, we enable broadcasting using setsockopt with SO_BROADCAST.

We want to store the sockets we create in an array, so we can access them later. If we assume a maximum number of interfaces we will support (lets call it kMaxSockets), we can just use an array of that length. So, putting it together:

#define kMaxSockets 16
...
int sock_fds[kMaxSockets];
int number_sockets = 0;
 
while ( cursor != NULL && number_sockets < kMaxSockets ) {
  if ( cursor->ifa_addr->sa_family == AF_INET 
          && !(cursor->ifa_flags & IFF_LOOPBACK) 
          && !(cursor->ifa_flags & IFF_POINTOPOINT) 
          &&  (cursor->ifa_flags & IFF_BROADCAST) ) {
 
    // Create socket
    sock_fds[number_sockets] = socket(AF_INET, SOCK_DGRAM, 0);
    if ( sock_fds[number_sockets] == -1 ) {
      // Error occurred
      return 0;
    }
 
    // Create address from which we want to send, and bind it
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_addr = ((struct sockaddr_in *)cursor->ifa_addr)->sin_addr;
    addr.sin_port = htons(0);
 
    int result = bind(sock_fds[number_sockets], (struct sockaddr*)&addr, sizeof(addr));
    if ( result < 0 ) {
      // Error occurred
      return 0;
    }
 
    // Enable broadcast
    int flag = 1;
    result = setsockopt(sock_fds[number_sockets], SOL_SOCKET, SO_BROADCAST, &flag, sizeof(flag));
    if ( result != 0 ) {
      // Error occurred
      return 0;
    }
 
    number_sockets++;
  }
  cursor = cursor->ifa_next;
}

Finally, as before, we can setup a broadcast address to send to, and use sendto to broadcast, this time for each socket we created:

// Initialise broadcast address
struct sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_BROADCAST;
addr.sin_port = htons(kPortNumber);
 
// Send through each interface
int i;
for ( i=0; i<number_sockets; i++ ) {
  int result = sendto(sock_fds[i], data, length, 0, (struct sockaddr*)&addr, sizeof(addr));
  if ( result < 0 ) {
    // Error occurred
    return 0;
  }
}

Note that the receive routine only needs a single socket, as we can receive on any interface when we use INADDR_ANY. So, the receive routine needs no changes from the single-interface version we saw in Part 1.

Here’s the test app with the above modification: broadcast_sample_all_interfaces.c

Again, compile by opening Terminal, and typing make broadcast_sample_all_interfaces or cc -o broadcast_sample_all_interfaces broadcast_sample_all_interfaces.c, then run it with ./broadcast_sample_all_interfaces "Message to send" to send, or just ./broadcast_sample_all_interfaces with no arguments to listen.

You may notice that multiple messages may be received: These have probably arrived via multiple network interfaces, virtual or otherwise. It’s usually a good idea to check for duplicate messages, if this is an issue for program operation, by including a sequence number into the message — this will be discussed in Part 3.

It also may be a good idea to ignore your own messages, which may find their way back to you. One way to accomplish this is to examine the source address (addr in the example above) and compare it with your local interface addresses (stored in addrs, above). If you get a match, the message came from you, and you can drop it.

Multicast

Broadcast is fine when everyone on the local network is interested in what you have to say. If this isn’t the case, though (lets face it, those Chuck Norris jokes aren’t for everyone), effort is wasted delivering to those who aren’t particularly interested.

Multicast works by using a specific address that one ‘subscribes’ to in order to receive messages sent to that address. So, it’s opt-in, allowing for better efficiency and one day, Internet-wide support for ‘to-many’ communications.

Well, in theory. Actually, multicast is still quite new, and for the most part — from what I understand — it behaves pretty much like broadcast on a local area network. However, support can only increase, and given that many services already use it — Multicast DNS (mDNS), also known as Bonjour, being one of the most well-known examples — it seems a good idea to follow their lead. Note also that IPv6, the successor to IP as we know it, and our saviour-to-be from our little Internet overpopulation problem (among other things), doesn’t even have broadcast provisions — the future is all multicast.

So, for these reasons, Talkie speaks multicast, instead of plain ol’ broadcast.

Making use of multicast is relatively straightforward: To receive, you join a multicast group using setsockopt with IP_ADD_MEMBERSHIP, and the address of the multicast group, which is in the range 224.0.0.0-239.255.255.255 (for IPv4, of course). To send, you just use sendto to transmit data to a multicast group address.

Using multicast on all network interfaces is just a little more complicated. Here’s how it’s done with Talkie:

Sending

The send routine is very similar to the one above, using broadcast. However, instead of using bind to specify the outgoing network interface and enabling broadcast, we assign a multicast interface using setsockopt with IP_MULTICAST_IF. And, instead of transmitting to the broadcast address, we transmit to the multicast group address.

Again, we loop through all network interfaces. This time, we pick out those that support multicast (ifa_flags & IFF_MULTICAST):

const struct ifaddrs *cursor = addrs;
while ( cursor != NULL ) {
  if ( cursor->ifa_addr->sa_family == AF_INET 
          && !(cursor->ifa_flags & IFF_LOOPBACK) 
          && !(cursor->ifa_flags & IFF_POINTOPOINT) 
          &&  (cursor->ifa_flags & IFF_MULTICAST) ) {
 
    // We will do some stuff in here
 
  }
  cursor = cursor->ifa_next;
}

And, after creating the socket, we assign the interface:

if ( setsockopt(sock_fds[number_sockets], IPPROTO_IP, IP_MULTICAST_IF, &((struct sockaddr_in *)cursor->ifa_addr)->sin_addr, sizeof(struct in_addr)) != 0  ) {
  // Error occurred
  return 0;
}

Finally, as a nicety, we can disable loopback so that we don’t receive our own messages. This isn’t 100% reliable, as certain network conditions can result in the local machine still receiving its outgoing messages, but it can improve efficiency:

u_char loop = 0;
if ( setsockopt(sock_fds[number_sockets], IPPROTO_IP, IP_MULTICAST_LOOP, &loop, sizeof(loop)) != 0 ) {
  // Error occurred
  return 0;
}

Now that our sockets are set up, we can prepare to send to the multicast address:

#define kMulticastAddress "224.0.0.123"
...
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = inet_addr(kMulticastAddress);
addr.sin_port = htons(kPortNumber);

And, as before, we send through each of the sockets we created:

int i;
for ( i=0; i<number_sockets; i++ ) {
  if ( sendto(sock_fds[i], data, length, 0, (struct sockaddr*)&addr, sizeof(addr)) < 0 ) {
    // Error occurred
    return 0;
  }
}

Receiving

Receiving messages from a multicast group on several network interfaces is a little more involved than doing so with broadcast: We need to subscribe to the multicast group from each network interface, explicitly. If we were to just specify no device in particular, the system would choose a single interface for us, neglecting the others.

Joining the multicast group for each interface takes place in a now-familiar loop:

const struct ifaddrs *cursor = addrs;
while ( cursor != NULL ) {
  if ( cursor->ifa_addr->sa_family == AF_INET 
          && !(cursor->ifa_flags & IFF_LOOPBACK) 
          && !(cursor->ifa_flags & IFF_POINTOPOINT) 
          &&  (cursor->ifa_flags & IFF_MULTICAST) ) {
 
    // We will do some stuff in here
 
  }
  cursor = cursor->ifa_next;
}

For each network device, we use the IP_ADD_MEMBERSHIP property with setsockopt to join — thereby subscribing to any messages sent to the multicast group address that reach that network interface.

First, we prepare the join request structure. This provides the multicast group address, and the network interface:

struct ip_mreq multicast_req;
memset(&multicast_req, 0, sizeof(multicast_req));
multicast_req.imr_multiaddr.s_addr = inet_addr(kMulticastAddress);
multicast_req.imr_interface = ((struct sockaddr_in *)cursor->ifa_addr)->sin_addr;

Now we use this structure to join the group:

if ( setsockopt(sock_fd, IPPROTO_IP, IP_ADD_MEMBERSHIP, &multicast_req, sizeof(multicast_req)) < 0 ) {
  // Error occurred
  return 0;
}

Now, a caveat: While it’s perfectly legal to join the same multicast group on more than one network interface, and up to 20 memberships may be added to the same socket (see ip(4)), for some reason, OS X spews ‘Address already in use’ errors when we actually attempt it.

As a workaround, we can ‘drop’ the membership first, which would normally have no effect, as we have not yet joined on this interface. However, it enables us to perform the subsequent join, without dropping prior memberships.

So, before we perform the above IP_ADD_MEMBERSHIP, we do a IP_DROP_MEMBERSHIP first:

setsockopt(sock_fd, IPPROTO_IP, IP_DROP_MEMBERSHIP, &multicast_req, sizeof(multicast_req));

This sets up our socket to receive messages sent to the multicast group that are received via any interface.

Here it is all put together: multicast_sample.c

Compile by opening Terminal, and typing make multicast_sample or cc -o multicast_sample multicast_sample .c, then run it with ./multicast_sample "Message to send" to send, or just ./multicast_sample with no arguments to listen.

Still to come

So, now we mostly have networking covered. There’s one obvious omission, though, for an iPhone app: Bluetooth. In Part 3, I’ll discuss how to perform communications over Bluetooth on the iPhone, in a way that’s compatible with the above generic network communications. I’ll also talk about how to connect to other devices automatically, without user intervention — This is one particularly popular feature of Talkie that allows it to behave more like a real walkie-talkie.

I promised in Part 1 that I’d talk about packet formats. We’ve covered a lot of ground in Part 2, however, so it shall be postponed to Part 3 — I’ll discuss how to ensure you get messages in the correct order by using sequence numbers, as well as providing for versioning and a few other bits and pieces.

Talkie is my newest product, the result of a collaboration with a good designer friend, Tim Churchward, who did the user interface.

Talkie is a little different from many of the other walkie talkie applications on the App Store (aside from the fact that much of it was written by me from our motorhome in Tunisia!), and I thought I’d write a little about some of the tech underpinning the app, and some of the choices we made. Along the way it may get a little tutorial-esque.

• This first part will introduce our initial motivations, and will talk about basic broadcast communications — the broadcast communications part may be very familiar to some, in which case it may be worth skipping to the next instalment.
• In the second part, I’ll continue the theme of networking, and will talk about what I ended up with for Talkie’s network code after addressing a couple of things, including switching to multicast.
• Finally, I’ll talk audio, dual platform development, and anything else I think of along the way (Actually, I’m aching to talk about one particular upcoming feature that had me jumping up and down when I first thought of it, but for now, mum’s the word on that one.)## Inspiration

Right from the start, we wanted a product that brought back the fun walkie talkie experience we remember from our childhoods. I’m talking colourful plastic, whip antennas and hiding in tree-houses. This was mostly Tim’s domain, so I shall leave it to him to discuss how he found that in the user interface.

It also meant stepping back from traditional voice chat, with manual call initialisation and termination and simple one-to-one calls. We wanted to mimic a radio, so that meant broadcasting as soon as you hit “Talk” — whereupon anyone in the neighbourhood would hear you.

Basically, we wanted to get as close to the real thing as was practical. This included the addition of a prominent ‘morse code’ button, of course, as well as a ‘squelch’ control for ‘fine tuning’, which simulated the static of bad reception.

Going dual-platform

Soon after I started developing Talkie, I realised I wanted a version for the Mac too.

Having Talkie both on the iPhone and on the Mac made sense, as we envisioned that a fairly common usage pattern would involve communication between a desktop and a handheld — say, someone wandering a campus with the iPhone in their pocket, staying in touch with friends at their desks.

We wanted to offer good value with Talkie, which is why we made Talkie for Mac available for free, when it’s used with Talkie for iPhone.

One of the very convenient things about developing for both iPhone and Mac is that the platforms are so similar, porting code is usually effortless. So, going dual-platform was an easy decision.

Finding common ground

The result of all of this was the need to develop or find a communication protocol and codec that would work across the iPhone and the Mac, over both Bluetooth for iPhone-iPhone communication, and Wifi, for communication between iPhones and iPhones, and iPhones and Macs.

Version 3.0 of the iPhone SDK introduced the GameKit framework and as part of it, GKVoiceChatService, which provides two person voice chat straight out of the box. Immediately it was clear that it wouldn’t serve our purposes — two person is not broadcast. I was also keen to provide a Mac version of Talkie, and given that GameKit is iPhone-only, it was time to go off the beaten track.

There are a variety of voice communication protocols out there, with varying licences and varying complexity and feature sets. I could’ve spent a week or two researching the options and evaluating them against our need for broadcast functionality, figuring out compilation and linking on the iPhone, resolving any compatibility or performance issues that arose, etc.

Or, I could spend a day putting together a few big building blocks provided by Apple (and the underlying Unix system) and have an easily tweakable working solution that precisely meets our needs and provides for future features.

It may not have been the most scientific approach, but the core of Talkie’s functionality was up and running on our home wi-fi network within about four hours, and it was working well.

Getting from a to b (and c, d, e, and f)

For those unfamiliar with the ins and outs of TCP/IP (the most common communication protocol — or rather, set of protocols — between computers, and the fundamental building block of the Internet), communication between computers can be connection-oriented, or connectionless.

Connection-oriented (a.k.a. ‘reliable’) communications are most common: They’re used when you open up a website, connect to iChat, or check your email. This is the TCP part of TCP/IP, and includes niceties like built-in guaranteed delivery via retransmission (providing your cat hasn’t eaten your network cable, of course) and packet ordering so you receive the messages in the right order. This only works between two computers, though – you and the server.

Connectionless (or ‘unreliable’) communications are much more open — it’s basically just a spray of messages: The data isn’t carefully ushered along, it’s just spat out and left to fend for itself. This is UDP, and is commonly used for time-sensitive applications like audio communication, network gaming, etc., where once a packet arrives late, it’s useless: No point mucking about re-sending lost data that’s going to be past its use-by date by the time it arrives. The other thing about UDP is that, because there’s no co-operation required from the destination (to acknowledge receipt, etc.), it lends itself to one-to-many communications — broadcast.

Just what we’re after.

So, we use connectionless communications (UDP) to send messages containing the audio, and anyone who receives them unpackages and plays the audio within.

The basic mechanics of this are fairly simple, once you get past the arguably cryptic C syntax.

Here’s how it works.

Sending

On the transmission end, on startup, you create a socket using the socket call:

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
...
int sock_fd = socket(AF_INET, SOCK_DGRAM, 0);
if ( sock_fd < 0 ) {
   // Error occurred
}

The socket, identified by sock_fd, will be created using the IPv4 domain (AF_INET), a ‘datagram’ type (SOCK_DFRAM: that’s connectionless communication — if we wanted connection-oriented, we could put in SOCK_STREAM here instead), using protocol 0, which is IP.

Then, we enable broadcasting:

int flag = 1;
int result = setsockopt(sock_fd, SOL_SOCKET, SO_BROADCAST, &flag, sizeof(flag));
if ( result < 0 ) {
   // Error occurred
}

That is, we set the SO_BROADCAST option at the SOL_SOCKET level to 1 (via the flag variable, which we pass in as a pointer), thereby requesting permission from the operating system kernel to broadcast.

Now we have the carrier pigeon coop at our disposal, we can start dispatching pigeons. First, we fill out the address:

#define kPortNumber 1234
...
struct sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_BROADCAST;
addr.sin_port = htons(kPortNumber);

(Note, htons converts kPortNumber into a format that can be understood by any computer, regardless of the way it represents numbers internally.)

Now, send:

NSData *dataToSend;
result = sendto(sock_fd, [data bytes], [data length], 0, (struct sockaddr*)&addr, sizeof(addr));
if ( result < 0 ) {
   // Error occurred
}

Receiving

That takes care of outgoing messages. To receive them, we want another socket:

int sock_fd = socket(AF_INET, SOCK_DGRAM, 0);
if ( sock_fd < 0 ) {
   // Error occurred
}

Now we specify where we want to receive messages from, by ‘binding’ the socket:

struct sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_port = htons(kPortNumber);
result = bind(sock_fd, (struct sockaddr*)&addr, sizeof(addr));
if ( result < 0 ) {
  // Error occurred
}

And we pass a buffer to recvfrom to fill up with tasty morsels of data:

#define kBufferSize 1024
...
char buffer[kBufferSize];
int addr_len = sizeof(addr);
int bytes_received = recvfrom(sock_fd, buffer, kBufferSize, 0, (struck sockaddr*)&addr, &addr_len);
if ( bytes_received < 0 ) {
   // Error occurred
}

Voila, we have bytes_received bytes sitting in buffer for us to do something with. Note also that addr now also contains the address of the sender. Now we can loop and continue to receive data as it comes in, passing the data off to some other part of the application to deal with.

We can put it all together into a simple test app: broadcast_sample.c

Compile it by opening up Terminal, and typing make broadcast_sample (or cc -o broadcast_sample broadcast_sample.c, if you like), then run it with ./broadcast_sample "Message to send", or just ./broadcast_sample to listen.

$ ./broadcast_sample 
Listening...
Hello world!

$ ./broadcast_sample 'Hello world!'
"Hello world!" transmitted.

Coming next

This will work happily between computers with just one network interface. But, if you have more than one (say, wireless, and an Ethernet connection too), you’ll notice that it will only communicate through one of the interfaces. That’s because just the default interface is used. You have to explicitly attend to each interface, to broadcast out of each one, and listen on each one.

In the next part of this series, I’ll write about how I addressed that, and about multicast, which is used in things like Bonjour (MDNS). I’ll also write about designing packet formats.

Thanks for reading!

Thinking it’s better to keep things in the family, and take better advantage of that traffic, I put together an image hosting setup of my own. Now viewers come to my own site, instead of someone else’s!

It looks like this

The setup consists of an Automator service, and some PHP smarts on the site.

Under Snow Leopard, I can right-click on an image in Finder, click ‘Send Image to Image Hosting’, type in a title, then paste the URL that’s automatically been put onto my clipboard into a tweet.

Here’s how it works – this assumes a moderate level of understanding of PHP and assorted web administration. Update: I couldn’t help myself; I made a WordPress plugin for this, with some extra nice features.

Presentation

First, we need to put together a script that will display the image. What’s more, it would be nice to give the image a title, and link it back to the Twitter account.

So, we’ll want a place to put images, and a way to store the corresponding title and account name.

I opted to store the latter in a file alongside the image called (imagename).meta. I store the title, then a newline, then the Twitter account name.

Lets create image.php — it should take an image name parameter, excluding the file extension, and display the corresponding image, with any corresponding meta data.

Here’s a sample:

<?php
$MEDIA_DIR = "twitter-images";
 
// Get the image name, but avoid any path shenanigans by just grabbing the 'basename' of the file
$image = basename($_REQUEST['name']);
 
// Make sure it exists, or report a 404 error
if ( !$image || !is_file("$MEDIA_DIR/$image.jpg") ) {
    include("404.php");
    return;
}
 
// Grab the meta data
if ( file_exists("$MEDIA_DIR/$image.meta") ) {
    list($title, $source) = file("$MEDIA_DIR/$image.meta");
}
 
// Set a default title if we don't have any meta data
if ( !$title ) {
    $title = ucwords(strtolower($image));
}
 
// Include the page header, if you like
include("header.inc.php");
?>
 
<div class="content">
 
    <img src="/<?php echo MEDIA_DIR ?>/<?php echo $image ?>.jpg" alt="<?php echo htmlspecialchars($title) ?>" class="image" />
 
    <h1><?php echo $title ?></h1>
 
    <?php if ( $source ) : ?>
        <p>
            From <a href="http://twitter.com/<?php echo $source ?>">
                <img src="/media/images/twitter.png" width="14" height="14" alt="Follow us on Twitter!">
                <?php echo $source ?>
            </a>
        </p>
    <?php endif; ?>
 
</div>
 
<?php include("footer.inc.php"); ?>

Make sure it works by making a sample image called, say, IMG.jpg, and an accompanying metadata file, called IMG.meta with a title on the first line and a twitter account name on the second, and opening up http://yoursite.com/image.php?name=IMG

Now, we want to keep URLs short, so they don’t take up too much valuable space in a tweet, so we need to use some tricks here.

We’ll set up a .htaccess file that uses mod_rewrite to keep the URL short.

Something like this should do the trick:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} ^/([^/]+)$
RewriteCond %{DOCUMENT_ROOT}/twitter-images/%1.jpg -f
RewriteRule (.*) /image.php?name=$1 [L]
</IfModule>

Note on the fifth line we are checking inside our Twitter images folder — make sure that matches your storage path.

This will take any requests that:

• Are located in the web-root (start with a slash, followed by anything but a slash until the end)
• Have a corresponding file within the directory we’ll store images in

Then, it’ll rewrite the request silently to use image.php.

Save the .htaccess file, and open http://yoursite.com/IMG. It should bring up the image, just as before.

Set up some CSS to make it look good, and you’re set. Here’s mine:

 
/* @group Twitter images */
 
.twitter-image .content {
	margin-top: 50px;
	margin-bottom: 50px;
}
 
.twitter-image .content img.image {
	-moz-border-radius: 20px;
	-webkit-border-radius: 20px;
	background: #f1f1f1;
	border: 1px solid #aaa;
	padding:  40px;
	margin-right: 60px;
	display: block;
	margin: 0 auto;
}
 
.twitter-image .content h1 {
	padding-top: 60px;
	text-align: center;
}
 
.twitter-image .content p {
	text-align: center;
}
 
.twitter-image .content a {
	text-decoration: none;
}
 
.twitter-image .content a img {
	vertical-align:text-bottom;
}
 
/* @end */

Automator

Now, with the web side set up, we need to set up an easy way to send content there.

I opted for an Automator service, which uses scp to upload the content. If your web host doesn’t provide for scp, you may want to use something else like ncftpput, or even an Applescript-able application. You’ll need to modify the Automator action’s shell script accordingly.

Here’s the Automator service I set up: Send Image to Image Hosting.zip.

For those interested, it contains the following shell script:

APPSUPPORT=~/"Library/Application Support/Image Hosting"
TMPDIR='/tmp/image-hosting.tmp'
DEST='me@mywebsite.com:www/mywebsite/twitter-images/'
URLPREFIX='http://mywebsite.com/'
UPLOADRETRIES=3
# GROWL="/usr/local/bin/growlnotify" # Uncomment this if you have Growl and growlnotify
 
[ ! -e "$APPSUPPORT" ] && mkdir "$APPSUPPORT"
 
for f in "$@"
do
  if [ ! -e "$f" ]; then
    osascript -e 'tell application "Finder"' -e 'activate' -e 'display dialog "Input file '"$f"' does not exist" with title "Image Hosting" buttons {"OK"} with icon Stop' -e 'end tell'
    exit
  fi
 
  # Generate random name
  while true; do
      name=`perl -e 'use MIME::Base64; $s=encode_base64(rand()); $s =~ s/=|\n//g; $s = substr($s, -5); print $s'`
      if ! grep "$name" "$APPSUPPORT/Taken Names" &>/dev/null; then break; fi;
  done;
 
  # Gather info
  default=`php -r '$s = basename("'"$f"'"); echo ucwords(strtolower(substr($s, 0, strpos($s, "."))));'`
  title=`osascript -e 'tell application "Finder"' -e 'activate' -e 'text returned of (display dialog "Enter image title" with title "Image Hosting" default answer "'"$default"'" buttons {"Cancel", "Continue"} default button 2)' -e 'end tell'`
  [ ! "$title" ] && exit
  defaulttwitter=`cat "$APPSUPPORT/Default Twitter Account" 2>/dev/null`
  twitter=`osascript -e 'tell application "Finder"' -e 'activate' -e 'text returned of (display dialog "Enter Twitter account" with title "Image Hosting" default answer "'"$defaulttwitter"'" buttons {"Cancel", "Continue"} default button 2)' -e 'end tell'`
  [ ! "$twitter" ] && exit
  echo $twitter > "$APPSUPPORT/Default Twitter Account"
 
  # Create temp files
  [ -e "$TMPDIR" ] && rm -rf "$TMPDIR"
  mkdir "$TMPDIR"
  cp "$f" "$TMPDIR/$name.jpg"
  echo "$title
$twitter" > "$TMPDIR/$name.meta"
 
  [ -f "$GROWL" ] && $GROWL -n 'Image Hosting' "Image Hosting: `basename "$f"`" -m "Image Hosting is uploading '$name'"
 
  # Upload
  retries=0
  while [ $retries -lt $UPLOADRETRIES ]; do
    result=`scp "$TMPDIR"/* "$DEST" 2>&1` && break;
    [ -f "$GROWL" ] && $GROWL -n "Image Hosting: `basename "$f"`" 'Image Hosting' -m 'Upload failed: Retrying'
    retries=$((retries+1))
  done
 
  if [ $retries -ge $UPLOADRETRIES ]; then
    osascript -e 'tell application "Finder"' -e 'activate' -e 'display dialog "Could not upload image.  Scp said: '"$result"'" with title "Image Hosting" with icon Stop' -e 'end tell'
    exit
  fi
 
  echo $name >> "$APPSUPPORT/Taken Names"
 
  osascript -e 'tell application "Finder" to set the clipboard to "'"$URLPREFIX$name"'"'
 
  [ -f "$GROWL" ] && $GROWL -n 'Image Hosting' "Image Hosting: `basename "$f"`" -m "Upload complete.  Available at $URLPREFIX/$name (copied to clipboard)"
done

To use it:

1. Unzip it,
2. Put it into Library/Services,
3. Double-click on it to open it in Automator,
4. Change the settings at the top of the ‘Shell Script’ segment —
   • Set DEST to the scp destination
   • Set up key-based SSH authentication, so you don’t have to enter the password to upload, or, if you’d prefer to use something other than scp to upload the file, you can change the upload command further down in the script, underneath the ‘Upload’ comment (look for scp)
   • Set URLPREFIX to http://yoursite.com
   • If you have Growl and growlnotify installed, then set GROWL to the location of growlnotify (highly recommended)
5. Save

Usage

If you’re using Leopard, you may need to log out, then log back in to see the service — I’m not entirely certain. If you’re on Snow Leopard, you should hopefully be set. Right-click on an image file in Finder, and click ‘Send Image to Image Hosting‘.

You should be asked for a title, then asked for a Twitter account name (this will be remembered for next time), then the upload will happen, and when it completes, the URL to the uploaded image will be put onto the clipboard so you can paste straight into Twitter.

Voila.

Access it by hitting Cmd-Shift-G in Finder, typing in /private/var/folders and hitting enter, then navigate around until you see something like a TemporaryItems folder – within will be a file called something like FlashTmp0, which is the .flv file.

Copy it somewhere and rename it, and it’ll play in VLC or any other player that supports the format (like Quicktime etc with Perian installed).

iPhoto ’09 introduced Flickr support, so that you can post photos and albums to your Flickr account. Unfortunately for some, it has some issues: A tendency to perform mass-deletions on uploaded photosets after the initial upload. For example, for me, having uploaded several hundred images, iPhoto has deleted almost all of them on two separate occasions, requiring the photosets to be constructed from scratch.

For those who were considering using Flickr from iPhoto, don’t, and save yourself the hassle (use FlickrExport instead). For those who have and are seeking a remedy to keep iPhoto away from Flickr, read on.

I encountered the uncomfortable scenario where iPhoto had forgotten about three quarters of the images in the uploaded photosets, and upon launch, looked like it was going to delete the remainder from my Flickr account. Rather than let this happen, I force quit iPhoto, and performed the following steps to force iPhoto to forget about my Flickr account. This is a very complicated process, not for the faint of heart, but it was necessary for me to save my Flickr account:

1. Right-click on the ‘iPhoto Library’ package, probably in the ‘Pictures’ folder
2. For the files AlbumData.xml and AlbumData2.xml:
   1. Backup the files somewhere
   2. Open the file in a text editor
   3. Search for the term ‘http://www.flickr.com’
   4. Wherever this is found, delete the surrounding text between <dict> and </dict>. There will be one of these dict structures for every photoset
   5. Save the file
3. Now the tricky part: Find an SQLite editor (I used Froq) and open up iPhotoMain.db within the ‘iPhoto Library’ package. To access the file, I had to copy it to my Desktop first, then open it there (back it up first).
   1. Click on the SqPublishedAlbum table to see the contents. This lists all of the published albums, including Flickr, Facebook, etc. Find the entries with a publishedURL at flickr.com, and delete them. In Froq, I had to take note of their primary key values, and manually execute an SQL query (DELETE FROM SqPublishedAlbum WHERE primaryKey = 12345). Remember those primary key values, because you’ll need them.
   2. Click on the SqAlbum table, and delete the albums with the same primary key values – if you used an SQL query as above, you can just re-use the same query but change the table name to SqAlbum.
   3. Repeat the same process for the AlbumsPhotosJoin table – in this case, you’ll need to delete entries with the sqAlbum field values matching the prior primarykey values. Something like DELETE FROM AlbumsPhotosJoin WHERE sqAlbum = 12345 will do it.
   4. Done with this database – commit and close, but remember the primary key values for the Flickr albums.
4. Next, open up the other SQLite database, iPhotoAux.db, after backing it up
   1. Delete the Flickr album records from the SqAlbumSubclasses table. Something like: DELETE FROM SqAlbumSubclasses WHERE primaryKey = 12345.
   2. Commit and close
5. Send a bug report to Apple

That’s it! Now, after starting iPhoto, the Flickr albums should be gone, safe at last. FlickrExport is a £12 plugin that provides better, more reliable functionality than iPhoto ’09.