Karl von Randow

Fix missing eth0 after changing network cards

Tue, 24 Jan 2012 08:11:48 +1300

After changing hard drives out of one server machine into another and booting, the network interface didn’t come up correctly. eth0 didn’t exist and gave an error when I tried to start it manually using ifup eth0. At first I suspected that the correct driver module wasn’t installed because the OS had been installed on another system with a different NIC; so I verified that the correct module was installed.

What in fact had happened is that udev has a feature where it remembers NICs by their MAC address and remembers which device name they’ve been assigned. So the hard drive (and the OS it contains) were running in a machine with another NIC that had been assigned to eth0, so when booted inside the new machine it assigned the new NIC to eth1. Unexpected!

This would often be pretty fantastic. Many years ago when I upgraded from a 2.4 to a 2.6 kernel I had an issue where the order of network devices changed - eth0 and eth1 had swapped. This caused some frustration when remote administering the upgrade from another country on Christmas day.

So you can either take what you’ve been given and change your network interface configuration to use eth1. Which wouldn’t be stupid. Or you can modify the udev rules to forget the original NIC and therefore assign eth0 to your new NIC on reboot…

Edit the /etc/udev/rules.d/70-persistent-net.rules file. At or near the bottom will be lines beginning SUBSYSTEM, which declares the MAC address and the assigned device name. Comment out or delete lines for NICs you no longer have. Then either comment out or delete your current NIC (otherwise it will stay with eth1 after reboot) or simply change the device name it has been assigned to eth0. Then reboot.

I had to discover this in a time critical situation - thank goodness for Google and for this post.

Restore the Linux console messages on boot

Tue, 24 Jan 2012 07:59:12 +1300

When I installed Ubuntu Lucid on a physical server for the first time I was unnerved by seeing a startup progress indicator when I’m used to seeing all the console messages scrolling by. If something is taking a long time, I want to see what it is. I know you can switch to VT-7, but I want them back on the main screen where they’re easy to see (and easy for someone else to see if I’m calling on the phone in an emergency).

It’s of course a simple configuration change.

Edit /etc/default/grub and change GRUB_CMDLINE_LINUX_DEFAULT. For the progress indicator it’s set to “quiet splash”. For the console messages just change that to “”. Source.

Installing 32 bit PostgreSQL on 64 bit Linux

Tue, 24 Jan 2012 07:58:52 +1300

During an emergency server upgrade last week I was faced with a problem. I’d installed 64 bit Linux to replace a 32 bit Linux box, copied across the PostgreSQL data directory, then installed the identical version of PostgreSQL, but 64 bit. It turns out that the data file format is incompatible between 32 and 64 bit PostgreSQL.

PostgreSQL is my favourite RDBMS, but it’s worth noting that MySQL had no such problem and automatically updated the data files. Thanks MySQL!

The 32 bit server install was dead, so I couldn’t do a dump from the 32 bit system. I needed to install 32 bit PostgreSQL on my 64 bit box.

First you need to install ia32-libs using apt-get install. This gets you the base libraries necessary to run 32 bit apps.

Then you need to manually download the 32 bit .deb files for PostgreSQL. You’ll find them under /ubuntu/pool/main/p/ in your local Ubuntu archive. Try doing an install of PostgreSQL as per normal to see which packages are required. The postgresql-common packages are architecture independent so you can install those using apt-get.

After you’ve downloaded the .deb files you must install them using dpkg -i —force-architecture .

After doing this I was able to start PostgreSQL. It said it failed during startup! But it did actually start. I wasn’t able to use psql, as the libreadline library wasn’t 32 bit, but I was able to use pg_dump, which was the critical bit.

Once I’d dumped the databases I quickly uninstalled the 32 bit PostgreSQL, moved the data directory and installed the 64 bit version and sat through the lengthy reload.

Charles 3.6.5 released

Thu, 08 Dec 2011 19:28:00 +1300

Charles 3.6.5 was released earlier today. It’s primarily a bug fix release.

For Java 1.7 a number of incompatibilities were fixed including a delay during startup and hung network connections.
Fixes for a networking regression in 3.6.4 - an improvement to the networking caused an irregular Connection reset (or similar) error to occur when browsing.
SSL behaviour improved including better error messages in the browser and fixing a regression in 3.6.4 that caused empty requests to appear.

The next version of Charles will be 3.7 and will include more advanced timing of DNS lookups, TCP connections and SSL handshakes, as well as improved kept-alive connection reporting so you can see when new HTTP connections are established. The first beta of 3.7 is available today.

Recovering lost Git commits

Wed, 07 Dec 2011 21:32:35 +1300

If you rollback to a previous commit you can lose the later commits if they’re not referenced by another branch. I just rolled back and lost a commit that I hadn’t pushed to the remote.

Easy fix.

git reflog

This will show you recent commits, and should include the commit that you’ve just lost. You can then merge it back in.

Migrating from SVN to Git

Wed, 30 Nov 2011 23:13:00 +1300

I’ve migrated several projects from SVN to Git in the last couple of weeks. In doing so I’ve worked out a little process that has seen me right in preserving all of the history. This process relies on the git-svn tool, which you may need to install as a separate package.

1) Create an authors.txt file. This is a file that maps the SVN usernames to the more verbose Git author names. I suggest starting with an empty file and adding to as errors crop up due to missing authors during the import. Alternatively you may be more organised.

Your authors.txt file will look something like this:

username = Full Name 
username2 = Full Name

2) Import the SVN repository into a new Git repository. Note that this command references the authors.txt file you created in step 1, it includes the username (if any) you require to access the SVN repository, the URL to the repository and the directory to create the new Git repository in. This also assumes that you use a standard trunk, branches, tags layout (if not, omit the -s), and that your URL will be the root of the repository (ie. the directory containing trunk). At any stage consider typing “git svn help” to get assistance.

git svn -s --authors-file=authors.txt --prefix=svn/ --username=svn_username \
clone https://url/path/to/svn/repo/ directory-for-git-repo

This process will take a while. It will fail with an error if it encounters a username that isn’t in your authors.txt file. Add the author and then re-run the command; it will continue from where it left off.

Once it’s complete, consider running it again just in case. My first attempt said it was finished and it wasn’t.

You now have a working Git repository. The SVN repository is hooked up as a remote and all of its branches and tags are prefixed with svn/, which makes them easy to find. The next step is to turn the SVN branches and tags into local Git branches and tags to complete the import.

3) Make local branches for each remote SVN branch. When we do the import we specify that all of the SVN branches go into the prefix svn/. I wrote a brief shell script to perform this step; all it does is to list the remote svn branches (omitting the tags and trunk), and then checkout each one-at-a-time to a local branch.

cd directory-for-git-repo
for i in `git branch -r --no-color | grep 'svn/' | grep -v 'svn/tags/' | grep -v 'svn/trunk' `
do echo $i
    git checkout -b `echo $i | sed -e 's|svn/||'` $i
done

4) Make local tags for each remote SVN tag. SVN tags are imported as remote branches, so as for the branches, we run a short script to create local Git tags for each SVN tag.

for i in `git branch -r --no-color | grep 'svn/tags/' | sed -e 's|svn/tags/||'`
do echo $i
    git checkout -b tag_branch svn/tags/$i
    git checkout master
    git tag `echo $i | sed -e 's|svn/||'` tag_branch
    git branch -D tag_branch
done

5) Make the master branch from the SVN trunk. I found that my master branch was a superset of every other branch after the git svn import, so in this step we rename the existing master branch, create a new one, and then delete the old one.

git branch -m master oldmaster
git checkout -b master svn/trunk
git branch -D oldmaster

And that completes my import process. The next step is probably to clone the repository, or possibly to push it to its new host. Cloning gets rid of all of the git svn configuration and remote branches and leaves you with a clean Git repository.

EGit fails to import an existing Git working copy

Wed, 30 Nov 2011 22:44:38 +1300

I used another Git tool to clone my git repository into my development directory, which is also my workspace directory in Eclipse. I then wanted to add this working copy as a project in Eclipse; it has a .project file as it’s already an Eclipse project (previously in an SVN repository).

Using the File > Import menu option and choosing Projects from Git, I was able to add my Git repository (which is inside the working copy), but when I go through to import existing projects it fails with an error:

Invalid project description.
/Users/karlvr/Development/projectname overlaps the location of another project: 'project name'

This turns out to be because importing projects from Git wants to make a new working copy in your workspace, much like with SVN where the wording was more obvious to me “Checkout Projects from SVN”

What I actually wanted to do, and what worked, was to import the project as an existing project and then activate it with Git. To do this, choose File > Import, then Existing Projects into Workspace. Choose your project directory, probably already located in your workspace, and import it. Right-click on the imported project in the Project Explorer and choose Team > Share Project. Choose Git as the repository type, then in the Configure Git Repository dialog choose “Use or create repository in parent folder of project”. EGit will find the .git folder at the root of your project and your project will now be in Eclipse and setup for EGit.

Optimising images / Optimizing images

Sat, 26 Nov 2011 10:30:00 +1300

Tomato tomato. If you’re putting images into an application, especially a mobile application, or on the web, or anywhere, it’s a good idea to optimise them before you deploy. In an iOS application the larger your application is the longer it takes to download and install. If you go over the magic 20MB limit, users aren’t even allowed to download it over 3G; they have to seek out a WiFi network. This doesn’t feel like it would be good for sales.

Xcode runs pngcrush on your PNG files, which does an okay job. It doesn’t touch the JPGs, and overall it turns out you can do better.

Enter ImageOptim. A free Mac OS X app (unfortunately not available via the Mac App Store due to licensing) that combines a series of command-line image optimisation tools; runs them all and chooses the best optimisation for each image. It also finds images recursively in folders, so I dragged and dropped the root Images folder to do a whole application.

The optimisations are lossless, so it doesn’t downgrade the quality of your images (this is something you should also experiment with separately, especially JPEG quality). The tools remove metadata and other information that you don’t need; they can also recompress or optimise the compression to gain a few more bytes.

There’s one extra optimisation tool that ImageOptim can use but isn’t bundled: PNGOUT. There are instructions for downloading PNGOUT on the ImageOptim webpage. I recommend getting it and configuring the path to it in the ImageOptim preferences; it often got the best results on my PNGs.

Thanks to Corwin Derkatch for putting me on to ImageOptim.

The coolest thing ever

Sat, 26 Nov 2011 10:13:08 +1300

It’s fair to say that when you enter the office, it is criminal not to have a short intro theme blasted out on the stereo for all to enjoy. What an entrance. It sets the tone. It sets the mood. And the ladies love it.

Fortunately, thanks to a confluence of technology, we blew this case wide open last week at Cactuslab. We now have the DHCP server running a script when known devices join the network which plays personalised MP3 files via the Airport Express to the office stereo at ear-popping volume. When you walk into the office with your WiFi device in your pocket… boom.

The first piece of the puzzle is integrating with the DHCP server. I run our office’s DHCP server on a Linux server, and it turns out to be quite easy to execute a script when the DHCP server hands out a new lease. The DHCP server executes a shell script, which takes as an argument the hardware address of the new device.

on commit {
    set clip = binary-to-ascii(10, 8, ".", leased-address);
    set clhw = binary-to-ascii(16, 8, ":", substring(hardware, 1, 6));
    execute("/usr/local/sbin/dhcpevent", "commit", clip, clhw);
}

The script looks for an MP3 file matching the hardware address in a folder on the file server. If an MP3 is found the script attempts to play it on the office stereo using raop_play, which connects to the Airport Express via AirPlay. Bob’s your uncle.

#!/bin/sh

HARDWARE=`echo "$3" | sed -e 's/:/-/g'`
INTROSDIR=/data/cactuslab/share/intros
AIRPORT_EXPRESS="10.1.10.135"

INTRO="$INTROSDIR/$HARDWARE.mp3"
if [ -f $INTRO ]; then
	raop_play $AIRPORT_EXPRESS $INTRO &
fi

There are a few problems with this setup. Most annoyingly, if someone is already playing something on the stereo then your intro can’t play. We’re resolving this issue by adding a second, dedicated, Airport Express. The two Airport Expresses will run through a mixer. I think we’ll set the mix on the intros Express slightly higher, you know, so you can really feel it. I’ll update on the success of this later.

The biggest problem, however, is that it takes a while for your iPhone to jump onto the WiFi network. So sometimes it’s less of an intro and more of a reminder that you’ve already arrived. It’s still pretty cool. I have a feeling there’s a déjà vu issue when a device rejoins the network… but that could be changed with some more clever scripting.

I recommend keeping the intros to about 15 seconds and adding a fade-out to the end of the MP3.

Keep it classy, and please let me know if you attempt this!

Charles 3.6.4

Wed, 16 Nov 2011 22:21:00 +1300

Charles 3.6.4 was released this week. It includes a number of bug fixes which are outlined in the version history. If you had trouble with Charles sometimes starting with a blank window, or freezing while using breakpoints, this might be the bug fix release for you. If you are interested in the HAR import/export capabilities, this is definitely the release for you.

I also want to draw attention to the PCAP import. You can capture traffic in a tool, such as Wireshark, and then save that traffic in a .pcap file and then import it into Charles. Charles analyses the HTTP traffic and pieces it back together as if Charles had been there to intercept it. This can be an option for capturing and analysing traffic in Charles when you can’t get an application to directly use Charles. Aside from bug fixes, Charles 3.6.4 also improves the PCAP import - ensuring that keep-alive requests and imported and putting things in the right order.

The other thing I want to highlight is Linux support. Charles has supported Linux since very early on in its life, but it’s never done a particularly good job at installing and running nicely as part of a Linux desktop. This is no doubt my fault as the vast majority of my Linux boxes are command-line only. Charles 3.6.4 features an improved start script for Linux, bigger icons, and most excitingly an APT package for Debian-based Linux distributions. I’ve particularly tested it on Ubuntu: the package installs Charles into your applications menus, and the package makes the process of getting new versions of Charles much easier. If you’re a Linux user who can’t use APT packages, get in touch and we’ll see if we can make other package types!

The next release of Charles is going to include more bug fixes and minor enhancements; I’m particularly focussing on fixing a few issues around Java 1.7 so if you’re having trouble with 1.7 please get in touch and try the latest beta.

Charles 3.6.4 and HAR (HTTP Archive)

Wed, 16 Nov 2011 22:12:05 +1300

Charles 3.6.4 is another minor update to the 3.6 release; it features a large number of bug fixes, including several improvements to the HAR import/export implementation. We introduced HAR import/export to Charles 3.6 and it has just gotten better (or really, in the case of the bugs, less broken). Thanks to everyone who has corresponded with me about particular issues (especially Alex Selivanov).

HAR is an archive format for HTTP requests and responses for interoperability between software packages. Adding HAR support to Charles makes it possible to capture requests in another application and import them into Charles for analysis, or to capture in Charles and export.

I’m on the lookout for interesting analysis applications that import HAR files… drop me an email if you have any you use.

Fix a sticky iPhone home button

Mon, 14 Nov 2011 13:47:00 +1300

I’ve had a sticky home button on my old iPhone 4 for many months. It wouldn’t consistently double-tap to bring up the multitasking menu, and it caused me to develop a deadly pincer-grip that made my new 4S squeal.

Yesterday, by chance, I saw my friend Tim Haines tweet about taking his wife’s iPhone into an Apple Store, where the genius went mad on the home button with an alcohol wipe. This afternoon I went to the chemist and purchased four alcohol wipes (skin cleansing alcohol wipes) that come in little paper sachets. I took one, and wiped as hard as I could on and around the home button, being sure to press it hard and long, rolling around on the button, letting the alcohol from the wipe seep into the cracks. Continue for 30 seconds.

Happily, it seems to have worked. I can now double-tap the home button consistently, and at NZ$0.20 a wipe I think it’s a no-brainer.

High bandwidth high latency performance tuning

Mon, 14 Nov 2011 10:29:00 +1300

We recently upgraded our office Internet connection to 20Mbps fibre. Interestingly, for local traffic we were seeing excellent speeds, but international connections were disappointing (1 - 3 Mbps).

In NZ we have quite a high ping to the US (where we’re primarily connecting to), and this requires a high TCP window size to be able to have sufficient data in the pipe to take advantage of the full capacity.

Our office has both Mac OS X machines and Windows 7 machines. I’ve been unable to solve the issue on the Windows 7 machines, they are currently capped at 1 Mbps.

For Mac OS X machines it’s easy to change the appropriate parameters temporarily.

First your need to calculate the appropriate value for your TCP window. Take your network speed in Mbps (eg. 20Mbps) and your typical round-trip-time in seconds (eg. 0.3s) and multiply them then divide by 8 (eg. 20,000,000 * 0.3 / 8) to get a reasonable TCP buffer size. For my example this is 750,000 which is substantially more than the default 65,536.

Try this in the Terminal (substituting your TCP window size from the calculations above):

sudo sysctl -w net.inet.tcp.sendspace=750000
sudo sysctl -w net.inet.tcp.recvspace=750000

Test it to see if you can notice a difference. We could particularly see the difference in uploads. Then to permanently enable it, edit (or create) /etc/sysctl.conf (you’ll need to sudo this) and append:

net.inet.tcp.sendspace=750000
net.inet.tcp.recvspace=750000

Thanks to Layton Duncan for confirming this was possible on Mac OS X when I was losing faith on Windows 7.

I’m still looking for how to solve this issue on Windows 7. Suggestions welcome.

For reference reading:

For bandwidth testing I used iperf on Linux, Windows and Mac OS X (jperf).

iOS 5: turn off multiple message reminders

Sat, 12 Nov 2011 10:35:18 +1300

When I hear my phone beep I read the message on the lock screen and then put it back in my pocket. I don’t need it to beep again. After upgrading to iOS 5 it’s a bit harder to find where to turn the multiple message reminders off.

To turn off multiple message reminders (iMessage) on iOS 5 open the Settings app. Choose Notifications. Scroll down and choose Messages. Scroll down and choose the Repeat Alert setting. Change this according to your attention level.

New blog

Sun, 30 Oct 2011 15:52:00 +1300

I’ve consolidated my old blog and my oldest blog into one new blog. I may post more often. I’ll definitely, maybe, post Charles Proxy release notes. I’ll sometimes post tips and tricks so that I can find them again.

Debugging Core Data SQL on iOS

Wed, 27 Jul 2011 14:16:00 +1200

You can pass a launch argument to your application from Xcode to enable debugging of SQL statements and Core Data timing to the console log.

In Xcode 4, edit your Schemes. Go to the Run phase, Arguments tab, and in the Arguments Passed On Launch area add:
-com.apple.CoreData.SQLDebug 1

You can tick this on and off as you like so you don’t have to delete it and won’t have to remember it!

Italy

Wed, 04 May 2011 11:26:00 +1200

Italy:

joshuabrown:

Check out the book I just published of my recent trip out to Italy. Every photo was shot on the iPhone 4 using Camera+. Get it here http://campl.us/

PostgreSQL sizes of tables

Tue, 08 Mar 2011 12:38:00 +1300

select tablename, pg_size_pretty(pg_relation_size(tablename)) as data, pg_size_pretty(pg_total_relation_size(tablename)) as total from (select tablename from pg_tables where schemaname = current_schema()) as foo order by pg_total_relation_size(tablename) desc;

Flickruppa • iPhoto export to Flickr. Simplified.

Wed, 10 Nov 2010 14:52:00 +1300

Flickruppa • iPhoto export to Flickr. Simplified.:

darren131:

If you use iPhoto 8+ and Flickr then you should check out this awesome plugin. Allows for simple export from iPhoto to Flickr (including EXIF, background uploading, GPS location, videos, sets, etc, etc, etc…) Brought to you by the same madman who built Charles, Birdbrain, Mobile Fotos, etc

Simulate memory warnings on the iPhone

Sun, 07 Nov 2010 11:46:00 +1300

When debugging on the simulator you can simulate a memory warning in your code by using the Simulate Memory Warning option in the Hardware menu. But on the device it isn’t so easy, but it’s really important to have thoroughly tested your application in the presence of memory warnings.

What I do is add some code to my App Delegate to periodically send a memory warning automatically. Usually every 5 seconds, or quicker.

- (BOOL)application:(UIApplication *)pApplication didFinishLaunchingWithOptions:(NSDictionary *)pLaunchOptions {
...
    [NSTimer scheduledTimerWithTimeInterval:5 target:self selector:@selector(simulateMemoryWarning) userInfo:nil repeats:YES];
...
}

- (void)simulateMemoryWarning {
    [[NSNotificationCenter defaultCenter] postNotificationName:UIApplicationDidReceiveMemoryWarningNotification object:[UIApplication sharedApplication] userInfo:nil];
}

I suggest surrounding the timer scheduling with an #if DEBUG or similar (assuming you have DEBUG=1 configured in your debug builds) so you don’t accidentally leave this in for a release build!