Fix missing eth0 after changing network cards
After changing hard drives out of one server machine into another and booting, the network interface didn’t come up correctly. eth0 didn’t exist and gave an error when I tried to start it manually using ifup eth0. At first I suspected that the correct driver module wasn’t installed because the OS had been installed on another system with a different NIC; so I verified that the correct module was installed.
What in fact had happened is that udev has a feature where it remembers NICs by their MAC address and remembers which device name they’ve been assigned. So the hard drive (and the OS it contains) were running in a machine with another NIC that had been assigned to eth0, so when booted inside the new machine it assigned the new NIC to eth1. Unexpected!
This would often be pretty fantastic. Many years ago when I upgraded from a 2.4 to a 2.6 kernel I had an issue where the order of network devices changed - eth0 and eth1 had swapped. This caused some frustration when remote administering the upgrade from another country on Christmas day.
So you can either take what you’ve been given and change your network interface configuration to use eth1. Which wouldn’t be stupid. Or you can modify the udev rules to forget the original NIC and therefore assign eth0 to your new NIC on reboot…
Edit the /etc/udev/rules.d/70-persistent-net.rules file. At or near the bottom will be lines beginning SUBSYSTEM, which declares the MAC address and the assigned device name. Comment out or delete lines for NICs you no longer have. Then either comment out or delete your current NIC (otherwise it will stay with eth1 after reboot) or simply change the device name it has been assigned to eth0. Then reboot.
I had to discover this in a time critical situation - thank goodness for Google and for this post.
Restore the Linux console messages on boot
When I installed Ubuntu Lucid on a physical server for the first time I was unnerved by seeing a startup progress indicator when I’m used to seeing all the console messages scrolling by. If something is taking a long time, I want to see what it is. I know you can switch to VT-7, but I want them back on the main screen where they’re easy to see (and easy for someone else to see if I’m calling on the phone in an emergency).
It’s of course a simple configuration change.
Edit /etc/default/grub and change GRUB_CMDLINE_LINUX_DEFAULT. For the progress indicator it’s set to “quiet splash”. For the console messages just change that to “”. Source.
Installing 32 bit PostgreSQL on 64 bit Linux
During an emergency server upgrade last week I was faced with a problem. I’d installed 64 bit Linux to replace a 32 bit Linux box, copied across the PostgreSQL data directory, then installed the identical version of PostgreSQL, but 64 bit. It turns out that the data file format is incompatible between 32 and 64 bit PostgreSQL.
PostgreSQL is my favourite RDBMS, but it’s worth noting that MySQL had no such problem and automatically updated the data files. Thanks MySQL!
The 32 bit server install was dead, so I couldn’t do a dump from the 32 bit system. I needed to install 32 bit PostgreSQL on my 64 bit box.
First you need to install ia32-libs using apt-get install. This gets you the base libraries necessary to run 32 bit apps.
Then you need to manually download the 32 bit .deb files for PostgreSQL. You’ll find them under /ubuntu/pool/main/p/ in your local Ubuntu archive. Try doing an install of PostgreSQL as per normal to see which packages are required. The postgresql-common packages are architecture independent so you can install those using apt-get.
After you’ve downloaded the .deb files you must install them using dpkg -i —force-architecture <debs>.
After doing this I was able to start PostgreSQL. It said it failed during startup! But it did actually start. I wasn’t able to use psql, as the libreadline library wasn’t 32 bit, but I was able to use pg_dump, which was the critical bit.
Once I’d dumped the databases I quickly uninstalled the 32 bit PostgreSQL, moved the data directory and installed the 64 bit version and sat through the lengthy reload.
Charles 3.6.5 released
Charles 3.6.5 was released earlier today. It’s primarily a bug fix release.
- For Java 1.7 a number of incompatibilities were fixed including a delay during startup and hung network connections.
- Fixes for a networking regression in 3.6.4 - an improvement to the networking caused an irregular Connection reset (or similar) error to occur when browsing.
- SSL behaviour improved including better error messages in the browser and fixing a regression in 3.6.4 that caused empty requests to appear.
The next version of Charles will be 3.7 and will include more advanced timing of DNS lookups, TCP connections and SSL handshakes, as well as improved kept-alive connection reporting so you can see when new HTTP connections are established. The first beta of 3.7 is available today.
Recovering lost Git commits
If you rollback to a previous commit you can lose the later commits if they’re not referenced by another branch. I just rolled back and lost a commit that I hadn’t pushed to the remote.
Easy fix.
git reflog
This will show you recent commits, and should include the commit that you’ve just lost. You can then merge it back in.
Migrating from SVN to Git
I’ve migrated several projects from SVN to Git in the last couple of weeks. In doing so I’ve worked out a little process that has seen me right in preserving all of the history. This process relies on the git-svn tool, which you may need to install as a separate package.
1) Create an authors.txt file. This is a file that maps the SVN usernames to the more verbose Git author names. I suggest starting with an empty file and adding to as errors crop up due to missing authors during the import. Alternatively you may be more organised.
Your authors.txt file will look something like this:
username = Full Name <email@example.com>
username2 = Full Name <email@example.com>
2) Import the SVN repository into a new Git repository. Note that this command references the authors.txt file you created in step 1, it includes the username (if any) you require to access the SVN repository, the URL to the repository and the directory to create the new Git repository in. This also assumes that you use a standard trunk, branches, tags layout (if not, omit the -s), and that your URL will be the root of the repository (ie. the directory containing trunk). At any stage consider typing “git svn help” to get assistance.
git svn -s --authors-file=authors.txt --prefix=svn/ --username=svn_username \
clone https://url/path/to/svn/repo/ directory-for-git-repo
This process will take a while. It will fail with an error if it encounters a username that isn’t in your authors.txt file. Add the author and then re-run the command; it will continue from where it left off.
Once it’s complete, consider running it again just in case. My first attempt said it was finished and it wasn’t.
You now have a working Git repository. The SVN repository is hooked up as a remote and all of its branches and tags are prefixed with svn/, which makes them easy to find. The next step is to turn the SVN branches and tags into local Git branches and tags to complete the import.
3) Make local branches for each remote SVN branch. When we do the import we specify that all of the SVN branches go into the prefix svn/. I wrote a brief shell script to perform this step; all it does is to list the remote svn branches (omitting the tags and trunk), and then checkout each one-at-a-time to a local branch.
cd directory-for-git-repo
for i in `git branch -r --no-color | grep 'svn/' | grep -v 'svn/tags/' | grep -v 'svn/trunk' `
do echo $i
git checkout -b `echo $i | sed -e 's|svn/||'` $i
done
4) Make local tags for each remote SVN tag. SVN tags are imported as remote branches, so as for the branches, we run a short script to create local Git tags for each SVN tag.
for i in `git branch -r --no-color | grep 'svn/tags/' | sed -e 's|svn/tags/||'`
do echo $i
git checkout -b tag_branch svn/tags/$i
git checkout master
git tag `echo $i | sed -e 's|svn/||'` tag_branch
git branch -D tag_branch
done
5) Make the master branch from the SVN trunk. I found that my master branch was a superset of every other branch after the git svn import, so in this step we rename the existing master branch, create a new one, and then delete the old one.
git branch -m master oldmaster
git checkout -b master svn/trunk
git branch -D oldmaster
And that completes my import process. The next step is probably to clone the repository, or possibly to push it to its new host. Cloning gets rid of all of the git svn configuration and remote branches and leaves you with a clean Git repository.
EGit fails to import an existing Git working copy
I used another Git tool to clone my git repository into my development directory, which is also my workspace directory in Eclipse. I then wanted to add this working copy as a project in Eclipse; it has a .project file as it’s already an Eclipse project (previously in an SVN repository).
Using the File > Import menu option and choosing Projects from Git, I was able to add my Git repository (which is inside the working copy), but when I go through to import existing projects it fails with an error:
Invalid project description.
/Users/karlvr/Development/projectname overlaps the location of another project: 'project name'
This turns out to be because importing projects from Git wants to make a new working copy in your workspace, much like with SVN where the wording was more obvious to me “Checkout Projects from SVN”
What I actually wanted to do, and what worked, was to import the project as an existing project and then activate it with Git. To do this, choose File > Import, then Existing Projects into Workspace. Choose your project directory, probably already located in your workspace, and import it. Right-click on the imported project in the Project Explorer and choose Team > Share Project. Choose Git as the repository type, then in the Configure Git Repository dialog choose “Use or create repository in parent folder of project”. EGit will find the .git folder at the root of your project and your project will now be in Eclipse and setup for EGit.
Optimising images / Optimizing images
Tomato tomato. If you’re putting images into an application, especially a mobile application, or on the web, or anywhere, it’s a good idea to optimise them before you deploy. In an iOS application the larger your application is the longer it takes to download and install. If you go over the magic 20MB limit, users aren’t even allowed to download it over 3G; they have to seek out a WiFi network. This doesn’t feel like it would be good for sales.
Xcode runs pngcrush on your PNG files, which does an okay job. It doesn’t touch the JPGs, and overall it turns out you can do better.
Enter ImageOptim. A free Mac OS X app (unfortunately not available via the Mac App Store due to licensing) that combines a series of command-line image optimisation tools; runs them all and chooses the best optimisation for each image. It also finds images recursively in folders, so I dragged and dropped the root Images folder to do a whole application.
The optimisations are lossless, so it doesn’t downgrade the quality of your images (this is something you should also experiment with separately, especially JPEG quality). The tools remove metadata and other information that you don’t need; they can also recompress or optimise the compression to gain a few more bytes.
There’s one extra optimisation tool that ImageOptim can use but isn’t bundled: PNGOUT. There are instructions for downloading PNGOUT on the ImageOptim webpage. I recommend getting it and configuring the path to it in the ImageOptim preferences; it often got the best results on my PNGs.
Thanks to Corwin Derkatch for putting me on to ImageOptim.
The coolest thing ever
It’s fair to say that when you enter the office, it is criminal not to have a short intro theme blasted out on the stereo for all to enjoy. What an entrance. It sets the tone. It sets the mood. And the ladies love it.
Fortunately, thanks to a confluence of technology, we blew this case wide open last week at Cactuslab. We now have the DHCP server running a script when known devices join the network which plays personalised MP3 files via the Airport Express to the office stereo at ear-popping volume. When you walk into the office with your WiFi device in your pocket… boom.
The first piece of the puzzle is integrating with the DHCP server. I run our office’s DHCP server on a Linux server, and it turns out to be quite easy to execute a script when the DHCP server hands out a new lease. The DHCP server executes a shell script, which takes as an argument the hardware address of the new device.
on commit {
set clip = binary-to-ascii(10, 8, ".", leased-address);
set clhw = binary-to-ascii(16, 8, ":", substring(hardware, 1, 6));
execute("/usr/local/sbin/dhcpevent", "commit", clip, clhw);
}
The script looks for an MP3 file matching the hardware address in a folder on the file server. If an MP3 is found the script attempts to play it on the office stereo using raop_play, which connects to the Airport Express via AirPlay. Bob’s your uncle.
#!/bin/sh HARDWARE=`echo "$3" | sed -e 's/:/-/g'` INTROSDIR=/data/cactuslab/share/intros AIRPORT_EXPRESS="10.1.10.135" INTRO="$INTROSDIR/$HARDWARE.mp3" if [ -f $INTRO ]; then raop_play $AIRPORT_EXPRESS $INTRO & fi
There are a few problems with this setup. Most annoyingly, if someone is already playing something on the stereo then your intro can’t play. We’re resolving this issue by adding a second, dedicated, Airport Express. The two Airport Expresses will run through a mixer. I think we’ll set the mix on the intros Express slightly higher, you know, so you can really feel it. I’ll update on the success of this later.
The biggest problem, however, is that it takes a while for your iPhone to jump onto the WiFi network. So sometimes it’s less of an intro and more of a reminder that you’ve already arrived. It’s still pretty cool. I have a feeling there’s a déjà vu issue when a device rejoins the network… but that could be changed with some more clever scripting.
I recommend keeping the intros to about 15 seconds and adding a fade-out to the end of the MP3.
Keep it classy, and please let me know if you attempt this!
Charles 3.6.4
Charles 3.6.4 was released this week. It includes a number of bug fixes which are outlined in the version history. If you had trouble with Charles sometimes starting with a blank window, or freezing while using breakpoints, this might be the bug fix release for you. If you are interested in the HAR import/export capabilities, this is definitely the release for you.
I also want to draw attention to the PCAP import. You can capture traffic in a tool, such as Wireshark, and then save that traffic in a .pcap file and then import it into Charles. Charles analyses the HTTP traffic and pieces it back together as if Charles had been there to intercept it. This can be an option for capturing and analysing traffic in Charles when you can’t get an application to directly use Charles. Aside from bug fixes, Charles 3.6.4 also improves the PCAP import - ensuring that keep-alive requests and imported and putting things in the right order.
The other thing I want to highlight is Linux support. Charles has supported Linux since very early on in its life, but it’s never done a particularly good job at installing and running nicely as part of a Linux desktop. This is no doubt my fault as the vast majority of my Linux boxes are command-line only. Charles 3.6.4 features an improved start script for Linux, bigger icons, and most excitingly an APT package for Debian-based Linux distributions. I’ve particularly tested it on Ubuntu: the package installs Charles into your applications menus, and the package makes the process of getting new versions of Charles much easier. If you’re a Linux user who can’t use APT packages, get in touch and we’ll see if we can make other package types!
The next release of Charles is going to include more bug fixes and minor enhancements; I’m particularly focussing on fixing a few issues around Java 1.7 so if you’re having trouble with 1.7 please get in touch and try the latest beta.
