Karl von Randow

IO performance investigation

Periodically it seems that one of my servers is suffering due to io. I’ve found two great tools for investigating: iostat and iotop.

iostat shows you the overall system IO statistics. BUT NOTE! The first output from iostat is always the IO statistics since boot. This is pretty much often useless, as I care about what it is now. And then when it doesn’t change when I rerun it I get confused.

Always run iostat in its repeating mode:
iostat -n 2

This will show the stats since boot, and then show you the current stats every 2 seconds.

iotop is pretty much like top and easy to use. Most importantly, compared to iostat, it shows you which process is using the IO. This is where you find the smoking gun.

I recommend running iotop with the -o flag, to only show processes that are doing IO. This seems obvious - hide the processes that aren’t involved.

Install iostat:
apt-get install systat

Install iotop:
apt-get install iotop

Turn a normal Git repository bare

If you have a regular Git working copy, and you want to make a bare repository out of it - it’s not quite as simple as just moving the .git folder out of the working copy. Almost, but not quite.

First move the .git folder out of the working copy. Say your working copy is  called Software, then you’d move Software/.git and call it Software.git.

The only other step you need is to tell git that it’s a bare repository:

pushd Software.git
git config --bool core.bare true
popd

Charles 3.7 released

The latest version of Charles contains several new features, which I will blog about shortly. But in this post I want to talk about the changes to the way Charles is distributed.

Previously Charles required a Java Runtime Environment (JRE) to be installed on your computer. This was sometimes a problem, but usually not. Particularly on Mac OS X where Java has been preinstalled for ever. However Apple has stopped preinstalling Java, and their Java implementation is soon end-of-life. Furthermore on Windows some strange goings-on have been reported with the Java installer and the side-loading of browser toolbars. And finally the security problems with the Java plugin, which don’t affect Charles, but nonetheless tarnish the requirement of installing Java… All of concern when requiring a Java installation.

Fortunately there is a simple solution. As of version 3.7, Charles now bundles a private JRE on Windows and Mac OS X. This makes the application download larger, but simplifies all of these issues. The private JRE doesn’t impact the rest of your system, whether you have Java installed or not, and it doesn’t include the Java plugin or install anything in your browser so there’s no concerns. I hope that this change makes installing and using Charles easier!

If you were previously concerned about having Java installed because of the Java plugin issues, you no longer require a Java installation for Charles so you are free to uninstall Java from your system. 

On Linux it’s a slightly different story. I could bundle a JRE for Linux, however the Java runtime packages tend to be updated along with other system packages, and bundling software that is available as a separate package is frowned upon. The best way to install Charles on Linux is to use the apt repository, which specifies the Java runtime as a dependency. I welcome feedback on this issue!

Finally in other Mac OS X news, I’m pleased to announce that Charles is signed with a developer certificate from Apple so you can open it without interference from Mountain Lion’s GateKeeper. And Charles also looks more Mac-like on Mountain Lion again, so your eye balls can relax.

Eclipse hangs on launch

For me it means the metadata in my workspace is corrupted. Rather than blow away the whole thing and start again I tracked down the troublesome file, and the solution.

From your workspace directory, on a command-line:

rm .metadata/.plugins/org.eclipse.core.resources/.snap

After that Eclipse launched and then rebuilt my whole workspace. Hope it works for you!

Merge in another git repository

So I had a separate git repository that I wanted to make part of my main repository, but I didn’t want to lose all of the commit history so I couldn’t just copy the files in.

The process turns out to be simple. Position yourself in the destination repository, and then add the repository you want to merge (call it “other”) as a remote. Checkout that remote branch, make any modifications (such as moving files around), commit it, go back to your master branch (or whatever your destination is) and then merge it in.

git remote add other /path/to/other/repo
git fetch other
git checkout -b other-working other/master

Now you have your other repository checked out, so you can move files around; such as putting them into subfolders. Remember to use git mv so git keeps a perfect track of what you’re doing.

git commit
git checkout master
git merge other-working

And you’re done.

Based on the accepted answer http://stackoverflow.com/questions/1683531/how-to-import-existing-git-repository-into-another

Aperture relocate videos for use in iMovie

Our Aperture library ends up with photos and videos. The videos tend to be quite large, so I remove the video files from the Aperture library periodically. I relocate them into project folders in the iMovie Events folder (which is actually named iMovie Events.localized, you just usually can’t see that).

Relocating masters from Aperture, rather than exporting and removing from Aperture, means I can still see my videos in the timeline with my photos in Aperture. Putting the masters into the iMovie Events folder means that I can easily access the movies in iMovie without having to use the Aperture integration in iMovie which I had some trouble with that I can’t remember!

Adventures merging Aperture 3 libraries

In our family we take a few photos, and they end up in different places. Two of those places are two laptops and two separate Aperture libraries. Two years ago I took the plunge and merged the libraries using Aperture’s library merging feature (to say nothing of the prior migration from iPhoto). I then used Duplicate Annihilator to remove duplicate photos, and generally tidied up my libraries. I then left both laptops with a copy of the same Aperture library (containing photos from 2011 and on, as my SSD couldn’t cope with much more).

Spin forward to 2013 when it once again seems prudent to take stock of our photos. Attempting to take the two Aperture libraries and merge them results in an error:

The library could not be imported because it is a duplicate of the library currently running.

Rats.

After attempting various tricks, including importing one library into a fresh library, I was unable to use Aperture to merge the libraries. I was able to add one library to the other, however then I’d have an enormous number of duplicates to delete and it felt like it introduced some unpredictability into the result… deleting duplicates feels like a last resort to me rather than part of the plan.

Time to start hacking.

Aperture libraries are packages, which are just disguised folders, and they contain masters, which are the original files. Using a few UNIX tools I decided to find all of the files in library B that are also in library A, at the exact same path in the masters folder, and remove them. This is based on the assumption that our libraries were exactly the same until they diverged. Obviously if both libraries have imported new photos separately I will end up with duplicates. Hopefully not too many.

Here is the simple script I wrote to first find all of the masters in library A and then to delete them. BACKUP YOUR LIBRARIES FIRST.

LIBRARY_A=/Path/to/library/A.aplibrary
LIBRARY_B=/Path/to/library/B.aplibrary 

pushd “$LIBRARY_A”
find Masters -type f | while read i; do
    if [ -f “$LIBRARY_B/$i” ]; then
        rm “$LIBRARY_B/$i”
    fi
done
popd 

This runs pretty quickly. When it’s done LIBRARY_B is smaller than it was. It is also a slightly broken Aperture library, in which a lot of originals are missing. To remove those broken images you need to open the library, hold down the option key and choose Generate Previews from the Photos menu (while not having any photo selected). Aperture will take a while, but it will go through all your photos and notice that ones that are missing. Once that process is complete, filter your library for “File Status” of “missing” and then delete those photos.

You still can’t import B into A, as Aperture will give the error mentioned above. But you can now import B into a new library. Then finally you can import that new library into A and then set about resolving any duplicates you did in fact end up with.

SenTestingKit async helpers

SenTesting is built into Xcode 4 for unit testing. One of its big limitations appears to be unit testing async operations, such as network requests. The best solution (aside from using another framework) appears to be to use semaphores to block the test until the operation is complete, and to run an NSRunLoop to enable things like network operations to receive callbacks.

I have used a simple set of 3 macros to make this easier to use:

#define ASYNC_TEST_INIT \
dispatch_semaphore_t __asyncTestSemaphore = dispatch_semaphore_create(0);

#define ASYNC_TEST_COMPLETE \
dispatch_semaphore_signal(__asyncTestSemaphore);

#define ASYNC_TEST_WAIT \
while (dispatch_semaphore_wait(__asyncTestSemaphore, DISPATCH_TIME_NOW)) \
[[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode \
                         beforeDate:[NSDate dateWithTimeIntervalSinceNow:10]];

Then you can write your async tests:

- (void)testDashboard {
    ASYNC_TEST_INIT

    [self doAsyncOperationWithSuccess:^(id result) {
        STAssertNotNil(result, @”Result is nil”);
        ASYNC_TEST_COMPLETE
    } failure:^(NSError *error) {
         STAssertTrue(NO, [error localizedDescription];
        ASYNC_TEST_COMPLETE 
    }];
 
    ASYNC_TEST_WAIT
}

If you want to chain a series of async operations you can add another async operation after the ASYNC_TEST_WAIT. Use ASYNC_TEST_COMPLETE and ASYNC_TEST_WAIT again as normal. You don’t need another ASYNC_TEST_INIT.