It’s a simple solution to a rather complicated problem: how do you backup very large virtual machines on an hourly basis with VMware Fusion, Mac OS X and Time Machine or Time Capsule? It’s easy!

Ever drop your laptop and lose your data? Or had a hard disk fail without warning, causing data loss and much grief all around? I know many people who have lost their data and found themselves grief-stricken. Was it was months or years since you last “backed up”? You’re not alone, but that doesn’t make things easier. All those files you’ve been working on for months. Gone. You never though it would happen to you.

I use a MacBook Pro laptop and the fantastic Time Machine automated backup system. It even works wirelessly and backups up huge amounts of data for me every day. There’s just one problem: I use VMware virtual machines extensively throughout the day, and can need to backup close to 100 gigabytes of information on a daily basis. That’s not practical using a standard 802.11n wireless connection today (the fastest available).

I solved the problem, though, thanks to some help from a few clever people. I found a way to slice my virtual machines (VMs) into tiny little bite-sized pieces that Time Machine can backup more easily. The solution is all to do with Apple’s sparsebundle disk technology.

What is a sparsebundle, and why should I care?

Sparsebundles are wonderful things that will make your life better. They are an OS X specific technology that works excellent with VMware Fusion, large virtual machines, Apple’s Time Machine offering and the Time Capsule wireless backup router/storage device. Sparsebundles can take any large number of files and folders on your computer and package them up into a single file that acts as its own disk or portable filesystem – a sparsebundle disk image. It’s like any age-old zip or tar file, only much better and optimized for modern filesystems. They are portable and can even be encrypted. They work seamlessly with the innovative OS X Time Machine filesystem. These sparesebundles are special kind of disk image that slices your data up into into many thousands of smaller 8 Mb pieces that can easily be backed up when they change. A practical application of this in backing up large virtual machines, which are often so large they’re difficult to backup on an hourly, or sometimes even daily, basis. Using sparesebundles with VMWare Fusion is such a great idea I wish I’d thought of it sooner but alas, other people have already written at length about them.

You should care because you can backup large virtual machines on an hourly basis, which is very difficult to achieve using even the fastest USB and eSATA technologies today without a little filesystem help – in other words, Apple’s Time Machine backup technology.

In my first few days of usage, my Time Machine backups went from over 80 Gb every hour to just 1 or 2 Gb on average. That’s a dramatic improvement. It makes wireless backups every hour feasible and even easy (automated). It is not at all difficult to get VMware Fusion to use virtual machines that are inside sparesebundles either, but it does take a little prep.

After searching the Internet about VMware Fusion, Time Machine and the issues with large virtual machines, I came across an excellent article by Mark Wheadon, called Efficient backups: storing VMs in a sparse bundle. It walks you step-by-step through getting it working with VMware. Everything is very easy and the benefits are almost immediate.

Caveat?

The only potential downside is a performance hit you may experience when doing very heavy disk activity, as sparsebundles cause additional overhead on the server or desktop. This is largely OS and disk dependant, and seemed to improve for me when Apple’s OS X went from Leopard to Snow Leopard. I suspect newer SSDs would further reduce any performance hit. Either way I think the excellent benefits of automated, hourly backups virtual machines outweigh the minor performance hit. The advantages in having automatic, incremental Time Machine backups of large virtual machines is clear. Careful attention to the individual virtual machine’s configuration settings will help mitigate any problems – in other words, don’t use VMware Fusion’s Snapshot feature while using sparsebundles! Just turn it that feature off and you’ll be glad you did.

In Summary

It’s nice to know that even if my laptop was dropped, destroyed or stolen, all the data on both my home and work computers, together comprising almost 500 Gb of data on my Macbook Pro as I write this, in early 2010. My data will always be backed up to save my butt if the computer is lost or stolen – right up to within an hour before I left the office and things went awry.