Menu

Safeguarding your photos

I have very few irreplaceable possessions. Very few of the objects I own have any sentimental value. I’d be gutted to lose our fridge magnet collection, but it wouldn’t be a dying regret if I failed to save them from a fire. But when people are asked what they would pull from an inferno, photographs are consistently…

Fridge MagnetsI have very few irreplaceable possessions. Very few of the objects I own have any sentimental value. I’d be gutted to lose our fridge magnet collection, but it wouldn’t be a dying regret if I failed to save them from a fire. But when people are asked what they would pull from an inferno, photographs are consistently mentioned – and this is absolutely true for me.

I take lots of photos, and video – over 150,000 files, 800GB and growing all of the time. Managing this data, and keeping it protected is a struggle. To give me some flexibility (and to keep costs down), my approach to backups has changed over the years – and this post describes my current setup.

Protecting unimportant data to save time – Ubuntu Time Capsule

My wife and I have MacBooks, and I have an old Mac Mini which acts as a media server. The DVD and CD rips on the media server, and the operating system, applications and other files on these machines are replaceable, so we don’t need a sophisticated solution to protect these machines. We just need something quick and easy to protect them, to save hours of hassle reinstalling them in the event that one of them should die.

As they’re all Apple computers, there’s a great solution for this: Time Machine. Time Machine will save snapshots of each computer to a USB drive, but you have to remember to plug it in. Or it’ll save backups over the network to a Time Capsule, if you’ve got a spare £250. Fortunately, there’s a cheaper solution: roll your own. With a cast-off laptop, or a cheap Raspberry Pi, it’s pretty straightforward to install Ubuntu and then set up Netatalk with the appropriate configuration to emulate a Time Capsule. Example guides can be found here and here. I’d advise you to put the backups onto a different partition from the main OS, and if you can, put each machine’s backups onto its own partition as Time Machine can get confused about remaining disk space if multiple machines are scrapping over the same spare capacity. Otherwise, if you know how to install Ubuntu and can drive the command line and use vi, emacs or nano, you should be able to get up and running pretty quickly.

Backup ServerPutting this backup server in a different part of your house, or outside in a garage as I’ve done in this photo, gives you some extra protection if a fire, flood or theft takes out your main machine. You can use PowerLine ethernet to boost your bandwidth past wireless speeds.

Protecting important data with rsync

Time Capsule is great but it sucks with large quantities of data. In my experience, it periodically decides it needs to rebuild your backups. Over a 100Mbit ethernet connection to a USB 2.0 HDD running on a slow machine, my 1.2TB photo partition can take days and days to back up – not practical as I now do all of my photo editing work on my laptop. I needed something more robust for these large data partitions so reached across to a technique I first saw used by the late, great Jon Peatfield to provide a backup service to my old workplace, DAMTP.

The approach uses rsync to copy changed data over the network to the backup server. As with Time Capsule backups, the previous backup is preserved using hard links (using rsync’s –link-dest option), which means that a huge number of previous snapshots can be kept without much disk overhead.

Finally, the backup triggers a script on the backup server which thins out the old backups. Because I trigger the backups using Eventscripts, whenever my external USB photo drive is plugged in, the thinning is geometric (preserving the second, fourth, eighth, sixteenth and so on) – this is better than keeping weekly, monthly and yearly backups when the backup frequency is erratic.

The scripts are simple, and assume that a private key ssh connection to the Ubuntu server has already been configured. Instructions for doing this can be found here.

Backup script for client mac:

#!/bin/bash
# from http://blog.interlinked.org/tutorials/rsync_time_machine.html

USER=timemachine
SERVER=timemachine.marlow.org.uk
SERVERSTRING=$USER@$SERVER
KEY=/Users/gareth/.ssh/laptop-rsync-key
LAUNCHDATE=`date "+%Y-%m-%d %H:%M"`

echo Backup script launched $LAUNCHDATE
echo 
echo Disk usage on $SERVER
echo
ssh -i $KEY $SERVERSTRING "df -h -t ext4"
echo
do_rsync()
{
 local SOURCE=$1
 local DESTINATION=$2
 local DATE=`date "+%Y-%m-%dT%H_%M_%S"`

 if [ -f $SOURCE/.backupexclude ];
 then
 echo "Backing up $SOURCE"
 echo 

 rsync -az \
 -e "ssh -i $KEY" \
 --delete \
 --delete-excluded \
 --exclude-from=$SOURCE/.backupexclude \
 --ignore-errors \
 --link-dest=$DESTINATION/backups/1 \
 $SOURCE/ $SERVERSTRING:$DESTINATION/incomplete-$DATE \
 && ssh -i $KEY $SERVERSTRING "/home/timemachine/backup_rotate $DESTINATION $DATE"

 else
 echo "$SOURCE unavailable or .backupexclude absent"
 echo 
 fi

}


do_rsync /Users/gareth /mnt/timemachine1/Volumes/gareth
do_rsync /Volumes/Data /mnt/timemachine1/Volumes/Data
do_rsync /Volumes/Images /mnt/timemachine1/Volumes/Images

and backup_rotate on the backup server:

#!/bin/bash
# geometric pruning from http://brentmuir.com/snapshot/win_snapshot
# takes the base directory and the datestamp as arguments
# Create a "backups" directory under the target base directory to use.

DESTINATION=$1
DATE=$2

echo Successful backup on $DATE
echo

mv $DESTINATION/incomplete-$DATE $DESTINATION/backups/0

for i in $DESTINATION/incomplete*; do
 if [ -d "$i" ]; then
 echo Purging partial backup: $i
 rm -rf $i
 fi
done



for i in $DESTINATION/backups/* ; do
 (( dirnum=${i:${#DESTINATION}+9} ))
 (( nextdir = $dirnum + ($dirnum >> 1) ))
 if [ $dirnum -ge 2 -a -d "$DESTINATION/backups/${nextdir}" ]; then 
 echo Purging full backup: $i
 rm -rf $i;
 fi
done

# create new temp backup dirs with new numbers
for i in $DESTINATION/backups/* ; do
 dirnum=${i:${#DESTINATION}+9}
 (( nextdir = $dirnum + 1 ))
 mv $i "$DESTINATION/backups/${nextdir}.temp" 
done

# remove the temp postfix
for i in $DESTINATION/backups/* ; do
 mv $i ${i:0:${#i}-5}
done

# Update the mtime of the backup to reflect the snapshot time
touch "$DESTINATION/backups/1"

To the cloud!

But what if a plane should fall out of the sky and squash the house. Or a lightening strike causes a power surge and fries everything electrical? A local backup will go a long way to protect your data but you really need a copy offsite. One option is to copy data to a USB hard drive and leaving it in work, or with a friend, but this requires effort and thinking. Effort and thinking are both susceptible to human error – not a good strategy to look after your most precious possession. Fortunately, cloud computing is a great solution. Ben Adderson (twitter) uses CrashPlan, which I considered, but in the end I plumped for Google Cloud Storage. For $10 per month, my documents are saved into Google Drive and available everywhere, computers, mobiles, tablets and anything with a web browser. But even better, the photos are in Google+ Photos. Google have quietly built a Flickr-killer. Online photo editing, native support for RAW files (including the Olympus ORF format that my camera generates), easy to share. It’s fantastic. If most of your backups are photos (which is where we came in) I think it’s fantastic – and it’s free if you have less than 15GB to store.

As a final bonus, you get Google+ Auto Awesome photos. Generated automatically from your backed-up photos, it’s the gift that keeps giving. Cracking.

IMG_0745-EFFECTS

 

PB169711-MOTION

 

 Conclusion

We all value our photos, yet they’re just as vulnerable to accidental damage when they’re digital. Getting them backed up is the only way to protect yourself – whether you use a cheap and easy offsite service like Google+ Auto Backup, or combine it with a more sophisticated home setup, as I’ve described here. If you don’t have something in place, there are plenty of solutions – you shouldn’t leave it too late.

Comments

Matthew

Matthew

I like rsbackup for a slightly more bells-and-whistles approach to rsync-based backups:
http://www.greenend.org.uk/rjk/rsbackup/

reply

gtm12

gtm12

Matthew, I like the look of rsbackup but the laptop (and photo drive) is only connected intermittently, which makes server-pull backups a bit more challenging.

Chris Owens

Chris Owens

So you’ve got 1TB of photos in the cloud? Wow. Another thing to consider (if not using the cloud): you need RAID or some other sort of redundancy for your physical backups. I have used 2 mirrored external drives, and am now using a RAID NAS. I place them near(ish) fire exits, with the understanding that if you only grab 1 (nonliving) thing on the way out in case of fire, then that thing should be one of the drives.

But yeah: one’s family photos are irreplaceable. It amazes me when people don’t spend some planning time and a few quid to make sure they’re protected from some likely problems. Hard drives do fail with an MTBF of a few years – meaning the hard drive with your photos on is very likely fail at least once in your life.

reply

gtm12

gtm12

RAID is a good solution, particularly if you can get up to RAID5 and therefore not waste so much space. I still think you need some kind of physical separation between you and at least one backup set, however – what if you weren’t in the house when it caught fire?

Chris Owens

Chris Owens

Absolutely. But … the last time I sorted myself out properly, cloud data cost way more than you’re paying (or I thought it did, anyway), so wasn’t then an option.

I understand that RAID 5 is parsimonious, but cost-wise 2x1TB drive vs 3x500GB or 4x333GB probably comes out about the same, or anyway the difference is less than 6 months’ cloud backup (e.g. I just bought a 3TB drive for 90 quid). When I started with the low-tech “2 mirrored drives” approach, the 1TB external drives only cost 70 quid each: not a beautiful approach technically, but not expensive.

Leave A Comment

Comments support plain text only.

Your email address will not be published.