Some thoughts on storage and backups

What are your images worth to you? And by worth I don’t necessarily mean in a financial sense; how would you feel if your primary hard drive toasted itself and you lost most or all of your images today? What would you do? Personal and family images would be be irrecoverable and gone forever. Your stock library would suddenly cease to exist. No more portfolios. Images that you hadn’t finished editing and obviously not yet delivered to clients…imagine having to ask for a re-shoot.

Clearly, backup is important. Very important. Currently, the weakest link is in the camera at the time of capture: most cameras only have a single card slot, which means only one copy of the image at any one time. Even those that have dual card slots and allow parallel writing (best configured to mirror to give optimal flexibility) create issues when browsing, deleting and otherwise managing files in camera – it isn’t seamless or intuitive, and the limitations of a button driven graphical UI make file handling less straightforward and easy that on a PC. And even if you mirror your files to both cards, if your camera happens to suffer say impact damage or a moisture-induced short, you can probably say goodbye to both cards. Incremental backup to a PC on site either via an Eye-Fi card, WLAN or tethering is an option, but can be time consuming and downright impractical if you’re a photojournalist.

My current workflow is still subject to this limitation:
– I shoot as normal, and backup to a computer as soon as I’m done with the shoot.
– I won’t format the cards until everything is backed up, which means that I have two copies of the files at any time (and yes, I have a lot of cards – SD: 3x32GB, 5x16GB, various odds and ends; CF: 2x16GB, 3x8GB, 3x4GB – it’s a lot considering I’m only dealing with 14MB per RAW file on my sole CF camera (D700).)
– At incremental stages – usually every hour or so when editing – I’ll run a time machine backup of my working machine’s drive, which gives me a third copy of the original files, and a second copy of the edited files.
– Hard drives are kept disconnected from the computer and power supply to prevent any accidental data corruption due to surges, odd software glitches etc.
– Once I’m done with the processing and editing, I make duplicate backups of all files onto a pair of hard drives.
– Time machine the main editing machine again – now I’ve got four copies of the finished files, and five copies of the original files. Still with me?
– I don’t keep the original files on my main editing machine because of space limitations, but I do keep the finished files.
– Now I can format the cards from the shoot – this leaves me with four copies of finished files, and two copies of the originals.
– You want to maximize redundancy and failure-proofing with your backups, so keeping one full set of files off-site is a good idea. It’s also nice because you can easily access your work if you’re not in the office. Both of the backup hard drives and the time machine drive I currently use are 1TB portables; they require no external power supply and are small enough to go anywhere. One stays in my bag, which is almost always with me.
– There’s yet another copy of my portfolio files and critical image files on my travel machine (a Macbook Air) and iPad, making six sets for critical files. I’m fairly confident with this setup that I’ll be able to find an image if I need to, even if two or more drives fail. (And that’s not counting the drives that I’ve archived, which I’ll touch on in a minute.)

Now you’re probably wondering what happens if my backup solutions run out of space – after all, these days 1TB isn’t that much, and the minute you touch video work (a whole new kettle of fish) then the storage requirements multiply exponentially.

Fortunately, the pace of file size increases has also roughly matched then pace of both increases in storage density and processing speed (score one for Moore’s Law). I double the size of the replacement drives I buy – usually one every year and a half to two years or so – and find that gives me enough breathing room. When the new drives are in, I’ll make a fresh copy of all the files (NOT using a cloning tool, because that can also clone over any errors and bad sectors the old drive has – negating some of the advantage of fresh drives) onto my new, larger drives.

I recently had my time machine drive fail; that, culminated with a general lack of space (only 50GB left and plenty of those enormous D800 raw files) made me upgrade to one of these array thingies. I’ve got a WD MyBoook Studio II 6TB array, which is set to RAID 1 (mirroring) and partitioned into a 1TB block for my time machine backups, and a 2TB block for storage. Firewire 800 is thankfully supported, which makes moving huge files around quite painless. I’ll probably add one of the new WD portable 2TB Firewire/ USB3 drives to replace my portable 1TB so I can keep an offsite backup of my work on me (and a handy archive) wherever I go.

Backup is something everybody needs to think about seriously, today. Tomorrow may well be too late – there have been several occasions where I’m glad I was running multiple backups; unluckily I had a Maxtor primary drive and a Maxtor backup drive fail within a day of each other – leaving me with just a single (fortunately non-Maxtor, those things are a disaster) drive. If you can’t afford to lose your images, don’t put it off! MT

Comments

  1. I have over 140,000 images. Finepix S1, S2 Raw files, D300, D700, and D800 raw and jpeg files. I use a DROBO with four, 2 TB drives. I also backup to lacie externals — 2ea. One is ALWAYS with me, the second I keep wrapped in a towel and in my car when I’m out traveling. I also have my Time Machine. I do not keep any images on either my iMac HD, or my macbook pro laptop. Nothing at all.

  2. Chris Cupit says:

    Thanks Ming,

    I lost all the pictures of my kids from 2004-2007, laptop died, portable drive the backup was on stopped working. I kept it in a drawer incase I got rich (heh) and professional drive recovery became an option. This year I pulled that same drive out of a drawer, drop kicked it across the kitchen (accidently) and suddenly it started working. I got the images off that pretty quick!

    My thoughts..

    If you buy drives to back up to dont buy them all at the same time and not all the same manufacturer. All manufacturers have issues or bad batches, you dont want all the drives you have to suddenly all start going wrong at the same time.

    If you backup to the cloud or any internet site BE CAREFUL.. lots of people lost everything when rapidshare recently was shut down with no notice and no chance to get back their (legal) data because people were storing illegal files on it. Companies go bust too, sometimes with little warning. Also think how safe your data is.. one of the cloud providers has turned off authentication by mistake a couple of times leaving all the private files open to anyone to read.

    Dont keep all your archive drives in one place. If your house burns down it does not help you very much if your archive was all stored under the stairs. Keep a copy at a friends/relatives/workplace.

    There’s always a chance you could lose everything.. just minimise the risk, spread it around and remember.. if you drop that drive and it stops working.. that’s not always the end. There are companies that can recover the data.. it might cost a LOT.. but it might be worth it.

    Chris

    • Great suggestions, Chris. One other option – better than the cloud – is to have your own server somewhere else, which you can also store a copy of your data on.

  3. Scott P. says:

    Thanks for your great blog, I find it a place of common sense amongst so much questionable opinion in the photography world.

    A question I am always mulling over is ‘long term file compatibility’. I save and back up my original Raw files (CR2, soon to be NEF) and my Lightroom catalogue which I use for 99% of my editing these days. I was also backing up a DNG file to my off site drive but haven’t been too diligent in this lately because the conversion process takes sooooo long.

    Most people have 100% confidence in the fact that Canon/Nikon will be here forever and file support is a given, but I remember when Kodak was the largest user of silver in the world and a brand up there with CocaCola as the most recognised around the world, now they’re bankrupt and on the edge of oblivion!

    Do you have an opinion on long term file compatibilty and what process do you use or recommend?

    Thanks
    Scott

    • No problem, Scott. Photography is subjective, and everybody has an opinion, and I’d be lying if I didn’t say that included me!

      On long term file compatibility – who says anybody is going to support DNG 40 years from now? Stranger things have happened. I figure so long as photoshop maintains back compatibility, you’re fine. The moment there are issues about compatibility after a new release, I’d be very, very concerned about converting all of my files over to a newer format, as well as keeping the old software around in case there are issues post-conversion. Even if that means keeping a c.2012 vintage Mac and OS around.

  4. Gustavo says:

    Thanks for the write up. i too havelost personal memories due to a Seagate portable HD failure. I learned not to buy Seagate again, but more importantly, that 1 backup HD is not enough!

    Thanks and keep ahooting!

    • The more backups the better – so long as your primary drive is well organized. Storage is cheap, images aren’t replaceable! The weakest point in the chain right now is the part immediately post-capture – few cameras do RAID in camera.

  5. Hi Ming,

    It truly refreshing how open you are about your tradecraft, but I thought I may share our insight from operating a business in Christchurch, New Zealand, as you may know there were two (in fact many thousands, of which about 6 or so where bad) very destructive earthquakes in 2010 and though 2011.

    What I can say is that you may never know when you may lose access to ALL of your stuff (really I mean ALL of it), and this has changed the way most business here now think about backup. After the dust had settled a bit you think, ‘right lets go and get out backups and setup and see if we can start things rolling again…’, Nope, the larger part of the CBD was a no-go zone, and so was the fancy ‘off-sight’ storage site where all the ‘save’ backup were, or the guys place that had the other-other backup.

    So now our backup process now includes a using ‘cloud’ (for some reason the accountants and lawyers here call it ‘blue-sky’) backups also, and it is done with the idea that you will need to use this too recreate your business. The local copies (hard disks etc) are for the small localised problems, fire, disk failure etc, but there is an automated, and proven-to-work copy, far away in another country, this lets me sleep.

    What I can also see from your thoughts above, is that backup is a working discipline, a rhythm, a mindset that requires little thought because it’s just the way it is.

    Again, thank you for you remarkable writing.
    Kind regards,
    Carl.

    • No problem, Carl. I think the best thing you can do re. backups is make sure you a) have one offsite b) possibly even have one with you at all times. I try to do b) on a small portable drive that lives in my briefcase. It’s not complete, but it sure as hell beats nothing. Is Christchurch pretty much back to normal now? I used to live in Wellington many years ago, and there was always a lot of concern around ‘the big one’ but not a whole lot anybody could do to prepare for it.

  6. Hi Ming – Have you looked into any cloud based alternatives for back up? The ability to store Raw formats seems key, but I haven’t found one that looks like a good solution. Thanks.

    • Honestly, no. The biggest problem I’m having now is bandwidth issues, especially in this country…even a 10mb file takes near forever to upload.

      • If you’re on a Mac, the best solution I’ve found is Crashplan. Backblaze is another popular option but it doesn’t let you specify folders.

        I have to admit it took a very long time for me to backup to Crashplan over the air, and I have the advantage of a fibre connection here. But it takes care of the off-site backup part of my backup plan, and it’s nice to have access to your files wherever you are, as long as you have an internet connection.

        • Thanks for the heads up. Still working on a local solution that doesn’t require enormous bandwidth – we have 30-50GB per month quotas here – which would mean that a backup for me takes ages, with somewhere around 2TB of files…

Trackbacks

  1. [...] Storage media. This one concerns me quite a lot: we’ve already seen big shifts from one type of digital media to another, with almost zero support once a type of card or drive or storage goes out of fashion. Self-burned CDs from just a few years ago seem to be hit and miss when it comes to readability; early DVDs are a tossup because of the format used; and has anybody tried to mount an old drive recently? They don’t always work. I’ve found the best way to keep your files and backups accessible is to use external hard drives, limit the amount of uptime they have (if you’re not using them, don’t turn them on) and change them every couple of years. This serves several purposes: firstly, maximizing forward compatibility; secondly, giving you more space; thirdly, hopefully more reliability as technology matures; and finally, the old drives can be kept somewhere as an incremental backup. And drives fail, too – you might want to take a look at my article on storage and backups. [...]

Thoughts? Leave a comment here and I'll get back to you.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 26,630 other followers

%d bloggers like this: