What are your images worth to you? And by worth I don’t necessarily mean in a financial sense; how would you feel if your primary hard drive toasted itself and you lost most or all of your images today? What would you do? Personal and family images would be be irrecoverable and gone forever. Your stock library would suddenly cease to exist. No more portfolios. Images that you hadn’t finished editing and obviously not yet delivered to clients…imagine having to ask for a re-shoot.
Clearly, backup is important. Very important. Currently, the weakest link is in the camera at the time of capture: most cameras only have a single card slot, which means only one copy of the image at any one time. Even those that have dual card slots and allow parallel writing (best configured to mirror to give optimal flexibility) create issues when browsing, deleting and otherwise managing files in camera – it isn’t seamless or intuitive, and the limitations of a button driven graphical UI make file handling less straightforward and easy that on a PC. And even if you mirror your files to both cards, if your camera happens to suffer say impact damage or a moisture-induced short, you can probably say goodbye to both cards. Incremental backup to a PC on site either via an Eye-Fi card, WLAN or tethering is an option, but can be time consuming and downright impractical if you’re a photojournalist.
My current workflow is still subject to this limitation:
– I shoot as normal, and backup to a computer as soon as I’m done with the shoot.
– I won’t format the cards until everything is backed up, which means that I have two copies of the files at any time (and yes, I have a lot of cards – SD: 3x32GB, 5x16GB, various odds and ends; CF: 2x16GB, 3x8GB, 3x4GB – it’s a lot considering I’m only dealing with 14MB per RAW file on my sole CF camera (D700).)
– At incremental stages – usually every hour or so when editing – I’ll run a time machine backup of my working machine’s drive, which gives me a third copy of the original files, and a second copy of the edited files.
– Hard drives are kept disconnected from the computer and power supply to prevent any accidental data corruption due to surges, odd software glitches etc.
– Once I’m done with the processing and editing, I make duplicate backups of all files onto a pair of hard drives.
– Time machine the main editing machine again – now I’ve got four copies of the finished files, and five copies of the original files. Still with me?
– I don’t keep the original files on my main editing machine because of space limitations, but I do keep the finished files.
– Now I can format the cards from the shoot – this leaves me with four copies of finished files, and two copies of the originals.
– You want to maximize redundancy and failure-proofing with your backups, so keeping one full set of files off-site is a good idea. It’s also nice because you can easily access your work if you’re not in the office. Both of the backup hard drives and the time machine drive I currently use are 1TB portables; they require no external power supply and are small enough to go anywhere. One stays in my bag, which is almost always with me.
– There’s yet another copy of my portfolio files and critical image files on my travel machine (a Macbook Air) and iPad, making six sets for critical files. I’m fairly confident with this setup that I’ll be able to find an image if I need to, even if two or more drives fail. (And that’s not counting the drives that I’ve archived, which I’ll touch on in a minute.)
Now you’re probably wondering what happens if my backup solutions run out of space – after all, these days 1TB isn’t that much, and the minute you touch video work (a whole new kettle of fish) then the storage requirements multiply exponentially.
Fortunately, the pace of file size increases has also roughly matched then pace of both increases in storage density and processing speed (score one for Moore’s Law). I double the size of the replacement drives I buy – usually one every year and a half to two years or so – and find that gives me enough breathing room. When the new drives are in, I’ll make a fresh copy of all the files (NOT using a cloning tool, because that can also clone over any errors and bad sectors the old drive has – negating some of the advantage of fresh drives) onto my new, larger drives.
I recently had my time machine drive fail; that, culminated with a general lack of space (only 50GB left and plenty of those enormous D800 raw files) made me upgrade to one of these array thingies. I’ve got a WD MyBoook Studio II 6TB array, which is set to RAID 1 (mirroring) and partitioned into a 1TB block for my time machine backups, and a 2TB block for storage. Firewire 800 is thankfully supported, which makes moving huge files around quite painless. I’ll probably add one of the new WD portable 2TB Firewire/ USB3 drives to replace my portable 1TB so I can keep an offsite backup of my work on me (and a handy archive) wherever I go.
Backup is something everybody needs to think about seriously, today. Tomorrow may well be too late – there have been several occasions where I’m glad I was running multiple backups; unluckily I had a Maxtor primary drive and a Maxtor backup drive fail within a day of each other – leaving me with just a single (fortunately non-Maxtor, those things are a disaster) drive. If you can’t afford to lose your images, don’t put it off! MT
Ming Thein | Photographer 
Hi Ming, I messgaed a while back with a question about the Q, and have since been lucky enough to get one. Coming ‘up’ from the Fuji X system, this camera is in a different league, and i love it so far. However the 40+MB raw files plus 8MB jpegs are going to fill up my current system and affect my local and remote backups pretty quickly… So , i was wondering what you use for storage and backups nowadays?
Thanks
Rod
This article may be helpful on that.
I’m on a redundant pair of 4-bay 16TB Drobos, a further 6TB WD RAID 1 pair for non-critical stuff, critical portfolio sits on both my main machine drive and laptop drive, and I have a pair of 3TB WD Passport USB3 drives for further critical backups of the collection that I carry with me. There are two SSDs which are my scratch drives on the road (though one of the 1TB ones recently died and needs to be replaced) and about a further 1TB in SD cards.
Storage may be cheap, but it’s still worth spending time curating more tightly than ever – if anything, our human attention spans are shorter… 🙂
Hi Ming,
I did a search for storage and came up with this article. You essentially answered all my questions.
I am wondering how many original photos you have and how much space they would take up if you just had one copy?
My guess is somewhere around 300,000, and about 6TB. Raw files vary in size, and I shot a lot more quantity earlier on. Less now but the files are larger.
Dear Ming,
About Backup, a topic discussed a lot of times and never really understood by the broad mass….
I do have a technical background and where i come from backup looks like this
1.) you got the computer with one person working on it and connected to the internal network, all data stored on the computer is backuped locally, the main work should be stored on the network
2.) the network “drive” consists of two raid´s running parallel (not raid1! if you erase something on Raid1 you don’t have it anymore! so its no backup as per definition, its failure safety….) and Raid number one gets all the data raid number two gets… actually a backup software runs in the background and copies the data… like time machine more or less, just better and really expensive 😀
3.) once per day (usually during the night) raid number one gets mirrored onto a tape drive or tape library and one network technician collects the tape and puts it into a fireproof safe wich is far away from the raid´s, there are enough tape´s for a month to switch through, so they had a month worth of data if someone erased it all
4.) the master backup (archived projects and everything else important) was stored on the raid for everyone to access it (read only) and onto two tape´s stored on two different locations
thats a really rough picture of how my old company had their data secured
what i do at home:
i got my macbook pro with three hard disk partitions (before that it was a mac pro with internal raid5, a SSD for OS and external Raid5 for time machine…)
partition1 is OS and Software and all the private stuff
partition2 is Raw
partition3 is Work (live data)
-> partition1 is constantly backed up onto time machine (either onto one hard disk or time capsule…) and my office stuff is in my dropbox 🙂
-> partition2 is the same as partition1 but, in addition to it, i backup the data to a own hard disk and onto a raid5 with 8 disks wich, itself backups my raw data onto an external drive every hour, i do the same like you with the cards, so i do have my raw data at least on three different places because the raw data is the stuff you cannot replace or redo… i do have an additional extern hard drive with all the raw on it if i need something away from my office…
-> partition3 is all my photoshop and lightroom stuff, backuped like partition1 and also onto my raid5 but not as paranoid as my Raw data, here is my live data on wich i am working on… if something happens here its a pity and work time is lost but, its redo able…
partition2 and 3 get backuped onto Tape drive´s, i am currently using LTO5 wich gives my about 1,5Gb per Tape, they have a store time of about 50-100years (i will copy them onto a new tape after 25years) and are quite cheap compared to HDD´s, the only drawback is the tape drive itself, its about 1500$ – 2000$ including a good backup software (-> http://www.tolisgroup.com they helped me a lot and they do have own solutions for creative people like video and photographer…)
LTO is big consortium consisting of some quite big companies, so the possibility that it crashes and no hardware is available is quite minimal, the industry (electronical) uses a lot of tape technology so it will be around 😉 and, the update their hardware quite intelligent, so for example, LTO5 can read and write LTO4 and read LTO3… you get the picture 😉 every 5-7years a new technology comes around, if we calculate 5 years per update my LTO5 tape´s can still be read in a LTO8 drive in 15years… the hardware can be attached to any computer and OS, so its quite bulletproof…
The tape´s do not require anything, the just sit there on the shelf 🙂 i do not own a safe so i have two copies, one at my office, one at a really secret place (my parents house :D)
about external hard disks… if you want to keep them alive as long as possible (thats true for all hard disks) please leave them running and only buy server grade hard disks, get a uninterrupted power supply (batterie supply) for you main station and don’t think about any power surges or anything (at least here in vienna, where i am currently living, i had nothing like that in the last 25 years! maybe its different somewhere else in the world…)
Backup is a bitch, its time consuming and expensive… but i think about the next 25-50 Years, i cherish my pictures and i actually want my grandchildren to have something to look at…
the best backup is the one wich is actually made… so at least get a time capsule or anything similar and let it be done automatically… murphy does not sleep, the day your drive crashes with your raw data on it is the day where you forgot to backup your data and erased the cards to early! 😀 we are living in the picture less decade, i guess 95% of the pictures taken from 2004 till 2014 won’t survive to 2024…
have a great day
Robert
Thanks for the detailed suggestions!
I can’t help but think that the 95% that don’t survive is not a bad thing. At least it means that hopefully only those images created by people who were consciously thinking about both image and process survive…
I have over 140,000 images. Finepix S1, S2 Raw files, D300, D700, and D800 raw and jpeg files. I use a DROBO with four, 2 TB drives. I also backup to lacie externals — 2ea. One is ALWAYS with me, the second I keep wrapped in a towel and in my car when I’m out traveling. I also have my Time Machine. I do not keep any images on either my iMac HD, or my macbook pro laptop. Nothing at all.
Thanks Ming,
I lost all the pictures of my kids from 2004-2007, laptop died, portable drive the backup was on stopped working. I kept it in a drawer incase I got rich (heh) and professional drive recovery became an option. This year I pulled that same drive out of a drawer, drop kicked it across the kitchen (accidently) and suddenly it started working. I got the images off that pretty quick!
My thoughts..
If you buy drives to back up to dont buy them all at the same time and not all the same manufacturer. All manufacturers have issues or bad batches, you dont want all the drives you have to suddenly all start going wrong at the same time.
If you backup to the cloud or any internet site BE CAREFUL.. lots of people lost everything when rapidshare recently was shut down with no notice and no chance to get back their (legal) data because people were storing illegal files on it. Companies go bust too, sometimes with little warning. Also think how safe your data is.. one of the cloud providers has turned off authentication by mistake a couple of times leaving all the private files open to anyone to read.
Dont keep all your archive drives in one place. If your house burns down it does not help you very much if your archive was all stored under the stairs. Keep a copy at a friends/relatives/workplace.
There’s always a chance you could lose everything.. just minimise the risk, spread it around and remember.. if you drop that drive and it stops working.. that’s not always the end. There are companies that can recover the data.. it might cost a LOT.. but it might be worth it.
Chris
Great suggestions, Chris. One other option – better than the cloud – is to have your own server somewhere else, which you can also store a copy of your data on.
Thanks for your great blog, I find it a place of common sense amongst so much questionable opinion in the photography world.
A question I am always mulling over is ‘long term file compatibility’. I save and back up my original Raw files (CR2, soon to be NEF) and my Lightroom catalogue which I use for 99% of my editing these days. I was also backing up a DNG file to my off site drive but haven’t been too diligent in this lately because the conversion process takes sooooo long.
Most people have 100% confidence in the fact that Canon/Nikon will be here forever and file support is a given, but I remember when Kodak was the largest user of silver in the world and a brand up there with CocaCola as the most recognised around the world, now they’re bankrupt and on the edge of oblivion!
Do you have an opinion on long term file compatibilty and what process do you use or recommend?
Thanks
Scott
No problem, Scott. Photography is subjective, and everybody has an opinion, and I’d be lying if I didn’t say that included me!
On long term file compatibility – who says anybody is going to support DNG 40 years from now? Stranger things have happened. I figure so long as photoshop maintains back compatibility, you’re fine. The moment there are issues about compatibility after a new release, I’d be very, very concerned about converting all of my files over to a newer format, as well as keeping the old software around in case there are issues post-conversion. Even if that means keeping a c.2012 vintage Mac and OS around.
Thanks for the write up. i too havelost personal memories due to a Seagate portable HD failure. I learned not to buy Seagate again, but more importantly, that 1 backup HD is not enough!
Thanks and keep ahooting!
The more backups the better – so long as your primary drive is well organized. Storage is cheap, images aren’t replaceable! The weakest point in the chain right now is the part immediately post-capture – few cameras do RAID in camera.
Hi Ming,
It truly refreshing how open you are about your tradecraft, but I thought I may share our insight from operating a business in Christchurch, New Zealand, as you may know there were two (in fact many thousands, of which about 6 or so where bad) very destructive earthquakes in 2010 and though 2011.
What I can say is that you may never know when you may lose access to ALL of your stuff (really I mean ALL of it), and this has changed the way most business here now think about backup. After the dust had settled a bit you think, ‘right lets go and get out backups and setup and see if we can start things rolling again…’, Nope, the larger part of the CBD was a no-go zone, and so was the fancy ‘off-sight’ storage site where all the ‘save’ backup were, or the guys place that had the other-other backup.
So now our backup process now includes a using ‘cloud’ (for some reason the accountants and lawyers here call it ‘blue-sky’) backups also, and it is done with the idea that you will need to use this too recreate your business. The local copies (hard disks etc) are for the small localised problems, fire, disk failure etc, but there is an automated, and proven-to-work copy, far away in another country, this lets me sleep.
What I can also see from your thoughts above, is that backup is a working discipline, a rhythm, a mindset that requires little thought because it’s just the way it is.
Again, thank you for you remarkable writing.
Kind regards,
Carl.
No problem, Carl. I think the best thing you can do re. backups is make sure you a) have one offsite b) possibly even have one with you at all times. I try to do b) on a small portable drive that lives in my briefcase. It’s not complete, but it sure as hell beats nothing. Is Christchurch pretty much back to normal now? I used to live in Wellington many years ago, and there was always a lot of concern around ‘the big one’ but not a whole lot anybody could do to prepare for it.
Hi Ming – Have you looked into any cloud based alternatives for back up? The ability to store Raw formats seems key, but I haven’t found one that looks like a good solution. Thanks.
Honestly, no. The biggest problem I’m having now is bandwidth issues, especially in this country…even a 10mb file takes near forever to upload.
If you’re on a Mac, the best solution I’ve found is Crashplan. Backblaze is another popular option but it doesn’t let you specify folders.
I have to admit it took a very long time for me to backup to Crashplan over the air, and I have the advantage of a fibre connection here. But it takes care of the off-site backup part of my backup plan, and it’s nice to have access to your files wherever you are, as long as you have an internet connection.
Thanks for the heads up. Still working on a local solution that doesn’t require enormous bandwidth – we have 30-50GB per month quotas here – which would mean that a backup for me takes ages, with somewhere around 2TB of files…