The Backup Dilemma

_RK59494-Edit_scn

So many things in our lives and businesses are in continual catchup mode: a work in progress, but never quite up to snuff. When we consider the images and files on which our business revolves, you don’t watch to be caught with an inadequate or broken backup process. The problem is: how to implement a process that balances simplicity with robustness (and preferably does not require a hard drive army that costs more than your camera body).

There are two types of photographers: Those who backup religiously, and those who will learn to.

When you weigh the time and equipment investment in setting up a strong, reliable backup system with the devastation of losing a client’s photos (or, not too far behind it: hours, days, or weeks of your work time), the scale for me really tips in favor of a solid investment. As a professional photographer, I can’t even imagine the loss of person hours, reputation, potential law suits, and other dastardly things that could result from losing a clients’ images. Peace of mind is irreplaceable for me.

I keep hearing over and over in Facebook groups and other circles from photographers who have a corrupt memory card, lost image files, or a crippled Lightroom catalog who are in a frantic panic and are unsure how to recover. While good recovery techniques are essential, the ultimate solution is prevention. Take steps now to prevent a loss before it happens by establishing a strong backup strategy and sticking to it. You owe it to your business and your clients.

I have spent a good number of years building and honing a strong backup strategy for my business and I want to use this page to share what I have learned and how you can create a bullet-proof backup plan for yourself. Below I will share the process and  gear to make it happen.

The Approach: What I Aim to Achieve

So what I am trying to achieve here? Each of us must find the right balance and tailor a backup plan that meets your business needs and fits within your resources.  Let me layout my goals in my backup system:

  1. A rock solid system that ensures my clients’ images and my work are safe and secure.
  2. Preserve original raw files and derivatives (not necessarily all of them, but at least my final culls). I don’t want to have to throw away my camera raw files simply because I have no space to preserve them.
  3. Defend against the enemies that can destroy our data.
  4. Provide multiple levels of backup (from original files to our latest and greatest stored offsite).
  5. Automate. I should be able to go to sleep or walk away from my desk and know that if I don’t return that day, my files will be backed up automatically.

Preserve Original Camera Raw Files and Derivatives

By derivatives I am referring to Lightroom Catalogs, Photoshop .psd files and other files carrying our valuable editing work that are created from the raws. Some photographers dispose of their original raw files after delivering images to their client (saving only the exported jpg files). To me, this is unfathomable and it just astounds me each time I hear a photographer say they do this.

My feeling is that such a practice arose at least party from a lack of storage, workflow and backup process to manage a growing set of raw files (versus not perceiving any potential value from ever saving the original raws).

To me, saving raws is the ultimate time machine. In the course of munching and processing files, nothing I have done can ever block my path back to square one. Over time, I may see an image in a new light, or grow as a photographer. How many times have we looked back at work from one, two, three years ago (or longer) and wanted to cover our monitor out of shame? (e.g.: “Jeez can’t believe I liked that shot”, or “wow, that color is way off, what the heck was I thinking”, etc.)

lr_2012_process

There are also plenty of times that newer software can improve older edits. For instance, the 2012 raw process (raw converter) in Lightroom was revolutionizing to my raw workflow. The overhauled tone control, noise reduction and sharpening allowed non-destructive improvements that were not possible on raws in previous versions. Several files (especially grainy night shots) came back to life after re-processing the original raws I had saved using the newer Lightroom process version. Had I not saved the camera raw files, I could not take advantage of this benefit.

While it is not my intention to debate raw versus jpg, save to say that if you do like and use raws, keeping them if you are able is a great goal to achieve.

I do not save all raw files captured. After I deliver a wedding, I throw away the raws of images that were not included in the final culled edits. As with many photographers, my keep rate is about 1/3 of what I shoot.

 The Enemies: Destroyers of Data

So what are we fighting? A good backup strategy protects against:

  • Human error: accidental deletion, overwriting files.
  • Equipment failure: failing drives and memory cards.
  • Malice in your palace: Theft and burglary in your home or office;
  • Digital demons: We must also not forget about the dangers of cyber theft and vandalism (google “ransomware” for an example of what I mean).
  • Fire: The thought is terrifying, but what if your home or office is destroyed by fire?
  • Regional destruction. What if more than just your location is destroyed by events such as natural disaster?

For me, there is no reason that any of these enemies need prevail. With the right gear and of course, religious adherence to your backup plan, we should be well guarded against each calamity.

Provide Multiple Levels of Backup

One copy of your files, or a single backup should not be considered an adequate backup. For instance, what if you inadvertently delete a file? Then you run a new backup. You have now created a backup that is lacking the file you accidentally deleted. Having multiple levels of backup allows a window to recover a lost files before a new, more recent backup stomps on an older copy that contains the file in contention.

Also, although even I shudder to think about it, having a backup drive on your desk is not even close to a solution. Think theft. Think fire. Getting files out of harms way is critical.

Automatic

If a system can’t at least to some degree run itself, we are going to fall behind. With automated software and a couple specific good practices in place, we can remove the mental burden and worry over a lot of aspects of protecting our data.

The Strategy

With this background in place, let’s dive in.We start with the following assumptions which serve as premises to the plan I have put in place:

  • Online backups are not suited to handle a complete raw history. My backup archives right now are measured in the terabytes (over 3 TB now). At the absolute maximum sustained upload transfer rate of 1.5 megabits per sec on my business class cable internet (real-world upload), that would take over 23 days to upload just a single backup job. If a backup job can’t be completed by the time I wake up in the morning, it’s not really a working model for me. The data is not safe during that period (and is changing as I work on it). Not to mention: The cost of the online storage would be overly expensive, especially if I want to keep an old backup job or incremental versions on hand.
  • This plan is to backup raw image files, psd files, Lightroom catalogs, album designs, and such. Files such as word processor documents, spreadsheets, pdfs and such I prefer to just use a “good ole” cloud services such as Dropbox, iCloud, or Google Drive. Easy, fast, and well suited to the relatively small amount of these types of files.
  • You will hear many say that storage is cheap (hard drives, etc.). That is not exactly true. Contrary to what you hear, storage is not cheap. Cheap drives are cheap. Good storage: fast SSD drives, RAID, and high quality drives cost significantly more (and will pay you back). This is another point of prevention. I hear so many photographers struggling with issues of drive failure. I have not lost a file on a drive in, I don’t know how many years. Probably 8-9 years. Why? I do not buy cheap hard drives. Yes, I know, knock on wood… More on the gear coming below.
  • You may not need or want to implement every piece or level of backups described here. But starting at the top and at least knowing what is possible allows you to strategically throttle back in your plan on if you need to work up to it, or already have a solution to address a certain point.

Contrary to what you hear, storage is not cheap. Cheap drives are cheap. Good storage: fast SSD drives, RAID, and high quality drives cost significantly more.

The WOW of SSD Drives: You Could Be Missing Out

IMG_20140225_101647_270_scn

One of fastest ways to turbocharge an older Mac or PC is SSD drives.

If you have not discovered the wonder of solid state drives (SSD), you are missing out. Any of these things sound familiar?

  • Slow, clicking hard drives that seem to blink and thrash for a long time.
  • Listening to the “whirring” sound while waiting for your drives to wake up or spin up so you can work.
  • Bad sectors, corrupt disk surfaces, crashed heads, defragmenting.

With SSD drives: all gone. SSD drives have no moving parts. There is nothing spinning, nothing moving, no heads, no mechanical parts to wear out, no surfaces to get corrupt. It’s flash memory, like what is in your mobile phone and many other contemporary devices. And: It is markedly faster. If you have a late 2013 MacBook Pro, you are already using it (these models come standard with flash memory instead of moving hard drives).

Case in point: I cloned my existing, old fashioned hard drive (it was one of the faster Western Digital black drives, so not slow to begin with). I upgraded the main drive in my computer that has the operating system with an SSD drive (an exact clone, no reinstall of the operating system or software). Check out these before and after times:

SSD Turbocharging

TaskOld Hard DriveSSD DriveImprovement
Boot to login screen76.0 sec41.0 sec46% faster
Login to desktop105.0 sec45.0 sec57% faster
Start PhotoShop CS 201436.0 sec8.3 sec77% faster

This improvement came form absolutely nothing else except swapping out my old spinning-platter hard drive with SSD drives. Pretty remarkable. Now consider what this type of SSD speed boost can do for loading your camera raws, Lightroom catalogs and PSD files. Yup.

Now consider what this type of SSD speed boost can do for loading your camera raws, Lightroom catalogs and PSD files.

Or in our case, speeding up your backups. Read on for more.

And it’s not just the speed: since upgrading to SSD: corrupt drives, crashes, and random glitches have disappeared. Now, SSD drives can of course fail as any computer component can, but since there are no moving parts, the chances are a hairline fraction of what they used to be with traditional hard drives. Even the need for RAID (drives that keep copies of your data for each other in case one fails) has virtually gone away.

Dual Memory Cards: It Starts at the Point of Capture

The dual card slots on my Nikon D4 (XQD and CF).

The dual card slots on my Nikon D4 (XQD and CF).

The safety of your precious work starts at the camera. In older days, only professional camera bodies contained dual memory card slots capable of recording a backup copy of each raw images file at the point of capture. Now, many more bodies including mid-level and lower-end camera bodies contain two memory cards slots. Sometimes there are two of the same kind (such as two CompactFlash/CF cards, sometimes there are two different kinds such as on CF card, and one SD card).

Both of my main camera bodies posses two card slots: My Nikon D4, and my D800. If you need one reason to upgrade your camera body, never mind gaining additional megapixels. I would argue that dual card slots is much more valuable to your business than additional megapixels.

If you need one reason to upgrade your camera body, dual card slots is it.

When I return to the studio office, I have two complete copies of every raw file. Bam: just drop kicked corrupt memory card danger. The statistical likelihood of the same image file becoming corrupt on two memory cards is dramatically lower than with just a single card (barring of course, a problem with the body, or dropping the entire camera in a bonfire and things like that).

Dual card slots on my Nikon D800 (CF and SD)

Dual card slots on my Nikon D800 (CF and SD)

What do do now? Let those nice comfort-inducing memory cards sit on your desk for a few days until you have the chance to scan through the entire set and ensure your images are in tact. I love knowing that what I came home with on the wedding day is still sitting right there.

 

The Memory Card Backup

But what if you a shoot the next day and need to erase your memory cards? Enter this little guy. This is a 512 GB Nexto DI unit that inhales the contents of my memory cards at high speed and hangs onto the exact copy of the original captured raw memory card. If I then need to reformat the cards, I can do so even if I have not had time to verify all images before the next shoot.

The Nexto DI creates a verbatim copy of each memory card before they are reformatted.

The Nexto DI creates a verbatim copy of each memory card before they are reformatted.

Bonus tip: One of my second shooters had a memory card come off her camera corrupt a few years ago (no dual card slots). My computer and hers could not read the card, even though I tried two different card readers. To our surprise and delight, the Nexto drive could read the memory card accurately. I was able to obtain all images off the card by just plugging in the Nexto into the USB slot on my computer.

Of course, the Nexto has other handy benefits. I can copy cards in the field and store them in the room safe or house safe at a hotel (so my only copy of the images is not sitting on the cards in my car or hotel room). There are a number of great competing products (and newer Nexto models) available.

If I run out of space on the Nexto device, I purge the oldest copies of each job to free up space (which I have usually long since verified).

Primary Data Drives

IMG_0466_scn

Now that the memory cards are safely backed up, I copy them into my computer and onto a primary data drive. My newest, fastest external drive is my Drobo Mini. This little guy is a pretty slick invention and has served me very reliably for the entirety of last season. It is seamlessly expandable, supports the fastest SSD drives and has Thunderbolt and USB3 connections. When it fills, I can just open the front door and shove another drive into it. The Drobo automatically recognizes the new drive and starts using it. It has four slots in it. I have two 1 TB SSD drives inside of mine currently. The 1 TB drives are expensive but having headroom and speed is worth it.

 

drobo_drives

The Drobo Dashboard software that comes with the Drobo Mini shows the health of the drives inside and the amount of free space left.

 

IMG_0468_scn

The Drobo has my latest sessions and weddings on it. I also have an older external drive (looks more like the brick doorstop drives we are used to) that has older jobs (and some jobs once they are finished on the Drobo get moved to this one). This has two Western Digital Red drives inside it, 3 TB each.

As a sidenote, I used to only buy Western Digital drives. The Red drives are business class and provide the best reliability I have seen. Most folks who have problems have bought the Green drives, blue drives, or a similar consumer level product.

Even though this is an older, more traditional enclosure with spinning hard drives in it, I still have never had any sort of problem with it.

 

 

Primary Backup Drives

IMG_0483_scn

My primary backup drives are just store-bought Western Digital USB 3 external drives like this one. This 4 TB model has been doing pretty well the last year or so. Pretty vanilla, but they have a good amount of space and the USB  3 connection is very fast on a good computer. I have two or three of them for backups. Since these are just backups and not primary drives, they are regular old hard drives (as of this writing, the largest SSD you can buy is 1 TB and as I mentioned above, they are expensive). So we really don’t need that for the backup drive.

On this drive I keep the backup archives from the backup software (described below) and often an exact copy of the entire directory for each shoot. I call this my “safety net”. If all else fails, or my culling accidentally deletes an image I needed, I can go back to this safety net. It is very similar to the copy of the memory cards I made above, except it is connected to the computer and does not need to be deleted as quickly to free up space.

The Backup Software

How do my backups get onto the backup drives? I have been using NovaStor NovaBackup and its various incarnations for the last 12 years and has served me very well. I use Full, then automatic Incremental nightly backups of the files that changed just on that day.

The wedding that I shot last Saturday? That baby needs to be tucked nicely in bed.

I chose to keep a separate job for each year’s work to keep things clean and organized. I find this works well, since I don’t really need my work from 3 years ago to be aggressively backed up since it is not likely to change much. However my lifeblood flows through my current work. The wedding that I shot last Saturday? That baby needs to be tucked nicely in bed.

Each backup job is scheduled to run automatically overnight.

This level here gives me automatic nightly incremental backups of all work that I did on each day. It’s a “lights out” approach. I can stop work and go to bed and know that my files will be backed up. I get emailed in the morning if any backup fails so know if any issues exist.

This is a Windows/PC app. I use a high end Windows 7 desktop as my primary desktop computer, a late 2013 MacBook Pro for my laptop, and an iPad Air 2, iPhone 6 for my mobile devices. I know what you’re thinking. I will explain in another post 🙂

Offline Backup

Everything I have covered thus far might be well and good but several of the enemies of our data can still easily pluck them from our secure hold.

Both the primary and backup drives that I have mentioned are connected to the computer. What does this mean? They can be zapped by anything that affects my computer: human error, viruses and malware, an electrical surge, or even fire and theft in my office.

The WD Passport drives hold 2 TB each, use fast USB 3 connection, and have a handy clamshell case for easy transport.

The WD Passport drives hold 2 TB each, use fast USB 3 connection, and have a handy clamshell case for easy transport.

Enter the “offline backup“: a copy of your data that is not connected to anything electric or electronic. There are no wires and no way to access it except but to have physical access in my office or studio. For this purpose, I have two tools: small portable hard drives and a waterproof fire safe. Don’t be fooled by inexpensive firesafes: a high pressure firefighter’s water hose can blast through the seam and fill the inside with water. Ensure your safe is waterproof also.

Inside this safe goes the small Passport drives. The best place for this safe is away from your desk, preferably at the opposite end of your office or house from where your computer is. This way you’ll have the best chance of having the safe survive even if your office succumbs to fire or theft.

There is no need to update the offline backup unless significant new work is present: after a new shoot, or after a heavy day of work. The amount of risk is up to the photographer. The more frequently you update, the shorter distance you have to fall should a disaster of some sort strike.

 Off-site Backup

IMG_0487_scn

These little guys are pocket size and transport nicely.

The final cornerstone to a solid backup strategy is an off-site backup: a copy of your data stored at another geographic site. It is not physically anywhere near your home or office. It should be a completely geographic location: even if your primary location is destroyed completely by fire or theft or natural disaster, you still have a copy of your data off site in a different location.

Any other location will suffice. The further away, the better. I know one colleague who works part time in an office and just brings one of his Passport drives to his office and just keeps it in his desk. One idea is to partner with another photographer in your region to swap drives and keep one for each other. If you have a friend or relative who is willing to let you plop a drive in a drawer or the back of a closet, you can rest assured that even if disaster strikes at your main office or studio location, your data exists elsewhere.

The End State

So where does this leave me? Here is what I have. I now have 7 copies of my files:

  1. A copy of all camera raw files on the duplicate memory cards from my camera’s dual card slot.
  2. A copy of all memory cards on my Nexto device. Now I can overwrite the memory cards when needed.
  3. Primary data: My raws, catalogs, and psd files on my primary data drive. I do my editing and culling work here.
  4. Safety net: An exact copy of all the original files on a second hard drive. Did I accidentally nuke a good image and need to recheck? I go here. Quick and simple.
  5. Primary backup: A backup archive of my work, backed up automatically every night onto my primary backup drive.
  6. Offline backup: a copy of my backup in a waterproof firesafe at my office location, physically NOT connected to my computers in any way (Passport drives). Computer problems can’t touch it: this backup can’t be accidentally deleted  and the drive can’t be fried by an electrical surge.
  7. Off-site backup: A copy of my backups at a different geographic location (Passport drives). If, heaven forbid, my office is rummaged or destroyed by fire or natural disaster, a copy exists at a different physical location, safe and sound.

Herein lies our multi-level backup strategy: Many failures would have to occur in order for permanent data loss to occur. One failure in one of the levels above leaves other copies that can be used.

In Summary:

Backup Strategy for Professional Photographers

Backup Strategy for Professional Photographers

In Conclusion

We have to tel ourselves it’s just a matter of time before files go AWOL. Whether it’s just a quick pull from last night’s backup, or a disaster comes from our backup strategy. Not ever component is necessary and certainly do not need to occur all at once. But growing into a good solid backup strategy is key to weathering a data loss without skipping a beat. As a working pro, we sleep soundly knowing our work and our client’s images are protected. If you want it all like I do (keeping your original raws and backing them up too), a strategy like this one can help!

If you have tips and ideas of your own or would like more info, don’t be shy. Let me know your thoughts below.

-R