I have accumulated terabytes of family photos and videos and I will continue to add more. Storage is cheap and getting cheaper. However, it turns out that managing terabytes of data isn't trivial. At least one of my hard disks have failed each year.
How do I store the data? How to back them up? How to minimize the chance of loss for a low cost?
Well, here's my latest "simple" solution.
- One external Thunderbolt four hard disk case containing:
- One hard disk for all my data (DATA)
- One hard disk for cloning (DATA-CLONE)
- One hard disk for Time Machine (TM)
- One hard disk for Crashplan (CP)
- One external hard disk double dock.
- Several bare hard disks for offline clones of DATA. (All named DATA-OFFLINE)
- A few bare hard disks for offline clones of CP. (All named CP-OFFLINE)
- Time machine backup of DATA to TM.
- Crashplan backup of DATA to CP.
- Four times a day clone of DATA to DATA-CLONE using Carbon Copy Cloner (rsync wrapper).
- Clone DATA to DATA-OFFLINE if DATA-OFFLINE is plugged in.
- Clone CP to CP-OFFLINE if CP-OFFLINE is plugged in.
- Regularly, I'll plug in a DATA-OFFLINE disk or a CP-OFFLINE disk into the hard drive dock.
- Occasionally, I'll move a DATA-OFFLINE or CP-OFFLINE disk to an offsite location.
- Occasionally, checksum all the files.
This solution satisfies my requirements:
- Not cloud based.
- Survive two or more simultaneous hard disk loss.
- Fast recovery.
- Do not cause more loss during recovery.
- Survive single site loss.
- Survive accidental or malicious data corruption or loss.
- Survive software bugs in a backup application.
- Manageable without a Computer Science degree.
Here are some stuff I've learnt along the way:
- It takes more than 10 hours to copy 2TB of data using USB3 or Thunderbolt. Good luck if you are still stuck with USB2.
- Get a much larger hard disk than you need. If you have 2TB of data, get 4TB hard disks.
- rsync is your friend. It can incrementally clone your data. It is a time-tested workhorse. CCC is just a nice wrapper around rsync.
- Time machine provides versioning. But it uses hard links and doesn't efficiently handle renaming of files or folders.
- Don't try to clone a Time Machine disk. Time Machine now supports multiple backup disks. Do that instead.
- Crashplan also provides versioning. However, it is harder to recover data from Crashplan.
- If you have an up-to-date clone, recovery is easy. Just rename the clone as the new data disk. Then update another clone. Incremental updates should be fast.
- If you swap in a clone for the data disk, you have to tell Time Machine to ignore the GUID mismatch. Otherwise, Time Machine will copy everything again.
- Do not store your data on your internal hard disk or SSD.
- Do not use RAID.
- RAID is for availability, not backup.
- Using RAID correctly is hard.
- Reconstructing a RAID array can trigger further failures.
- RAID reconstruction can fail due to latent disk errors.
- RAID reconstruction is slow.
- The hardware RAID card/device is a single point of failure.
- Hardware RAID devices may decide to break your RAID setup for unknown reasons.
- You may not be able to buy a replacement RAID card/device.
- You need RAID compatible hard disks.
- Software RAID can also break due to temporary slow drive problems or cable problems.
- Do not use network attached storage.
- Lower priced NAS have slow processors and limited memory.
- Gigabit ethernet is limited to about 100MB/s. A modern hard disk easily achieved 170MB/s.
- Most NAS use proprietary RAID implementations. If the NAS box fails, good luck.
- If not using RAID, why use a NAS?
- It is yet another firmware to update.
- My NAS developed a memory problem and randomly corrupted my pictures. There were no hardware checks.
- Do not use USB2/3 external hard disks.
- USB cables become lose or unplugged easily.
- Each external hard disk also needs a power supply.
- USB hubs do not work with external hard disks.
- Many USB external hard disks do not have cooling fans. They run very hot and they die. The same models run well in a proper case with fans.
- You can't read SMART status and hard disk temperature through the USB interface.
- Both Seagate and WD use usb-sata adapter that "change" the geometry of the hard disk. If you remove the hard disk from the case and use it with a dock or plain SATA cable, you won't be able to read the data. The partition table is probably shifted.
- Do not use external dual hard disks like WD My Book Duo. I've run into total data loss firmware bugs due to the RAID adapter.
- Use bare 3.5" hard disks. You get to select the brand and model of the hard disks.
- Use a multi-disk external casing with proper cooling.
- Use Thunderbolt. You can read the SMART status and temperature.
- Switch to SSD when prices come down.
- 2.5" disks are slower but run cooler and more silently.
I have intentionally omitted detailed recovery processes. I hope this helps somebody. Your mileage may vary.