Sweden, Ireland and Canary Islands

Below are some photos from my autumn 2011 trip…

Valuable thoughts about backing up data — comment in Slashdot

>>> A blind “Let’s just backup the whole server” isn’t an effective backup strategy …

If I was your boss I’d fire you on the spot for being that monumentally stupid. The cost ratio of an additional ~20GB of tape compared the enormous cost to the business if the server can’t be restored quickly and reliably is staggering. Think $1 saved while risking millions of dollars of potential lost work! (1.6TB LTO5 tape = $80) 

  1. “what if you upgrade to a larger disk” — there is no backup system on Earth that can’t restore to bigger disks. Most systems can restore to smaller disks too.
  2. “what if the server is slower than others and people start moving their data to something” — you back up both of them. Restoring too much is almost never as bad as restoring too little.
  3. “just blanket-backing-up is likely to lead to problems later on” — no, it doesn’t. Achieving 100% coverage ensures that no matter where you data was on a server (or which server), it’s on a tape somewhere.
  4. “notifying IT of new things that need to be backed up” — have you met humans? This never happens reliably, and can’t ever be made reliable.

Back in the real world, a 100% complete backup of a typical Windows server can be restored without knowing the password, to dissimilar hardware (even virtual machines), and without needing the “original install disks”. When it’s done, it’ll boot up, maybe reboot once or twice to fix up its drivers, and then your server is back, working as it did before. Compare that to a “data only” or partial backup. Now suddenly you’re chasing down design documentation, passwords, IP addresses, software, serial numbers, and you haven’t even started to restore anything yet. The clock is ticking, and the customer is breathing down your neck. 

>>> A week’s work should take no more than two weeks at ABSOLUTE maximum to recreate

Recreate from what? Memory? Including data that was 100% electronic, and never seen by a human? How do you recreate your emails? How do you recreate your audit logs? How do you type back in non-textual data like digital images or audio recordings? How would you even know what’s missing? 

>>> I’d rather have a decent monthly, than an imperfect daily

That’s a false dichotomy. The total data stored is the same, you’re just altering the frequency. The same amount of storage is needed, the same bandwidth is needed, and it ends up costing the same.
I’d rather have three backups a day than monthly backups. Losing a day of work could mean a contract fails to go through. I’ve been in a position twice now where users have come to me literally crying and begging to retrieve a document they only started working on that morning that they had deleted accidentally… at 8pm, minutes before a deadline for a multi million dollar deal. After experiences like those I’ve often set up incremental backup frequencies as rapid as 15 minutes.
So lets recap… your sum total DR experience is you once walked into a poorly supported environment, and gave them some even worse advice, without ever being in a position to be responsible for an actual real world recovery.
Well, take some advice from someone who’s restored terabytes of data, and was responsible for the protection of over a petabyte spread across thousands of servers at over a dozen organizations:
#1 There are no time machines — you cannot go back in time to fix a mistake in your backup strategy after a disaster. It’s too late. You’ve fu..ed up, it’s your fault, and you can never, ever, fix it.
#2 Back up everything — I love genius IT folk who like to shave 1% off their backup times by excluding those useless temp and log files, also excluding ‘useless junk’ like their database transaction logs in the process. Oops. See rule #1. Pro tip: Your “dev” servers have the only copy of the source code to your business app that cost your organization half a million dollars of consulting hours to write. It needs backing up too. Oh, that infrastructure server has no ‘user data’ on it? Sure, ok, can I unplug it and take it? No? Hard to recreate? You don’t know how it’s configured in the first place? Back it the f..k up.
#3 Restore out-of-place — it’s hilarious when the IT guy decides to quickly fix a small problem by overwriting the last remaining good copy of their data with a completely invalid backup. Also, see rule #1.
#4 Backups can’t be considered successful until restored to an isolated system — I’ve seen KB articles along the lines of “backup reported as successful, but no data written to tape”. I’ve lost count of sites where people think that Active Directory or eDirectory don’t need backing up, because restoring a single network connected member server ‘always worked before’. Sure… if it can replicate from a peer. Try that again without the network cable plugged in, see how well it goes.
#5 Only full restore times matter — got a whizz-bang incremental deduplicating backup system? Good for you. Guess what: all DR restores are full restores by definition. How many weeks did you say that will take? Say goodbye to your business…
#6 Back it up yourself — I really don’t care how many pretty green ticks your all-singing, all-dancing enterprise backup whizz-bang system reports every day. It’s probably broken, and ntbackup will be done in 10 minutes. I brought my own disk, because it’s fast and it works, unlike the tape you’ve overwritten a thousand times, left on the dashboard of your car in the sun, like to throw around and play catch with, and may have spilled coffee on at some point. Oh, and it has data from last night, already ten hours old and counting. My way the latest backup will be complete and ten minutes old when I “break a working critical system and be held responsible for getting it all back up”.

Ironically, it’s never been easier to get fast, full, and cheap backups even for the smallest businesses. Windows Server 2008 and later has a built-in backup system that does both file and image backups. The latter is particularly impressive, because it takes a quiesced snapshot, and backs up everything to a VHD disk image that is both directly bootable on the original hardware or can be used directly as a virtual machine disk or can be mounted as a virtual drive and browsed like a normal disk or can be restored to 100% working condition with any windows bootable recovery disk. Meanwhile, USB3 disk docks [link to shop with USB3 dock] are faster than tape, cost about $100, and take $150 2TB “cartridges” [link to SATA HDD 2TB].

by bertok on Friday November 18 at Slashdot. Again, I could not resist to republish it, Original here.