Your Backup Job Is a Liar (It Just Doesn't Know It)

Somewhere on a dashboard right now sits a job that ran last night with a green checkmark next to it. "Backup completed successfully." It has said exactly that every night for six years. Nobody has looked at it with real suspicion in ages, because a green checkmark exists to end curiosity, not invite it.

Then a script with write access it should never have had truncates a production table, or a ransomware note shows up where the file server used to be, and someone finally goes to pull the thing that checkmark was supposedly protecting. What they get back is a file that won't mount, a database dump three schema versions out of date, or a restore that technically works but reports an ETA of fourteen hours against a recovery time objective that promised four. None of those outcomes triggered an alert. All of them are backup success stories, right up until the moment they aren't.

The checkmark measures the wrong thing

A backup job succeeding tells you that bytes moved from one place to another and nothing threw an error along the way. Fine, that's real information. It is also a far smaller claim than most people give it credit for. It says nothing about whether those bytes are complete, whether they're internally consistent, or whether anything on earth can still read that format in five years.

Between "wrote successfully" and "restored successfully" lives an entire ecosystem of quiet failure: silent bit rot nobody catches because nothing ever tries reading the file back, a schema mismatch between the dump and the current app version, a backup license that lapsed eight months ago without anyone noticing, credentials that got rotated in the vault but never updated in the restore runbook. None of these produce a red X. Every single one of them shows up during an actual incident, which is the worst possible time to learn about any of it.

Two teams can point at an identical unbroken streak of nightly successes and be standing in completely different realities. One ran a full restore last quarter with a stopwatch running. The other has never once pulled that trigger and is, in the plainest sense of the word, hoping.

Hope is not a recovery objective

"We have backups" gets said in planning meetings the way "we have insurance" gets said by someone who has never filed a claim. It sounds like an answer. It functions like a shrug. The honest version of that sentence is closer to "we have a process that writes data somewhere and we assume it comes back intact," and nobody puts that on a slide, because it invites exactly one follow-up: when did you last restore from it, and how long did that take?

If nobody in the room can answer with a date and a number, the recovery plan is a belief system wearing a policy document's clothes. Auditors sometimes catch this eventually. Incidents catch it immediately, and charge interest.

Test the restore, not the backup

The fix is unglamorous, which is probably why it keeps getting skipped. Restore something. On purpose, on a schedule, before an incident forces the question. Pick a system, pull its backup, spin it up somewhere isolated, and time the whole exercise with an actual clock instead of an assumption. Start with the boring, forgotten system rather than the flagship one, since the boring system is exactly where nobody has looked since the day it was built.

Then hold the resulting number next to the RTO written in the BCM binder. A four-hour promise sitting next to a fourteen-hour reality is not a rounding error. It's evidence that the plan was fiction for as long as it went untested, dressed up in official formatting so nobody would ask.

Run that drill often enough and the drama drains out of it, which is exactly the goal. A restore that's routine will behave itself on the night it actually matters. A restore that's never happened once is a coin flip in a very expensive suit.

Keep writing those bytes every night. Just retire the sentence "we have backups" and replace it with the only version that means anything: "we restored this on this date, in this many hours, and it worked." Everything shorter than that is a hope wearing a checkmark.