The Backup Generator Fallacy

Ask an engineer how resilient their system is, and count the seconds before they mention a spare part. Two generators instead of one. A hot standby database syncing quietly in another building. Redundant fiber paths that, if anyone bothered to check, both run through the same conduit under the same bridge. Redundancy is the thing that shows up on the architecture diagram, earns its own line item in the budget, and makes an auditor nod along. Alone, it's also a bit of a lie people tell themselves.

Redundancy answers a narrow question: what happens if this specific part breaks. It's insurance against a failure mode somebody already imagined, bought in advance and left idle until the day it earns its keep. Useful, and easy to defend at budget time: point at the invoice and say, "see, we bought two." Nobody is arguing against a spare tire.

Resilience answers a harder question, one that doesn't fit neatly on a procurement form: what happens when the thing that breaks isn't the thing anyone planned for. Picture the generator that runs fine but has no fuel, because the delivery contract lapsed six months ago and nobody noticed. Or the failover database that comes up cleanly and quietly starts serving stale data, because the sync job everyone assumed was running had died weeks earlier. Redundancy assumes the failure has already been catalogued somewhere. Resilience is what's left when it hasn't.

What gets funded

Redundancy is also the easier thing to build, and easier things get funded. You can put "N+2 power" in a pitch deck. You can show a diagram with two of everything and watch people visibly relax. It photographs well. Resilience mostly looks like nothing: a team that has practiced failing on purpose and knows how to make a system degrade gracefully instead of falling over completely. It also looks like a habit of saying "I don't know what's happening yet" out loud without it costing you your job. None of that shows up on an invoice. None of it is a SKU. So it loses budget fights to the second generator, every time, even though the second generator has never once been the thing that saved anyone from a failure nobody had catalogued.

The systems that fall over hard, the ones that make the news, are rarely short on spare parts. They tend to be stuffed with them. What they're short on is a plan for the failure that doesn't match any of the spares. Redundancy is built for the failure you expected. The interesting failures are, by definition, the ones you didn't.

Graceful degradation is the tell. A resilient system, hit by something nobody predicted, gets worse in a controlled way: it sheds load, drops a feature, gets slower, and stays up in some diminished but honest form while humans figure out what's happening. A merely redundant system either works at full spec or doesn't work at all, because nobody built the middle setting. That middle setting is expensive to design and boring to talk about at a conference, which is exactly why it rarely gets built.

The people matter more than the org chart wants to admit. A small team that has drilled outages, that trusts each other enough to hand off a burning problem at three in the morning without ego, will out-recover a much better-funded shop that has never rehearsed anything worse than a scheduled fire drill. None of that comes from a hardware invoice. It comes from deliberately breaking things on a schedule and watching what happens, over and over, until nobody panics.

So next time someone tells you their system is resilient because they bought two of everything, ask what happens when both of the two break for the same reason on the same day. If the honest answer involves a group chat full of people yelling and nobody in charge, congratulations: you've found a very expensive way to be fragile.