Two of Everything, One Point of Failure

Engineers love pairs. Two ISPs, two power feeds, two DNS providers, the word "redundant" goes on a diagram, the auditor nods, and the incident review closes with a satisfied checkmark. Nobody asks what sits underneath the boxes.

A diagram shows logical paths, and logical paths get drawn by people who have never picked up a shovel. Take the classic pair of diverse internet links from two different carriers. It sounds solid: two contracts, two account managers who have never met. Then a construction crew hits a single conduit under one road, and both carriers turn out to lease space on the same trunk fiber, because building a second trench under that stretch of asphalt cost money nobody wanted to spend. Everyone had signed off on "route diversity" in a vendor questionnaire, and the questionnaire was telling the truth as far as anyone had bothered to check. The redundancy was real on the invoice and fictional in the ground.

Power plays the same trick with more drama when it fails. Two feeds into the building, two breakers on the wall, both tracing back to the same substation three streets over. That setup survives a blown transformer inside the building without blinking. A fire at the substation takes out both feeds at once. Somebody verified the building had two power inputs. Nobody walked the wire back to the point where two became one.

The audit that stops at the box

Most redundancy audits stop at the box on the diagram and never touch the world behind it. A diagram labeled Provider A and Provider B satisfies the checklist. Whether their uplinks share a rack, or even a city, rarely comes up, because answering it means phone calls, paperwork, and occasionally a finding nobody wants written into a compliance report. The vendors themselves are usually the only source for that diagram, and a vendor rarely volunteers the shortcut it took to save money on trenching.

DNS is where this turns almost funny. Two services can look fully independent, with different vendors and different login pages, and still resolve through the same upstream DNS infrastructure sitting underneath half the internet. The client list of a big DNS provider often includes services that market themselves against each other. Nobody checked, because checking meant admitting the two vendors both outsource to the same quiet giant nobody had heard of. When that layer has a bad day, both of your "independent" dependencies go down in perfect sync, and the postmortem discovers a shared ancestor nobody had drawn.

Then there is the SPOF with a heartbeat. Somewhere in almost every organization, one person is the only one who knows how to renew a particular certificate. The private key lives on a specific laptop. The renewal process is a ritual only they remember, written down nowhere useful. That person taking a two-week vacation is a known risk. That person accepting a better offer in March is an incident with a due date. No amount of server redundancy fixes a process with exactly one human in it.

The gaps repeat because they are boring and structural, and that is exactly why they survive audit after audit: nobody gets promoted for tracing a fiber route through three subcontractors to find one shared duct, and nobody puts "asked the DNS vendor's vendor who their vendor is" on a slide. Digging past the diagram risks finding something you now have to explain upward, and that alone keeps most audits comfortably shallow.

If you want to test redundancy for real, stop reading diagrams and start reading trench maps and org charts. Ask your two carriers for their last-mile routes and see how much overlaps. Ask your power company which substation feeds which feed. Ask who your DNS provider's DNS provider actually is. And find out today what happens to your certificates when the one person who understands them takes a better offer in March. Everyone finds this out eventually. The only choice is whether it happens on a calendar reminder or during a pile of expired-certificate alerts at 2 a.m.