Incident Response, NOG Style: No SLAs, Just Reputation

Something breaks at 2am. A peering session drops half its traffic, or a customer link goes dark and nobody upstream will own it. You open a ticket. It sits.

Somewhere in that chain sits a network you've never spoken to, staffed by people you've never met. No contract compels them to answer you. There's no regulator who can make a phone ring on the other end, either.

That's the part outsiders rarely get about network operations once something crosses organizational boundaries. People imagine the internet stays up because of protocols and governance bodies. Mostly it stays up because a few thousand engineers across a few hundred networks know each other well enough to text at 2am.

I've watched this play out enough times that it stopped surprising me, though it still feels a little absurd. A ticket to a transit provider's support queue can sit for an hour while someone works out your tier and whether this counts as an emergency by their internal definition. A message to someone I met at a conference years ago gets a reply in minutes, because they recognize the name and know I don't send messages like that for nothing.

That relationship wasn't built for this. It came out of a shared complaint about a broken CLI once, or years of small talk at the same conference, until a name became a face became someone you'd trust with a 2am message. None of it was planned. It just accumulates, and on one specific night it turns out to be the fastest route to a fix.

The real escalation path

Ask people who've run incident response across network boundaries what actually happens first, before any ticket gets acknowledged, and most describe some version of the same thing: who do I know that can reach someone over there. The NOC contact on the website routes to voicemail half the time anyway. What works is a person you already know. Sometimes a former colleague who switched companies. Sometimes someone half-remembered from a mailing list years back. You message someone you trust, they message someone they trust, and before long the right engineer is looking at the same traceroute you are.

None of that chain works without everyone in it having a reputation to protect. Page someone at 2am over something trivial and you spend trust you might need later for something that actually matters. You learn where that line sits mostly by watching other people misjudge it. There's no formal penalty for escalating around the normal channels. The cost is social, and it gets paid out in how fast people answer you next time.

I wrote before about how this kind of trust gets built, mostly at the same events, over the same years, for reasons that have nothing to do with any specific outage. Incident response is where it gets spent. Usually at the worst hour, under real pressure, with someone else's uptime on the line.

What unsettles me a little is how little holds this up structurally. It depends on a fixed, fairly small set of people staying reachable and staying willing. Every retirement quietly deletes a few of these paths. Every career change scrambles a few more. Nobody replaces them on any schedule, new engineers build their own version of this network the slow way, one favor and one shared incident at a time, and there's a real gap while that happens.

I don't think there's a fix for this, and I'm not sure I'd want one. Formalizing the goodwill would probably kill the part that makes it work. People show up for each other because it's personal. A contract wouldn't get the same phone answered at 2am.

Most nights it's enough. There's no backup plan for the night it isn't, and I don't think anyone's in a hurry to write one.