What a Playbook Actually Looks Like

The pitch for an autonomous ops agent is always the same: something breaks at 3am, the agent looks at it, decides what to do, and fixes it before anyone wakes up. I built exactly that kind of thing for my own server fleet, something I call Magatama, and then spent months making sure it does almost none of that.

The agent watches logs, metrics, and alerts across a handful of machines I run myself. When it sees a condition it recognizes, it can act without waking me up. The word doing the work there is recognizes: the condition has to match a document I wrote in advance, word for word, before the problem ever happened. If nothing matches, it does not act on its own. It escalates and waits.

Writing one down

A playbook starts with the trigger, and the trigger is where I spend most of my time. "Service looks unresponsive" is not a trigger. "Health check on this endpoint returns non-200 for three consecutive checks over ninety seconds, and the process has been running longer than five minutes" is a trigger. If I can't write the condition precisely enough that two different people would agree on whether it fired, the agent doesn't get to act on it, it can only report.

Then the remediation, which is the part nobody wants to hear about because it's tedious to write. The section is a numbered list of exact steps in order, with the exact command at each one, not a summary of what should happen. Saying "restart the service" tells me nothing about which service, which mechanism, what timeout, or what to check afterward. I write each step as though I'm handing it to someone who has never seen this system before and won't get the chance to ask a clarifying question, because that's functionally what's happening at three in the morning.

Every playbook also has to define what "safe" means for its own category, because the definition changes. For some categories, safe means the fix is idempotent: you can run it five times and nothing gets worse. For others, safe means it runs exactly once and then locks until a human clears it, because a second attempt on that category tends to compound the damage instead of fixing it. That constraint goes into the document itself. I don't want the agent inferring it from some general policy that's supposed to cover every case at once.

And every playbook needs a defined failure branch, which is the part I almost skipped early on. The failure branch does not include trying something else, that's exactly the improvisation I'm trying to design out of the loop. The shape is almost always the same: stop, don't attempt a different fix, page me, attach the logs. A playbook without a graceful failure branch is worse than no playbook at all, because it hands the agent a false sense of having a plan.

The reason I insist all of this gets written by me ahead of time, instead of generated by the model in the moment, comes down to confidence versus correctness. A model can produce remediation steps that read well and are wrong in a way that only shows up once you've operated the actual system for a while. A playbook I wrote carries the scar tissue of having thought about the failure case slowly, at a desk, with no alert going off. I don't trust the model to reproduce that under pressure. I'm honestly not sure I'd trust myself to reproduce it under pressure either, which is the actual argument for writing it down before the pressure starts.

None of this scales the way the pitch implies. I write these one at a time, usually after something has already broken and I've cleaned it up by hand once already. The next thought is always the same: fine, this one gets automated next time. The library grows by maybe one or two entries a month, and there's a long list of categories sitting there unwritten, waiting for a slow Saturday. The whole process is tedious in a way "autonomous" is supposed to not be. But the alternative is trusting a model's judgment about my own infrastructure at the exact moment judgment is hardest to check, and that's not a trade I'm making.