Rollback Is a Feature, Not an Afterthought

I built Magatama to watch over my own servers because I got tired of finding out things were broken from a user instead of a dashboard. It reads logs, correlates alerts across the fleet, and for a decent chunk of routine problems it just fixes them without asking: restart a hung service, clear a temp directory that's full, rotate a log eating disk. Low-stakes, reversible-in-spirit stuff.

Rollback is where I drew a hard line. The agent can look at a bad deploy or a botched config change and write up exactly what it would undo and why. It cannot press the button. A human has to read the plan and approve it before anything destructive actually runs.

Somewhere in building this thing I started weighing every autonomous action Magatama could take by two questions instead of one: is this safe to do, and is this safe to undo. Most of its toolkit passes both easily. Rollback is the one place where the second question turned out to be harder than the first. That's the whole reason it stayed plan-only.

That line took me longer to draw than it should have. On paper, "undo the last thing you did" sounds like the safest action you could hand an autonomous system. It rewinds to a state that, presumably, already worked five minutes ago. If there's one category of action worth trusting to a machine, reversing its own mistake looks like a strong candidate.

Two things changed my mind.

First, an automatic rollback can eat the evidence of what actually went wrong. Say the agent narrows a spike in errors down to three plausible causes and picks the top one to revert. If it's right, nobody notices. If it's wrong, the revert has just erased the exact state, logs, and process behavior that would have told a human the real cause a few minutes later. Now the system is still broken, minus the clues.

Second, rollbacks aren't as symmetric as they feel. A config revert five minutes after the change is close to clean. The same revert twenty minutes later, after something else on the fleet has quietly started depending on the new state, is a different action wearing the same name. An agent that treats "undo" as inherently low-risk applies that assumption everywhere, including the moment it stops being true, and it has no reliable way to notice when that moment arrives.

What Magatama produces on anything destructive is a plan. It lays out the diff it wants to revert, its best guess at why, the exact order of steps, and whatever else touches that piece of state. Then it stops and waits. I read it, usually on my phone, and approve it or don't. Most of the time it's right and approval takes ten seconds. Occasionally I catch a dependency it missed, and that pause is the reason the gate is there at all.

None of this would feel defensible without backups running underneath it, on their own schedule, independent of whatever the agent thinks is happening. The approval step keeps a human in the loop on judgment calls. The backups exist for a different reason: so a bad judgment call, mine or the agent's, doesn't turn into something unrecoverable. Worst case on a bad approval means restoring from last night's snapshot.

I'll admit the temptation to loosen this is real, usually around 2 a.m. when an alert fires and I just want the fleet quiet again. I've thought about carving out an exception for "obviously safe" rollbacks, a single config file, nothing else touched. So far I haven't gotten around to it, and honestly I'm not sure I will. Most nights the approval step costs ten seconds. It only has to earn its keep once.