Six Pillars, One Server Fleet

I built Magatama to watch my own servers, the ones at home and the handful I rent, and to actually do something when something looks wrong instead of just paging me at 3am. It started as one mental model: an agent that gets progressively more autonomous, climbing some invisible ladder from reading logs to fixing things while I sleep. I had a roadmap. It had phases. It looked tidy on a whiteboard.

That roadmap didn't survive contact with the actual work. What replaced it is six strands I now treat as permanent categories, not a checklist: reliability and diagnosis, gated execution, hardening the agent against its own attack surface, observability, alerting, and recovery, the discipline of always being able to undo whatever it just did. None of them finishes. They just demand different amounts of attention depending on the week.

Why split it up at all

A single roadmap hides how differently these problems actually behave. Telling a real failure apart from a metric that just looks alarming is nothing like deciding whether the agent gets to touch something without asking me first. Bundling both under one "autonomy" heading let me convince myself that finishing one made the others less urgent. It never did. I've had stretches where the diagnosis logic held up fine and the gating quietly let something through it shouldn't have, and stretches where it went the other way.

Gated execution is the strand I'm most stubborn about, mostly because I don't trust myself here any more than I trust the agent. Every unsupervised action has to earn its place on the list separately; there's no "it worked once" shortcut. The list mostly gets shorter over time rather than longer, because I keep finding cases where "fine most of the time" isn't good enough for something running against machines I depend on.

The strand about securing the agent against itself is the one that made me properly paranoid, in a useful way. Something with permission to act on infrastructure is also a new way into that infrastructure, and I've spent more hours than I expected imagining what a hostile log line or a crafted alert could talk it into doing. That work doesn't converge to done. It's closer to keeping an eye on a pilot light than closing a ticket.

Observability and alerting used to be one thing in my head and aren't anymore. Observability answers "can I see what happened." Alerting answers "did it tell me at the right moment, and only then." I've tuned the alerting side harder than anything else here, largely by getting paged for things that turned out to be nothing and getting annoyed enough to go fix it.

Recovery is the strand I resisted longest, because admitting you need a clean way to undo an autonomous action feels like admitting the autonomy isn't ready. It isn't fully ready, and I don't think it ever fully will be.

Why it doesn't end

That's the honest reason I think of all six as permanent rather than a project with a finish line. Each one quietly degrades on its own if I stop paying attention, even when nothing else about the system has changed. New failure modes turn up. New ways for the agent to be overconfident turn up. I used to feel a little bad about not having an end date for this. I don't anymore; running infrastructure never had one either, and something watching it over your shoulder inherits the same condition.

I still keep a roadmap document, mostly out of habit at this point. Nobody reads it but me, which is probably the right amount of ceremony for six things that will never be finished.