Watching the Watcher: Observability for an Autonomous Agent

Magatama watches my server fleet so I don't have to stare at dashboards all day. It pulls in advisories, log anomalies, the usual pile of things that need triage before they turn into a real problem, and filters all of that into findings I can act on quickly instead of digging through raw feeds myself. That part has worked well for a while now. What I hadn't thought through was what happens when Magatama itself stops working properly, quietly, without anyone noticing.

Here's the failure that made me think about it. One of the external feeds Magatama pulls security advisories from just stopped updating. No error, no crash, nothing that looked like a fault. The endpoint kept responding normally, the poller kept running on its usual schedule, and Magatama kept reporting that there was nothing new to flag. Which looks identical whether the world has genuinely been quiet or the feed died a while back and nobody checked. I only caught it by accident, cross-referencing something manually for an unrelated reason. That's not a monitoring strategy. That's luck, and I don't love building on luck.

So there's an obvious question sitting underneath this: who watches the watcher. And right behind it, a slightly more annoying one: who watches that. Left alone, this turns into the kind of problem you can chew on at your desk for way too long: layers of monitoring stacked on more monitoring, each one theoretically able to fail in the same silent way as the layer below it.

I don't actually think the regress goes on forever, in practice. It only feels infinite if you assume every layer carries the same risk and is equally hard to verify some other way. That assumption is usually wrong. Most of the real risk sits in one or two specific spots, and once those are covered, each additional layer you bolt on buys you less.

For Magatama, the dangerous spot was never "is the agent alive." I'd know about that fast, the whole fleet would stop getting checked, and that kind of absence is loud on its own. The dangerous spot was the opposite kind of failure: a dependency going quiet in a way that's indistinguishable from nothing being wrong. A crash announces itself. A feed that stops updating just sits there looking completely normal, and normal doesn't page anyone.

What I built is narrow on purpose. Each external feed Magatama relies on for its own decisions gets an expected update cadence. If a feed goes past that window without producing anything new, Magatama raises a finding on itself, using the same severity-tagged format it already uses for everything else it watches. No new alerting system, no second agent watching the first. Just the existing findings pipeline, pointed at one more category of problem: silence.

I did think about going one layer further, something that would tell me if the heartbeat check itself stopped running. I talked myself out of it. The heartbeat logic is small and rarely touched, so its most likely failure mode is me breaking it during a deploy, not it quietly rotting away over time while I'm not looking. I already review deploys carefully. Long, unnoticed rot was the actual gap, and that's specifically what the heartbeat check now closes. Stacking another layer on top of it would mean defending against a risk I'd mostly invented to justify building more, rather than one I'd actually watched happen.

That distinction is the part I keep having to relearn. When a failure can go unnoticed for a long stretch, it's worth a dedicated check. When the check's own failure mode is something normal review would already catch, adding a second check on top mostly just makes the architecture diagram longer and gives me one more thing to maintain. "Add more monitoring" always feels like the responsible move in the moment. Sometimes it's just anxiety wearing a tool belt.

The fleet dashboard looks the same as it did before any of this. The only real change is one new category of finding: feed silence. It's fired once so far, on a feed I'd already half-suspected of being flaky. Nothing much to show for it visually. It's a small, boring catch, the kind that only exists because I stopped one layer short of where the anxiety wanted me to keep going.