Why I Let My Own Agent Say "I Don't Know"

I built Magatama to keep an eye on a small fleet of servers I run myself, check logs, flag anomalies, tell me what's actually wrong before I find out the hard way. A few months in I ran into a design question that took longer to settle than most of the actual engineering: what should the agent say when it genuinely doesn't know?

The easy answer is to make it say something anyway. That's just what language models do by default. Give one a pile of logs and a vague symptom and it will hand you a diagnosis, phrased with total confidence, whether or not it actually found the cause. Silence isn't really in the training distribution. A plausible paragraph is.

When confidence is the failure mode

For a while I let that slide, because a confident answer feels like progress. You get a name of a service and a suggested fix, and it all reads clean. The problem showed up when the diagnosis was wrong. A wrong confident answer doesn't just fail to help. It sends you somewhere. I'd restart a service that had nothing to do with the actual issue, and by the time the symptom was still there I'd burned the better part of an evening before circling back to look at the raw data myself. The agent hadn't been useless. It had been worse than useless, because it spent my attention on the wrong hypothesis while the real one sat there unexamined.

So I added a third answer to the set Magatama is allowed to give, alongside "here's the problem" and "everything's fine": unclear, needs a human look. Sounds like a small addition. It wasn't.

The first version of "unclear" became a crutch almost immediately. Any time the evidence was even slightly ambiguous, the agent reached for the safe answer instead of doing the work to narrow things down. Fair enough, if punting costs nothing and guessing wrong costs something, of course it punts. I ended up tuning the scoring so an unnecessary "unclear" carries a real cost too, just a smaller one than a confident wrong answer. The agent has to earn the shrug. It doesn't get to default to it.

Garbled isn't the same as fine

The second piece had less to do with what the model said and more to do with what happens when I can't get an answer at all. Early on, if a health check timed out or a response came back garbled, the pipeline had two bad options: swallow it and move on, or log it as "no issue found." Both are the same mistake wearing different clothes. A dropped response and a clean bill of health look identical downstream, and downstream is exactly where I'm deciding whether to go to bed or go check something.

So an unreachable or malformed response now gets its own state, separate from healthy and separate from a confirmed problem: degraded. It shows up differently in whatever I'm looking at that day, and it never gets quietly merged into "all clear." If the model couldn't answer, that's information too, and it's usually the kind that matters most right before something breaks.

None of this makes Magatama smarter, and in the normal case it makes it look less impressive, because "unclear" doesn't read like a strong output the way a decisive diagnosis does. But I'm not running a demo. I'm running infrastructure I'd like to keep working, and for that job an agent that tells me when it's guessing is worth more than one that never admits it does. I'd rather get paged with "I don't know" than get a clean-looking report that's quietly wrong and only find out once I've already acted on it.