Red-Teaming My Own Autonomous Agent

Magatama is the agent I built to watch my own server fleet. It reads alerts, proposes a fix along with the exact playbook it wants to run, waits for me to confirm, then executes and logs everything. The design was supposed to give it enough autonomy to be useful and not enough to do real damage on its own. I trusted that design for a while. Then I decided to stop trusting it and go find out where it was wrong.

So I ran a real audit against my own automation, not a quick pass. I split the review across independent reviewers, each covering a different risk dimension, six in total, and had them cross-check each other’s findings adversarially before anything counted as confirmed. A finding didn’t make the list because one reviewer thought it looked bad. It had to survive someone else trying to argue it wasn’t real. The goal wasn’t “find bugs” in the usual sense. It was narrower and less comfortable: find every way the agent’s own autonomy could be turned against me.

Five things came out of that process as confirmed and worth fixing immediately. All five got fixed the same day, and every fix moved the system toward being more restrictive, never more permissive. That direction mattered to me as much as the fixes themselves.

1. A quiet path to the cloud. If enough of my local model endpoints went down at once, a cascading failure, the agent’s model-calling logic had a fallback chain that could eventually reach an external cloud AI service. Nothing malicious had to happen for that to matter. My own internal alert data, the stuff describing what’s actually wrong on my machines, could have left my hardware without me ever deciding that should happen. I fixed it by making local-only the hard default and adding a second local fallback option, so the chain gets two more chances to stay home before it would ever reach a cloud provider.
2. A filter that only watched one door. The agent scans incoming untrusted text, alert descriptions, threat feed content, for prompt-injection attempts before handing it to the model. That filter existed and worked fine. It just wasn’t wired into the automated triage-and-fix path that actually runs unattended, only into a manual diagnostic endpoint I use by hand. The path that matters most, the one running while I’m asleep, was passing raw untrusted text straight to the model. I moved the filter to the front of that real path.
3. Letting the model pick its own target. During triage the model writes a short text field naming which system component looks responsible for a problem. That text was then trusted downstream to pick which actual service got acted on: restarted, firewalled, reconfigured. Put plainly, the model’s own words could choose what happened to my infrastructure next. A poisoned external feed could, in theory, have talked the agent into taking down something that was never actually broken. Now that path is only trusted when the finding comes from a scanner I control internally, never from anything sourced externally.
4. A liveness check that failed in the wrong direction. One check exists purely to ask whether an issue is still a real problem before closing it out. If the model’s answer came back garbled or clearly off, the check defaulted to treating the issue as resolved. That is a fallback that fails silently in exactly the direction you don’t want: real, open problems getting marked closed because the answer looked strange. I flipped the default. A garbled answer now leaves the finding open for me to look at.
5. An open door in a settings page. One page let me configure URLs for internal integrations. Nothing stopped that field from being pointed at a cloud metadata endpoint, a well-known way to get a server to leak credentials it holds about itself. Server-side request forgery, in plain terms: tricking a server into making a request on your behalf that it should never make. That door is closed now.

If I had to pick the one that mattered most, it is the first: forcing all of the agent’s own reasoning calls to stay local by default, with a second local option before cloud is ever on the table. It closes the data-leak risk directly. It also makes the system more resilient, since staying local means it does not depend on an external service being reachable at three in the morning when something is on fire. Fixing the security problem and fixing the reliability problem turned out to be the same fix.

I will not pretend this was comfortable. You build something, you give it real permissions on real machines, and you convince yourself it is careful because you were careful when you wrote it. Then you go looking specifically for ways it could be tricked into working against you, and you find five of them in one pass. More than half of the five trace back to untrusted external text somehow reaching a decision that touches real infrastructure. That is the actual shape of the risk with an agent like this: something it reads convinces it to take an action that looks reasonable and is wrong.

None of this happened in production, which is the part I care about. I would rather find these on my own terms, with a room full of skeptical reviewers trying to break my own reasoning, than find out because some external feed I do not control got a little too clever. The gated design is still doing its job: propose a fix, wait for confirmation, execute, log everything. This round of review just checked something one layer deeper than the gate itself, whether the reasoning behind each proposal could be trusted in the first place.