Teaching NOGnet to Watch Itself

NOGnet runs without a team. There's no on-call rotation, no second engineer to hand things off to, just me and whatever hours are left over once the rest of the week is accounted for. Most of the time that's fine. People log in, matches get made, sponsors get their booth traffic, nothing needs me. Then something breaks in the middle of an actual event, at an hour that has nothing to do with my own time zone, on the other side of the world, and I'm asleep.

I can't solve that by being more disciplined about checking my phone. I've tried keeping the laptop nearby during events and it doesn't survive contact with actual sleep. So instead I built something that watches the platform for me and, within strict limits, does a bit of the fixing itself before I've even opened the laptop.

It's a background check that runs every few minutes and looks at three things: whether authentication is behaving normally, whether server errors are spiking in a way that looks like a repeated failure rather than noise, and whether the IP-ban and rate-limit lists look sane. Nothing exotic. Mostly the same things I'd glance at manually if I happened to be awake and a little paranoid.

The part I actually like is what happens after three authentication failures in a row. The watchdog does two things at once. It pages me, so I know something is wrong no matter what happens next. And it hands the error signature to a small language model I run on my own hardware at home, and asks it to take a guess at what's going on.

That model isn't improvising. It checks the error against a catalog of failure conditions I've written down ahead of time: a missing database column after a schema change, a file-permission problem, a database connection that's gone stale and needs to be dropped and reopened, a corrupted local client cache. Ordinary stuff. The kind of thing that's tedious to diagnose at 3am and worse to diagnose off a phone screen half asleep.

For a defined subset of those, the ones I'm confident about, it applies the fix itself. For anything outside that list, it writes down what it thinks is wrong and stops there, just a note waiting for me when I wake up.

That subset is small on purpose. I keep going back and forth on whether to let the model use more judgment about what counts as safe to touch, and I keep landing on no. The list is short, hand-written, and reviewed by me, not something the model infers on its own from context. I'd rather it stay conservative and leave something for me to fix by hand once in a while than get clever about a failure mode I haven't already thought through and signed off on.

One rule in the watchdog came from a mistake elsewhere in my setup, not from NOGnet itself: demo and test accounts are only ever allowed on a separate test deployment, never on the production one. Written down like that it sounds obvious. It wasn't obvious in the moment it mattered, which is why there's now a flag whose entire job is making sure that boundary never quietly drifts again.

The overall effect is a little funny and a little unsettling. I get paged by a piece of software about a problem in that same piece of software, and by the time I've picked up my phone and read the message, it's sometimes already fixed the thing it just told me about. I didn't plan that exact sequence when I started building this. It just fell out of running the page and the diagnosis at the same time instead of one after the other.

I don't think of it as a step toward some self-managing platform. It's closer to writing down the boring troubleshooting I already do, in a form that can run without me, for the handful of failure modes that are dull enough to deserve that treatment. Everything else still lands on my phone, and I still have to get up.