The Ghost Process and the NaN Weights

I caught it by accident, which is how you catch most things worth catching. I was checking on my actual training pipeline, the one job I expect to be running at any given moment, and the process list had two entries where there should have been one. Same kind of job, different process, doing its own thing on its own schedule. Not one I’d started. Not one I recognized at all.

I killed it. Seemed like the obvious move when you find something you didn’t start touching a model you care about. Twelve minutes later it was back. Not from anything I’d done, I hadn’t logged in again, hadn’t typed a command that could explain it, which is the part that actually bothered me. A process that restarts on its own after you kill it means something is spawning it, and I had no idea what that something was.

Without a real answer, I did the blunt thing: renamed the old script’s file on disk so nothing could execute that path again, even by accident. Ugly fix. It worked immediately. I moved on, half satisfied, telling myself I’d find the actual cause later.

Later arrived faster than I wanted. A few days on, trying to redeploy a model the normal way, I found an entry sitting in my GPU queue marked “waiting,” owned by something that wasn’t any script I’d run that day. I pulled the thread and found a second old service, entirely separate from the first, running quietly on its own port like it had always belonged there. It had kicked off its own training run that same morning, also outside the pipeline I actually trust.

At that point I stopped patching symptoms and went looking for the source. It turned out to be an old scheduled job, a weekly retrain set for two in the morning on Sundays, left over from an earlier version of the system that I’d replaced and, apparently, forgotten to fully decommission. Nobody had removed the entry when that version was retired, so it had just sat there, doing exactly what it was configured to do, for who knows how long. I killed both running processes, backed up the old schedule file just in case I’d misjudged something, and deleted the entry.

I’ve repeated the same line to myself more than once since: no training happens outside the one official pipeline, full stop. Written down it sounds obvious. It clearly wasn’t obvious enough, or I wouldn’t have had two zombie processes quietly retraining a production model on a schedule nobody set on purpose.

That would have been enough for one week. It wasn’t the whole week.

Around the same time, I was chasing a different problem: an automated alert-diagnosis pipeline, the small model that reads system alerts and explains what they mean, kept failing in ways that didn’t add up. I went looking for why and found something almost embarrassingly simple first. The model hadn’t actually been redeployed after its most recent training run. A step that’s supposed to happen automatically just hadn’t, for days, without a single error logged anywhere I’d normally check.

So I looked at why the deploy step hadn’t fired, and that’s where it stopped being simple. The training run behind it had gone completely off the rails partway through: every one of its 224 weight tensors had turned into NaN, not-a-number, the signature of an optimizer blowing itself up on the final step. The system had caught this. It flagged the run as failed internally, exactly the way it was supposed to.

And then it shipped the broken weights anyway. The fallback logic was built to fall back to the best checkpoint from earlier in the run if the final one looked bad. When it couldn’t find a good one, its answer was to ship the final weights regardless, on the theory that something is better than nothing. Nobody had accounted for the case where the final weights are actively poisoned. The file landed in the exact spot a healthy model would have landed in.

Here’s the part that still gets me. I’d already, separately, kicked off a manual deploy of that same model earlier, unrelated to any of this, just trying to get it live. That job was sitting in the shared GPU queue waiting its turn, ready to push those exact broken weights the second the GPU freed up.

I killed it just before it got there. Not minutes of margin. Seconds, by my own rough count, working backward from when I noticed to when the queue would have cleared. Afterward I loaded the weight file myself and checked every single value for being a finite number, mostly out of morbid curiosity about how bad it was. Every value failed. Not most. All of them.

I closed the gap the only way that made sense: a hard check, at every point a model can ship, that refuses anything containing a NaN or non-finite weight, no exceptions, no “ship it anyway because something is better than nothing.” Then I kicked off a clean training run to replace the corrupted one and left the broken file where it was, as a reminder to myself.

It’s a genuinely funny story now: two rogue training jobs and a near-miss deploy of a completely broken model, all inside the same stretch of days. It was not funny during the ten minutes it took to find that queued deploy job and kill it. My hands were not steady.

Running your own training infrastructure means being your own on-call engineer, your own QA team, and your own postmortem author, often within the same afternoon. Nothing here was exotic: a leftover scheduled job entry, a fallback path with an unhandled edge case, both fixed within a day of finding them. What stuck with me is how little warning either one gave. No alert, no red banner. Just a process count that didn’t match what I expected, and a queue entry with an owner I didn’t recognize. I check both by hand now, every few days, on top of whatever automated checks I’ve since added.