The Corrupted File That Looked Fine From the Outside

LLM Gym runs training jobs overnight, mostly unattended. A fine-tune on one of the small adapters chugs along for a few hours, and every so often it writes a checkpoint to disk so a crash doesn't cost the whole run. That part has worked for months. The part that didn't work is the part I want to write about.

One night the machine restarted mid-write. I still don't know exactly why, could've been a crash, could've been a power blip, could've been the box just deciding it needed a nap. Doesn't matter. What matters is that a checkpoint file was partway through being written to disk when the process writing it stopped existing.

The next morning nothing looked wrong. The file was there, in the checkpoint directory, right next to all the other checkpoints from that run. Its size was in the right ballpark, a few hundred megabytes, roughly what a checkpoint that size should weigh. No error in the logs, because the thing that would have logged the error never got the chance to finish its sentence. As far as any casual glance was concerned, this was just another checkpoint.

It sat there for two days before anything tried to actually load it. An eval job picked it up to resume from, and immediately fell over with a parse error that told me almost nothing useful. My first assumption was bad code on the loading side, because why would the file itself be the problem? It existed. It had a normal name. It was the right rough weight.

Turned out the write had gotten most of the way through and then simply stopped. The header at the front of the file was intact, that's usually written early, but the tail end, the part that gets flushed last, was just gone. Not corrupted in the sense of wrong values. Missing. Like a shipment that arrives with the right box and the right packing slip, except somebody forgot to load the last few crates.

The thing that annoyed me, once I understood it, was how confidently wrong my mental model had been. I'd been treating "a file exists at the expected path with a plausible size" as a decent enough proxy for "this file is valid." It isn't. Those are two completely different claims, and my checkpointing code had never once asked the second one.

The fix, in two parts

First: stop trusting existence and size as a stand-in for validity. Now, before anything is allowed to load a checkpoint, it opens the file and checks for an actual structural marker, something that can only be there if the write finished cleanly, not just a byte count that's roughly in range. A truncated file fails that check immediately and gets flagged instead of loaded.

Second, and honestly the more important one: the write path itself. Checkpoints now get written to a temporary file first, fully flushed to disk, and only then renamed into the real checkpoint location, one atomic rename, not a stream of bytes landing directly at the final path. If the process dies mid-write, what's left behind is an orphaned temp file with an ugly name, sitting off to the side where nothing will ever mistake it for a real checkpoint. The spot where a healthy file is expected to live only ever gets a file dropped into it once that file is complete.

Neither of these is a new idea. Databases have done write-then-rename for decades, and any halfway serious package manager does the same thing when it unpacks something onto your disk. I just hadn't needed to think about it until a training job quietly handed me a half-finished file wearing a finished file's name and address.

The actual fix took maybe twenty minutes. Figuring out that a completely normal-looking file was the culprit took most of an evening.