Disk Fills Up Eventually: Building an Offload System

Every fine-tuning run on my home setup leaves a trail behind it. There's a raw checkpoint from partway through training, and a couple of converted, quantized files for actually serving the result. At the very end sits the part I actually wanted: a small adapter file, tiny next to everything that produced it. Everything upstream of that adapter is disposable. I just didn't treat it that way for a long stretch of running this thing.

The LLM Gym is what I call the fleet of small models I keep fine-tuning on hardware sitting in my house. It's not one model, it's a rotating cast of adapters, each specialized for a narrow task, each one the product of a training run that eats disk space the way a wood chipper eats branches. Run enough of those on a rolling basis and you get an arithmetic problem that isn't subtle: the drive is finite, and the runs keep coming.

For a while my plan was to notice the disk was filling up and go delete things by hand, usually while half paying attention to something else. That plan works exactly as well as you'd expect from something that only happens when I remember to do it, which is inconsistently. More than once a training job has failed partway through with a disk-space error, and I've spent the next while trying to figure out, under mild panic, which of the leftover directories are safe to remove.

So I built a watchdog instead. It's a small background process that walks the training output directories, checks the status of each model, and for anything that's been fully trained, evaluated, and deployed, decides what to do with the large intermediate files still sitting next to it.

The decision, most of the time, is to delete it outright. Archiving to slower storage is available for smaller files, but the big checkpoints and quantized exports mostly just go. That took me a while to be okay with, deleting training output feels like burning your notes. The reasoning the whole system leans on is that the huge intermediate files are fully reconstructable from two things I keep forever: the adapter weights and the base model. If I ever need one of those files again, I can regenerate it. Keeping it around permanently means paying disk cost every day for something that was never at risk of being lost.

That logic holds up fine when I'm explaining it to myself. Writing code that acts on it without asking me first is a different feeling entirely. Give a script delete permissions and one bad assumption, and you find out the hard way what "regenerable" meant. So the watchdog has a short, strict list of things it will never touch: the adapter weights themselves, the record of which adapter is currently the champion for its task, and anything it can't positively confirm has a completed evaluation and a deploy marker behind it. No confirmation, no deletion, the file just sits there, untouched, however old or however large.

I still read through its logs most mornings, mostly out of habit at this point rather than distrust. It's freed up noticeably more space than I ever managed by clicking around manually, and I haven't yet needed one of the deleted files back. I assume that streak ends at some point, and when it does I'll regenerate whatever I need and lose an afternoon to it, which still beats losing a weekend to babysitting a nearly full drive.