Continuous Training Without Losing Your Mind

I have a bad habit with long-running jobs. Doesn't matter if it's a backup, a render, or a training run, I check on it. Checking doesn't do anything. The job runs at the same speed whether I'm watching a log scroll by or asleep on the couch. But something in me needs to open the terminal every twenty minutes anyway, just to confirm the universe hasn't quietly broken while I wasn't looking.

This became a real problem once I set up what I've been calling the LLM Gym, a small home-hardware setup that trains a fleet of small language model adapters, cycling through them on a schedule instead of me kicking off each run by hand. The whole point was to stop babysitting training jobs. What I actually built for the first few weeks was a very effective way to babysit them from my phone instead.

I checked constantly. One adapter finishes a cycle, I check the loss curve. Another is halfway through, I check anyway, like staring at an incomplete graph would tell me something the finished one wouldn't. None of it changed the outcome. It just meant my evenings had a low hum of obligation running under everything else, the sense that I should glance at the dashboard one more time before bed.

The checking was never really about the models. It was about not trusting the system to tell me when something had gone wrong. If I didn't look and a run had quietly failed six hours earlier, I'd find out at midnight, annoyed at myself for not catching it sooner. So I looked, because looking felt like insurance, even though it insured against nothing at all.

What actually fixed it

Two things, and only one of them was code.

The first was adaptive pacing. Originally every adapter retrained on the same fixed interval no matter how it did, which meant I was burning GPU hours retraining a model that had just gotten measurably worse, right alongside one that had just posted its best run in weeks. Now the gap before a model comes up for retraining stretches out after a weak or regressive result. There's no point hammering the hardware on something that just failed; give it room, let more data accumulate, try again later. A strong run shortens the interval instead, so a model on a good streak gets another shot sooner rather than waiting its turn behind everything else. The schedule reacts to how each model has been performing instead of ticking along on a timer.

The second was notifications I could trust. Something that would reliably flag a bad regression or a dead process, instead of a wall of logs I had to go interpret myself. Once I trusted the silence, not checking stopped being a gamble. I stopped opening the dashboard because there was nothing left in it that a notification wouldn't have told me first, and usually faster.

There was a third variable early on I didn't expect: quiet hours. For a while training had to pause at fixed times of day because something else needed the same GPU, and I hadn't worked out how to share it gracefully yet. So I just blocked off windows, no training between certain hours, full stop. It worked, but it meant the pacing system was reasoning about model quality inside a cage that had nothing to do with quality. A model could be on a great streak, earning a shorter retrain interval, and then hit a wall because the clock said no.

Once that other workload stopped competing for the hardware, I dropped the quiet hours entirely, and the pacing math got noticeably simpler. The only thing left deciding when a model retrained was how it had performed, not what time it happened to be. I hadn't clocked how much those fixed windows were quietly distorting the schedule until they were gone.

I still open the dashboard sometimes. Old habits don't die just because you've proven them pointless. Mostly it's out of curiosity now instead of compulsion, which is a smaller thing than it sounds like, but it's the thing that got my evenings back.