One GPU, Many Models: Building a Fair Queue

LLM Gym runs on one machine. That's the whole premise: a single box, humming along, cycling through a small fleet of language model adapters and retraining each one as new data trickles in. For a long time I only ran one training script at a time, by hand, and watched it like a nervous parent.

Then I added more adapters, and more adapters meant more scripts, and eventually I stopped thinking carefully about whether something else already had the GPU. Every script I'd written carried the same quiet assumption: it owned the card, alone, for the whole run. I'd written each one myself, at different times, for different adapters, and never once considered that two of my own creations might show up looking for the same GPU at once.

They did. Two jobs would land within seconds of each other, both try to allocate memory as if they had the machine to themselves, and one would die with an out-of-memory error that told me nothing useful. That was the good outcome. The bad one was quieter: both jobs survived long enough to write checkpoints, and I'd come back later to an adapter that had trained on some half-overwritten mess of GPU state. Nothing crashed. The run just went bad, and I wouldn't find out until the model started producing nonsense much later.

A lease, not a lock

The fix is almost embarrassingly simple. Before any script touches the GPU, it has to ask for it first. I built a small shared lease: a file-backed marker recording who currently holds the card, when they grabbed it, and how long they're allowed to keep it. A script has to acquire the lease, do its work, and release it. If the lease is already held, it waits.

What I underestimated was the ordering. My first version just had every waiting script poll the lease and grab it the instant it opened up, which sounds reasonable until you watch what happens in practice. Whichever script polls most aggressively wins, run after run, and the adapters attached to slower, more polite scripts barely get GPU time at all. That wasn't a queue. It was just whichever process checked the lock file hardest.

So I made it first-in-first-out instead. A job registers a ticket the moment it wants the GPU, and the lease goes to whoever has been waiting longest. Loudness doesn't count anymore. It's not much code, nothing clever, but it turned an unpredictable scramble into something boring and fair, which is exactly the property I want from infrastructure I'd rather not think about at two in the morning.

The tedious part

Writing the lease took an evening. Making everything actually use it took most of a weekend, and that was the part I hadn't planned for. A queue only protects you if every single thing capable of touching the GPU goes through it, and I genuinely had no idea how many things in my own codebase qualified until I went looking.

There was a debugging script from months earlier, the kind you write once to sanity-check an adapter's output and then never delete. It loaded straight onto the GPU with no idea the lease existed. There was a preprocessing step that occasionally ran a small model for filtering and had never needed to coordinate with anything before, plus a couple of half-abandoned evaluation tools I'd genuinely forgotten I'd written.

None of these were exotic pieces of software. They were just old, predating the lease by months, and each one was a perfectly good way to reintroduce the exact collision I'd just spent a weekend fixing. Finding them meant grepping for anything that imported the GPU-facing library and then tracing every hit back to figure out whether it was a real training job or some forgotten side tool I'd meant to clean up ages ago. Tedious is the right word for it. None of it was technically hard. It just required patience, and a willingness to admit I don't actually remember everything I've written.

The queue works now, as far as I can tell. Every training run goes through the lease, in order, one at a time, and I haven't had a corrupted checkpoint since. I still occasionally find an old script I forgot existed, and each time I brace a little before checking whether it talks to the GPU directly.