The Dashboard Was Lying to Me

I built a small training platform at home, the kind of thing I tinker with most evenings, that I call the LLM Gym. It runs on a single Mac Studio, and its job is to keep fine-tuning a fleet of small, specialized language model adapters: task-specific models that get retrained as I collect more examples and find more things worth fixing. I didn’t set out to build a platform. I set out to stop three or four one-off training scripts from silently fighting each other over the same GPU, each one convinced it had the machine to itself.

The workflow is straightforward on paper. Collect training examples. Grade them and sort them onto a quality ladder: bronze, silver, gold, platinum. Train. Then run every freshly trained model through a battery of test prompts, graded by a second local model acting as an impartial referee. If the new version actually beats the current champion, head to head, on a frozen test set that never touches the training data, it gets promoted. If it doesn’t, it sits on the bench. That last part matters: the held-out set stays held out permanently, specifically so it can never leak into what the model learns from.

For a while, the numbers this whole apparatus produced were close to meaningless, and I didn’t know it. It took embarrassingly long to notice, because the setup looked disciplined from the outside: a pipeline, a scoring system, a promotion gate that only let a model through if it earned it. It just turned out none of the scores were measuring what I thought they were measuring.

Part of why it took so long to catch is that I’m the only one looking. There’s no second engineer glancing at the dashboard and asking why the referee scores look suspiciously flat, or why one field has sat empty for weeks. When you’re the one who built the trainer, the referee, and the dashboard that reads both, it’s easy to trust a number simply because you’re the one who wired it up in the first place.

Three separate bugs were stacked on top of each other, quietly compounding. The first is the simplest, and somehow the one I’m most annoyed about: the training loop only ever scored a model by its training loss. It never actually ran the model through the real test battery afterward. The one number that was supposed to represent actual quality, the number I looked at to decide whether a model was any good, was just permanently empty.

The second bug was worse, because it looked like it was working. The referee model, the one grading every answer from every candidate, had gotten silently misconfigured into the wrong mode at some point. It was scoring every single response as a flat zero, for reasons that had nothing to do with whether the answer was actually any good. A strong answer and a nonsense answer landed on the same score. I just hadn’t looked closely enough, for long enough, to notice that zero was showing up everywhere.

The third bug was the sneakiest of the three. One code path forgot to tell the referee model what task the candidate model was even supposed to be doing. So the referee would look at a genuinely correct answer, one it had no context for, and mark it wrong simply because it had no idea what counted as correct in that situation.

None of these three would have stood out from staring at any single number in isolation. Each one degraded the signal in a different way, and together they added up to a dashboard that looked functional but told me almost nothing true.

Once I found and fixed all three, I re-ran my existing models through the corrected pipeline with nothing else changed: same weights, same test set. Several of them jumped by twenty points or more overnight, with the models themselves completely unchanged. What had changed was the measurement, not the model.

The more sobering discovery came right after that. I started comparing each fine-tuned model back against its own untouched starting point, the base model before any of my fine-tuning had touched it. On several models, my training had made things actively worse, by as much as forty to sixty points on my own scoring scale. On one particular model, the one I think of as my flagship, it was a clear, large improvement. That range is a lot to sit with: the same training process producing a real win on one model and real damage on another.

That comparison is now a mandatory gate before anything ships. No model gets to replace its predecessor without first proving, on the frozen test set, that it beats its own untouched starting point, specifically, rather than just beating whatever the previous champion happened to be at the time.

Plenty of fine-tuning runs fail, and that’s fine, that’s what the promotion gate is for. For a stretch of time, though, I couldn’t tell good training from bad. I made promotion and shipping decisions off numbers that looked precise and meant nothing. A broken measurement doesn’t announce itself the way a crashed training job does. It sits there, calm and quietly wrong, until you go looking hard enough to doubt it.