The Judge Model Problem — Who Grades the Grader

I run a small rig at home that fine-tunes a fleet of adapters on top of a base language model. A few GPUs, a queue of candidate checkpoints waiting to get promoted or thrown out, nothing exotic. The part that actually kept me up at night wasn't the training. It was the grading.

Every candidate answer gets scored by a second model before it's allowed anywhere near a real evaluation set. I call it the judge. It reads a prompt, reads the candidate's response, and hands back a score. Clear some threshold and the candidate advances. Miss it and the run gets logged as a failure and the next checkpoint takes its turn. Simple enough on paper.

Except the judge is a language model too. It doesn't get a pass on the failure modes it's supposed to be checking for in everyone else. Somewhere in the middle of wiring this up I had to stop and admit I was trusting one model's opinion of another model's output, and I hadn't done a single thing to earn that trust.

The first crack showed up almost by accident. I fed the judge the same candidate answer three times in a row, mostly out of curiosity, and got three different scores back. The gap was small, a point or so on a ten-point scale, yet enough that a threshold of 7 would pass on one run and fail on the next for the identical input. At that point the number coming out of the judge wasn't measuring the candidate at all. It was measuring whatever the judge happened to feel like that particular pass.

The cause turned out to be embarrassingly mundane: I'd never pinned down the judge's sampling settings. Default temperature, no fixed seed, and a free-text output that I was then regex-ing a number out of, hoping the format held. So I locked it all down. Fixed temperature and seed, and a structured output format where the judge has to return its score in a defined shape instead of a paragraph I'm hoping contains a digit somewhere. Same input now produces the same score, every single time. That didn't make the judge correct. It just made it consistent, which I'd somehow skipped past as a precondition for correct.

Consistent is not the same as sane

A deterministic judge can still be a bad judge, just a reliably bad one. It could score every single answer a 2 out of 10, or every answer a 9, and nothing in the training logs would tell you that, the run would just look uniformly terrible or uniformly great, and you'd have no way to know if that's the candidates talking or the judge. A whole overnight training run can burn its GPU hours promoting garbage because the grader quietly decided garbage is fine, and it'll do it with total confidence.

So before the judge is allowed anywhere near a real batch, it has to clear two planted questions first: one answer I've written to be obviously excellent, and one I've written to be obviously bad, wrong facts, broken formatting, the sort of thing that shouldn't survive contact with any reasonable grader. If the judge doesn't put real distance between those two scores, I don't trust anything else it says that session, and I go looking for what changed. It costs maybe thirty seconds of compute. It's also caught real drift more than once, usually after I'd tweaked a prompt template and hadn't noticed it quietly shifted the judge's whole sense of scale.

None of this actually resolves the deeper problem. I'm still using a model to decide whether another model did well, and somewhere that chain has to end with me reading outputs myself and using my own judgment, just on a sample now instead of on everything. The determinism and the anchor questions don't close that loop. They just make sure that when I do sit down and check by hand, the automated scores I'm comparing against are measuring something real instead of noise wearing a number.