Frozen Test Sets and Why They Have to Stay Frozen

I keep a held-out test set for every adapter that comes out of my home training rig. Before I call a model finished, it runs against a batch of examples it never saw during fine-tuning, and that score is the only number I actually trust. Training loss lies to you constantly. A held-out eval, done properly, mostly doesn't.

The "done properly" part is where I keep getting nervous.

The whole premise of a frozen test set depends on it staying frozen. The moment one example from that set turns up in a training batch, the score it produces stops telling you anything real. The model has stopped demonstrating a skill. It has started reciting an answer it was shown during fine-tuning, and the number that comes out looks fine and means nothing.

What surprised me, once I started worrying about this in earnest, is how many ordinary paths lead there. None of them look like a mistake while they're happening.

The one I think about most is re-collection. A data source I scraped or exported months ago gets touched again, maybe I'm refreshing a dataset, maybe the source itself changes and I pull a fresh snapshot. If that source overlaps with wherever my eval examples originally came from, there's a real chance the new pull brings one of them back in, verbatim, with nothing anywhere flagging that this one has already been seen and is supposed to stay out. The ingestion doesn't look wrong at any step. The data is coming from a place it's allowed to come from, right on schedule.

The other path is more embarrassing, because it's just me. I'll be cleaning up a training set and spot an example that's unusually well-formed, and I'll think, great example, why isn't this in the training data already, and drop it in. Sometimes it's well-formed because I picked it for the eval set for exactly that reason, months earlier, and forgot.

Neither of these feels like negligence at the time. They feel like normal upkeep. There's no single reckless action to point at afterward, just an ordinary task that had a small, unadvertised chance of quietly poisoning the one number I rely on for anything.

What actually works

I stopped trying to solve this by being careful, because careful doesn't survive several months and a handful of dataset refreshes. What I do now is boring and mechanical, which is exactly why I trust it more than I trust myself.

Every example in the frozen eval set gets hashed the moment it's frozen. Full text, normalized first, so a stray line break or a trailing space can't sneak a duplicate past the check. That hash goes into a lookup table that lives right next to the eval file, not buried three directories deep in a config I'll forget exists in a year.

Then, every time anything gets merged into a training batch, it gets hashed the same way and checked against that table before it's allowed anywhere near the training run. No manual review step, no "I'll remember to check this one." If the hash matches, the example gets dropped and logged, and training carries on without it.

Hashing to catch duplicates is a decades-old trick, nothing I invented and nothing clever. What took me longer than I'd like to admit was realizing my own memory was never going to do this job, no matter how carefully I'd curated the original eval set. The check runs the same way whether a leak comes in through a fresh scrape or through me, at eleven at night, convinced I've just found a great training example.

Most of these near-misses never show up as a dramatic failure. They just sit there quietly, making one score look a little better than it should, until somebody happens to check. A broken eval doesn't announce itself. It keeps returning a clean number, month after month, and you go on believing it.