Bronze to Platinum: Grading My Own Training Data

I built a quality ladder for training data because I got tired of reading every question-answer pair by hand before a fine-tuning run. Bronze, silver, gold, platinum. Every example that comes out of the gym's pipeline gets scored the moment it's generated, length within a reasonable band, whether the answer trails off mid-sentence, some rough measure of coherence. No human reads this pass. No model reviews it either. The scorer runs, assigns a tier, and the example either queues for training or gets set aside.

I trusted that ladder more than I should have, and the way I found out was almost an accident. I was spot-checking a batch of gold-tier examples for something unrelated, mostly out of curiosity about what the top of the pile actually looked like, and a pattern started showing up. Long answers. Evenly paragraphed. Nothing obviously broken at the sentence level. And also, once I actually sat down and read them start to finish, nothing there.

Sentences that circled back on themselves, restating a point without adding to it. Phrasing repeated with small variations, like the model had gotten caught in a loop and kept talking instead of stopping. A few answers that were grammatically complete and perfectly readable but never engaged with the actual question being asked. My heuristics missed all of it, because I had never asked them to look for it. Length within range, no truncation, ends on a full sentence: those are decent proxies for "looks like a real answer." None of them measure whether it is one.

That's the annoying part. The tier system wasn't wrong about structure. It was accurately describing text that had the shape of a good answer with nothing inside it. A chunk of garbled, repetitive, semantically empty text had sailed to the top tier because it was long enough and tidy enough to read as substantial.

I pulled a bigger sample of gold and platinum data to see how far this went. It turned out to be a small slice of the tier, though large enough to rule out a fluke, and large enough that quietly deleting the bad rows and moving on wouldn't have felt honest.

So I went back and added checks that took longer to write than expected, because they meant thinking about failure modes instead of measuring surface properties. The first flags near-duplicate answers, so a response that repeats the same clause with cosmetic swaps stops getting scored as thorough. Then there's a rough pass for templated-sounding text, the kind of answer that could be dropped into a dozen unrelated questions because it never engages with the specifics of any of them. The last one is blunter and throws more false positives than I'd like: whether the answer addresses what was asked, since a tidy, well-formatted paragraph about the wrong topic had been earning gold stars too.

None of it is elegant. The templated-text check throws false positives on legitimately terse answers that happen to share phrasing, and I've adjusted the threshold three times since. But the problem I was solving was worse than a few false positives: a top-tier label that told me nothing reliable about whether the data was worth training on.

What sticks with me isn't really about the scorer itself. It's that the label and the quality it was supposed to represent had quietly drifted apart, and the only reason I caught it was idle curiosity on a random afternoon. If I hadn't opened that batch just to look, that data would still be sitting in gold, feeding straight into training runs, and I'd have no reason to doubt any of it. The ladder still runs on every example that comes through the pipeline. I just don't take the top rung as seriously as the label wants me to.