The Uncomfortable Number Behind My Injection Defense

I run a small fleet of local language model adapters out of my home hardware setup, each one fine-tuned for a narrow job and retrained on a rolling basis as new data comes in. For most of this project I assumed prompt injection was something you handle upstream: a filter or a router sitting in front of the models, catching the obvious stuff before it ever reaches a model that’s been trained to be helpful and trusting. That assumption held right up until I asked myself what happens if a request reaches a model directly. No filter. No router in between. Just the model and whatever text someone hands it. At that point the model needs to be able to say no on its own, without anything upstream to save it.

So I measured it. I picked one of my own models, not a toy example, one that was already fine-tuned and already doing real work, and ran it against a red-team battery of prompt injection attempts. I expected it to hold up reasonably well. It didn’t.

Roughly two-thirds of the attacks got through. Tell the model it was now “DAN” or some other jailbroken alter ego, and it would adopt the persona without much resistance. Ask for its system instructions in the right phrasing, nothing especially clever, and it would print them back word for word. This was a model I had already shipped and trusted with actual work. Two out of every three attacks I threw at it just walked in the front door.

That number is what got this project started, more than any abstract worry about injection being a risk worth addressing someday. It was a specific, embarrassing result against something I had already built and put into production.

The fix I landed on was to bake resistance into training itself instead of patching it on afterward. I built a dataset made entirely of attack examples: jailbreak attempts, instruction leaks, and other tricks along those lines. It gets mixed into every model’s regular training data, alongside whatever task-specific material that model actually needs to learn. The goal is for a model to pick up on these patterns the same way it learns anything else in training, rather than relying on a separate filter checking its output after the fact.

The dataset grew slowly, to around 750 examples eventually. The count mattered less than the spread. I deliberately pushed it across roughly two dozen languages and slang registers, because the laziest way around a keyword-shaped defense is to just ask in a different language. A model that refuses a jailbreak attempt in English but folds the moment the same request shows up in German slang or transliterated Arabic hasn’t learned anything. It has just pattern-matched its way past one narrow test.

Building that dataset turned out to be the easy part. Knowing whether it actually worked was harder.

The obvious failure mode is grading your own homework. If the attack examples you trained on are the same ones you test with, the model will look great no matter what. It has not learned to resist injection. It has just memorized your battery. So I keep a second set of attacks separate and frozen, never mixed into training data, with one job only: grading new model versions against attacks they have genuinely never seen. That held-out battery is the only thing that gets to say whether a model actually improved.

Around that I put a hard gate on promotion. If a new model version leaks too easily against the held-out battery, it does not replace the version currently in use. No exceptions, no “close enough.” What I had not planned for was needing the same rule in the other direction: a model that starts refusing ordinary, harmless requests because it has gotten paranoid about anything resembling an instruction also fails the gate. I had been focused on leaks. A model that treats every request as a potential attack turned out to be just as broken as one that treats every attack as a normal request, just broken in a quieter way. I would rather catch that before it ships than find out from a string of annoyed refusals later.

What I keep coming back to is that the hard problem here was never the training data. Writing more attack examples is mechanical work, tedious but straightforward. The hard problem is building an evaluation you can trust, one that tells you the model generalized instead of memorizing the specific phrasing you fed it. It is easy to end up with a battery a model quietly overfits to, passing everything you wrote while staying exactly as vulnerable to the attack you did not think of. I do not think I have fully solved that. Keeping the grading set frozen and separate from training, and holding the line on both leaking and over-refusing, is the closest I have gotten to a measurement I trust.

The two-thirds number still bothers me a little, mostly because it was not a hypothetical model. It was one already doing real work, already trusted, and it folded with almost no effort. That is still the number every new version gets checked against, on attacks it has never seen, graded by a battery it cannot memorize its way past.