There's a moment on every LLM project where someone asks the question that decides whether the feature will survive: "if we change this, how do we know we didn't break something?"
Classic software answered that question decades ago. You write tests. LLM features somehow escaped the discipline. Teams that would never merge an untested API endpoint will happily rewrite a production prompt, eyeball three outputs, and ship. Then a customer finds the fourth output.
This post is the playbook we use to close that gap, and it is week one of the prototype-to-production path we've written about before. None of it is exotic, and that's kind of the point.
"It feels better" is not a metric
The failure pattern is always the same. Someone reports a bad output. An engineer adjusts the prompt until that case looks fixed, checks a couple of others, and merges. Three days later a different case, one that used to work, comes back broken. Nobody notices for a week because nobody was looking.
This isn't carelessness. It's what happens when there is no instrument for measuring change. LLM outputs are high-dimensional and probabilistic, which is exactly the kind of behavior human spot-checking is worst at. The team isn't missing talent. They're missing a harness.
The moment you have one, the conversation changes shape. "I think this prompt reads better" becomes "this change moves grounding accuracy from 84% to 91% and breaks nothing." One of those is an opinion. The other is an engineering decision.
Build the suite from traffic, not imagination
The biggest mistake teams make is writing eval cases the way they wrote the demo: from imagination. Invented inputs inherit the same blind spots as the prompt they're supposed to test. The suite that actually protects you is built from what users really send:
- Start with 50 to 100 real cases. Pull them from logs. Twenty representative failures are worth more than a thousand synthetic successes, because you're building a net for the inputs that actually occur, not the ones you can think of.
- Over-sample the ugly ones. Truncated pastes, mixed languages, hostile tone, 40-page documents, questions that are out of scope on purpose. Production traffic is mostly edge cases anyway.
- Write down the expected behavior, not the expected text. For most features you don't want "the output equals this string." You want "the answer cites only the provided document," "the JSON validates against the schema," "the refusal routes to a human." Behaviors stay stable across model versions. Exact strings don't.
- Version the suite next to the code. Cases live in the repo, run in CI, and change through review, the same lifecycle as any other test. A suite in a spreadsheet is a suite nobody runs.
Score with code first, judges second
Scoring is where teams overcomplicate things. Half of what you care about can be checked with plain assertions, and deterministic checks should always come first: schema validity, required fields, length budgets, banned content, whether every quoted fact appears in the source. Cheap, fast, zero ambiguity.
The other half, things like tone, helpfulness, and whether an answer actually resolves the question, needs a judge model scoring against a written rubric. Two things make judge scoring trustworthy rather than circular:
- The rubric is written, specific, and versioned. "Score 1 to 5 for helpfulness" produces noise. "Fails if the answer recommends an action the document warns against" produces a signal you can debug.
- You audit the judge. Sample its verdicts weekly against a human read. When the judge and the human disagree more than about 5% of the time, fix the rubric before you trust another run.
The cost objection mostly dissolves once you do the arithmetic. A 100-case suite scored by a small judge model costs cents per run. Your engineers' time spent manually eyeballing outputs costs more per minute.
The suite is what makes everything else safe
Here's the part that earns the title of this post. Once the suite exists, it quietly becomes the most valuable AI asset the team owns, because it's the thing that makes every other decision reversible:
- Model upgrades become diffs, not leaps of faith. A new model version drops? Run the suite. If the score holds, migrate. If it regresses, you know within minutes, with the failing cases attached.
- Vendor lock-in loosens. Your quality bar is encoded in your cases and your rubrics, not in a particular provider's behavior. Swapping providers becomes an afternoon of eval runs instead of a quarter of re-validation.
- Incidents stop repeating. Every production failure becomes a new case, the way a bug becomes a regression test. The suite grows a memory of everything that has ever gone wrong. It catches what you change; production monitoring, the kind of anomaly feed we built into WhyLabs Observatory, catches what changes on its own.
- New engineers ship safely in week one. They don't need tribal knowledge of the prompt's ancient history. The suite enforces it for them.
Models depreciate, providers change pricing, prompts get rewritten. The eval suite is the part that compounds. That's why it's the product.
Where to start on Monday
The whole thing sounds heavier than it is. The first working version is a week of disciplined work, and the sequence matters more than the tooling:
- Day one: write the spec. Ten to twenty sentences. What must this feature always do, never do, and how would a careful human judge an output? Most teams discover halfway down the list that they disagree, which is the cheapest possible place to find out.
- Days two and three: collect the cases. Pull real inputs from logs, tag them by scenario, over-sample failures. Aim for coverage of behaviors, not volume.
- Day four: wire the scoring. Assertions for everything mechanical, a rubric-driven judge for the rest, one command that prints a score.
- Day five: put it in CI and get a baseline. From this day forward, no prompt or model change merges without the number moving in the right direction.
That's the whole discipline. It's also, not by coincidence, the first week of how we run LLM engagements at Nortik. Everything else we build on top of a feature is only safe if this harness is under it.
