From prompts to production: making AI systems measurable

A prompt that works once, in a notebook, against the one example you happened to test it on, is not a system — it's a demo. The gap between those two things is bigger than it looks from the notebook, and closing it is mostly what "productionizing" an AI feature actually means.

The gap that doesn't show up in a notebook

A prompt tested against three or four hand-picked examples looks solid because those examples were, consciously or not, selected to be the kind of input the prompt was designed to handle. Production traffic doesn't select itself that way — it includes the edge cases, the ambiguous phrasings, the inputs nobody thought to test, at whatever volume the feature actually gets. A prompt that's 95% right on three examples might be 95% right on three thousand too, or it might not be — the notebook genuinely cannot tell you which, because three examples isn't a sample, it's an anecdote.

Measurement instead of vibes-based iteration

The common failure mode after that realization isn't giving up on iterating the prompt, it's iterating on vibes: tweak the wording, run it against the same three examples again, decide it "feels" better, ship it. This is iteration without a way to know whether it's actually improving anything beyond the tiny set being eyeballed. The fix is unglamorous and specific: build an actual dataset of real or representative inputs, score the prompt against all of them with a consistent rubric, and treat that score — not a feeling — as the thing a change either improves or doesn't. This is more setup than "just try a different wording," and it's the only way to know whether a prompt change is actually progress instead of a coincidence that happened to look right on the one example someone glanced at after making it.

What a regression suite for prompts looks like

Once there's a scored dataset, the natural next step is treating it like a test suite: every prompt change gets run against the full set before shipping, not just the handful of cases the change was specifically targeting. This catches the failure mode that's otherwise invisible — a change that fixes the exact case it was meant to fix while quietly breaking three other cases that weren't being watched. Without a regression suite, that kind of regression ships silently and shows up later as a vague sense that "the AI feature got worse" with no clear cause, because nothing was tracking the cases that broke.

A useful regression suite isn't static either — every real production failure that gets found is worth adding to it as a new case, the same way a bug fix in normal software gets a test that guards against it recurring. Over time the suite becomes a genuine record of every failure mode the system has actually hit, not just the ones an engineer happened to think of in advance.

Deciding when a change is safe to ship

The endpoint this whole process is building toward is a real answer to a question that's otherwise just a guess: is this prompt change safe to ship without a human manually re-checking every case first. Without measurement, that's a judgment call based on how confident the change feels. With a scored regression suite, it's an actual answer — did the score hold or improve across the full dataset, did any previously-passing case start failing. That's the concrete difference between prompting as an ad hoc activity and prompting as an engineering practice: not that the prompts are necessarily better, but that whether they're better is something you can actually know, instead of something you're hoping.