Building a golden dataset before you trust any eval number

It's easy to install an eval framework, point it at a handful of hand-written test cases, get a score, and treat that score as ground truth about system quality. The framework did its job correctly; the number is still close to meaningless if the dataset behind it doesn't represent what the system actually encounters in production.

The framework isn't the hard part

RAGAS and OpenAI Evals both handle the mechanics well: running a prompt against test cases, scoring outputs against expected results or rubrics, aggregating into a metric. That's the easy, well-solved part. The hard part, and the part that actually determines whether the resulting number means anything, is what goes into the dataset in the first place — five synthetic examples someone wrote in twenty minutes measure almost nothing about how the system behaves on the actual variety of inputs it sees in production.

Real failures, not just synthetic examples

The highest-value dataset entries come from actual production failures — a real user query that produced a wrong or unhelpful answer, added to the golden set with a human-verified correct answer alongside it. This does two things a synthetic-only dataset can't: it grounds the eval in inputs the system genuinely receives, phrased the way real users actually phrase things, not the way an engineer imagines they would; and it turns every production bug into a permanent regression test, so a fix doesn't silently regress the next time someone touches the prompt.

Synthetic examples still have a role — covering edge cases that haven't happened yet in production but plausibly will, or stress-testing categories you have hypotheses about even without real-world evidence yet. But a dataset that's entirely synthetic and never incorporates real failures is testing the system against an engineer's mental model of failure modes, not the system's actual ones — and those two things reliably diverge.

Disagreement in labeling is a signal

When two people labeling the same golden-set example disagree about whether an output is correct, the instinct is often to have a tie-breaker resolve it and move on. That disagreement is worth treating as information about the task itself, not just noise to average away — it usually means the correctness criteria weren't as clear as they seemed, which is exactly the kind of ambiguity that will also confuse the model being evaluated. Examples with high labeler disagreement are frequently the most informative ones in the whole set, because they're marking a genuine boundary case, not a mistake in the labeling process.

The dataset needs to be rerun, not just built once

A golden dataset built once and never revisited slowly stops representing the system, because the system keeps changing — new prompt versions, model upgrades, new features that introduce new input types the original dataset never covered. The dataset needs to be rerun on every meaningful change (new model version, prompt rewrite, new retrieval strategy) — not just to confirm forward progress, but specifically to catch regressions the change wasn't intended to cause. A score that improves on the metric the change targeted while silently regressing on an untouched category is a common failure mode, and it's only visible if the full golden set gets rerun, not just the subset directly related to what changed.

The uncomfortable but useful reframe: an eval framework tells you how well the model does against the dataset you gave it. If that's the actual question you needed answered, the framework choice barely matters. If the dataset doesn't represent reality, no framework can fix that for you.