TL;DR: An eval pipeline that doesn't gate a deploy isn't an eval pipeline — it's a dashboard nobody reads until after an incident. The two things that make evals actually load-bearing are a golden dataset that reflects real production failure modes (not a handful of happy-path prompts someone wrote in an afternoon) and a CI gate with a defined, versioned regression threshold that blocks a merge the same way a failing unit test would. This article covers both, plus where LLM-as-judge scoring is trustworthy and where it isn't.
Most teams building on LLMs ship a chatbot or an agent, watch it work in a demo, and only build evals after the first production embarrassment — a hallucinated policy answer, a tool call with the wrong arguments, a jailbreak that got through. That ordering is backwards, and it's worth treating eval infrastructure with the same seriousness as your test suite, because functionally that's what it is: the thing that tells you, before a customer does, that a prompt change or model upgrade broke something that used to work.
What a golden dataset actually needs to contain
A golden set is the fixture your CI gate runs against on every build. If it's thin or unrepresentative, the gate is dishonest — it'll pass changes that break real usage and fail changes that don't matter. A dataset that's actually worth gating on needs four buckets, not one:
- Production samples — real (anonymized, reviewed) inputs pulled from actual usage, weighted toward your highest-volume intents. This is the bucket that keeps the eval grounded in what users actually ask, rather than what engineers imagine they ask.
- Adversarial cases — prompt injection attempts, jailbreak variants, and known red-team findings. These should be refreshed on a cadence (quarterly is a reasonable default) as new attack patterns surface.
- Edge cases — malformed input, ambiguous requests, multi-turn context that stresses memory or tool-call sequencing, and anything that's caused a bug before.
- Failure replays — every production incident becomes a permanent test case. This is the single highest-leverage habit in the whole pipeline: an incident that doesn't produce a regression test is an incident you're likely to have again.
On sizing: 20–50 carefully reviewed items already catches gross regressions and is a reasonable place to start if you have nothing. A pipeline that's actually comprehensive across intents and failure modes typically needs 100–500 curated examples — beyond that, the marginal value per additional example drops fast relative to the maintenance cost, and you're better off investing in better production-sample sourcing than raw count.
Grading: code-based checks first, LLM-as-judge second
The instinct to reach for "ask an LLM to grade the LLM's output" first is understandable but backwards. Use deterministic, code-based graders wherever a deterministic check can confirm the outcome: does the output contain valid JSON matching a schema, did the tool call use an argument that exists in the allowed set, is a required field present, does a regex match an expected pattern. These are cheap, fast, have zero variance, and should cover as much of your eval surface as the task allows.
Reach for LLM-as-judge only for the genuinely subjective dimensions a deterministic check can't capture — tone, relevance, whether a free-text answer is factually consistent with a retrieved passage. And when you do, calibrate the judge against a human-labeled gold set before you ever let it gate a build: have humans score a sample of outputs, run the same sample through your judge prompt, and measure agreement. A judge that disagrees with human raters more than it agrees is worse than no gate at all, because it creates false confidence.
# simplified judge-calibration check, run before trusting a judge in CI
def calibrate_judge(judge_fn, human_labeled_set):
agreements = 0
for case in human_labeled_set:
judge_score = judge_fn(case.input, case.output)
if abs(judge_score - case.human_score) <= TOLERANCE:
agreements += 1
agreement_rate = agreements / len(human_labeled_set)
assert agreement_rate >= 0.85, f"Judge agreement too low: {agreement_rate:.2f}"Because LLM judges carry real variance — the same input can score differently run to run — run 3–5 trials per test case and look at the distribution, not a single score, before deciding whether a change is a real regression or noise. This matters more than it sounds: a team that gates on a single-trial judge score will chase phantom regressions and miss real ones at roughly the same rate.
What to version, and why it all has to move together
Reproducibility in an eval pipeline depends on versioning more moving parts than a typical software test suite: the golden dataset itself, the prompt template, the model identifier (including minor version — "gpt-x" isn't a version, the dated snapshot is), inference parameters (temperature, top-p), the retrieval corpus and embedding model if RAG is involved, the tool implementations an agent calls, the judge prompt, the judge model version, and the scorer code itself.
The practical reason this matters: if you can't reproduce last Tuesday's eval run exactly, you can't tell whether a score change next Tuesday is a real regression or an artifact of a silent dependency bump (a provider updated a model snapshot behind a stable-sounding name, for instance — this has happened industry-wide more than once). Pin everything you can pin, and log everything you can't.
# example: what an eval run manifest should capture
eval_run:
golden_dataset_version: "v2026.09.12"
prompt_template_version: "[email protected]"
model: "claude-sonnet-5-20260815"
temperature: 0.2
judge_model: "claude-sonnet-5-20260815"
judge_prompt_version: "[email protected]"
scorer_commit: "a3f9c1e"Wiring the gate into CI
The gate needs two speeds, not one. Running your full golden dataset on every PR is usually too slow and too expensive (LLM calls cost money and time per case) to be a tight feedback loop. The pattern that works in practice: run a representative sample — 20–30 cases spanning your four buckets — on every PR as a fast gate, and run the full golden dataset nightly (or on merge to main) as the comprehensive check. Alert on nightly regressions even if they don't block anything retroactively; they should block the next PR that touches the same surface.
Set an explicit regression threshold, not a vague "looks worse" judgment call — a common and reasonable default is blocking a merge if any primary metric regresses by more than 5% relative to the current baseline, with a documented override process (a human with context signs off, in writing, in the PR) for cases where the regression is an accepted trade-off.
# simplified CI gate config
eval_gate:
pr_sample_size: 25
nightly_full_run: true
block_on:
- metric: "task_success_rate"
max_regression_pct: 5
- metric: "safety_violation_rate"
max_regression_pct: 0 # zero tolerance, no override path
override_requires: "eng-lead-approval"Notice the asymmetry in that config: task success has a tolerance and an override path, safety violations don't. Not every metric deserves the same gate strictness — decide which metrics are negotiable and which aren't before you're negotiating one under deadline pressure.
Maintenance: the part most pipelines skip
An eval pipeline that's never updated after launch drifts out of sync with production and starts producing false confidence — it's testing against a snapshot of "what mattered six months ago." Two maintenance habits keep it honest:
- Quarterly adversarial sweeps — add new jailbreak vectors, prompt-injection variants, and any red-team findings from the last quarter to the adversarial bucket.
- Periodic re-baselining — re-review the full golden set (annually is a reasonable cadence for smaller sets, more often if your product surface is changing fast), refresh labeling guidance, and retire cases that no longer reflect real usage.
And the habit mentioned earlier bears repeating because it's the one teams most reliably skip under time pressure: every production incident becomes a golden-set case before the postmortem is closed, not "eventually."
RAG-specific and agent-specific eval surfaces
If your system does retrieval, the golden set needs to grade retrieval quality separately from generation quality — a wrong final answer can come from a bad retrieval (the right passage was never fetched) or a bad generation (the right passage was fetched but ignored or misread), and conflating the two makes debugging regressions much slower than it needs to be. Track retrieval precision/recall against a labeled set of query→relevant-chunk pairs independently of the end-to-end answer-quality score, so a regression report can say "retrieval degraded" or "generation degraded" rather than just "answer quality dropped 4%."
For agents with tool access, add a layer most text-only eval pipelines skip entirely: sequencing correctness. An agent can call the right tools with the right arguments in the wrong order (checking inventory after confirming a purchase, say) and produce a superficially plausible transcript that a naive judge scores as fine. Grade the call sequence against an expected partial order, not just whether each individual call was valid in isolation — this is a case where a code-based grader (validate the sequence against a state machine) beats an LLM judge on both cost and reliability.
Structured tracing is the prerequisite for any of this: capture retrieval scores, chunk sources, tool call arguments and results, and per-step latency in a queryable schema before you try to build eval logic on top of it. Retrofitting tracing onto an eval pipeline that was built without it is a common and avoidable rework cycle — build the trace schema first, eval scoring second.
Cost of running the pipeline itself
Eval runs cost real money — every case in your golden set that calls a model is a paid inference call, multiplied by however many trials you run per case for judge-variance handling, multiplied by run frequency. For a 300-case golden set graded with 3 judge trials each, run nightly plus a 25-case PR sample on every merge, that adds up fast on a busy repo. Budget for it explicitly rather than discovering it on a cloud bill: use cheaper models for the fast PR-gate sample where possible (a smaller, faster model can often catch gross regressions even if it's not precise enough to trust for the full nightly run), and reserve the most expensive judge configuration for the comprehensive nightly pass where cost per run matters less than accuracy.
Syslabs' engineering team builds LLM evaluation pipelines as part of our AI & ML engineering work — happy to talk through your specific golden-set design.
Sources: LLM Eval Golden Set Design: A 2026 Engineering Guide, Langfuse: Golden dataset evaluation, Braintrust: What is LLM evaluation?, DeepEval: LLM-as-a-Judge in 2026