From demo to production: the LLM evaluation harness

An LLM feature that demos well is not a production system. What closes the gap is an evaluation harness — a graded dataset you run on every change, plus scored observability on live traffic — because when outputs are non-deterministic, "it looked right in the demo" is not a test.

The problem

Demos are curated. You pick the inputs, you retry the bad generations, and you show the run that worked. Production does the opposite: it feeds the model adversarial, ambiguous, and out-of-distribution inputs you never thought to try, at a volume where the rare failure is a daily occurrence.

The instinct is to reach for the testing you already know, and it doesn't fit. LLM outputs are non-deterministic and multi-form valid — two different strings can both be correct answers — so exact-match assertions either flake (any valid rewording turns the suite red) or get loosened until they pass trivially and catch nothing. Manual spot-checks don't scale past a handful of cases and can't see a silent regression: a prompt tweak that lifts one class of inputs while quietly breaking another. Without measurement, that trade is invisible, and you ship it.

What we did

The decision that matters is eval-as-a-gate over vibes-based QA. We build the feature around a harness with two loops instead of bolting testing on at the end.

Offline, in CI. A versioned dataset of representative and adversarial cases, each scored by graders chosen to fit the question: deterministic checks where the answer has structure (valid JSON, contains a citation, no leaked PII), and an LLM-as-judge with an explicit rubric where the answer is open-ended. It runs on every prompt, model, and retrieval change; a drop in the aggregate score blocks the merge the same way a failing unit test would.

Online, in production. Every call emits a trace — input, retrieved context, output, tool calls, latency, cost. A sampled slice is graded by the same rubric, so quality is a monitored metric with alerts, not an anecdote someone noticed in Slack. Drift and regressions page you before a customer files the ticket.

The alternative we explicitly rejected was exact-match golden outputs — snapshot the "right" answer, assert string identity. It's the wrong assertion for this domain: non-determinism plus semantic equivalence means it produces either flaky red or meaningless green. We assert on properties and graded quality instead of byte-for-byte identity. This is the evaluation and observability work that turns an impressive demo into a system you can put in front of customers — and it's the same discipline behind the systems we've shipped.

The result

The artifact is the grading model itself: match the grader to what the output actually guarantees, rather than forcing everything through one assertion style.

• Structural — deterministic checks (schema valid, citation present, no leaked PII). Catches malformed output, missing fields, and policy violations.
• Reference-based — similarity or overlap against a known-good answer. Catches drift away from a factual gold answer.
• Open-ended — LLM-as-judge against an explicit rubric. Catches tone, completeness, and reasoning quality where no single string is "correct."
• Production — sampled live traces scored by that same rubric. Catches real-world regressions, drift, and the long-tail failures your dataset never anticipated.

The first three layers gate the change before it merges; the production layer confirms the change behaves the same on traffic you didn't anticipate. Together they replace "did that feel better?" with a number you can trend.

Takeaway

If you can't measure whether a change made your LLM feature better or worse, you aren't iterating — you're guessing with confidence. Build the evaluation harness before you scale the feature, not after the first production incident.