Block 06 / 07
Iteration requires a measurement procedure.
Evidence that a system works is a measurement procedure over frozen cases. Freeze inputs. Name properties that must hold: schema, citation, refusal, absence of invented APIs, latency, cost. Score them with code, a cheaper model, a human rater, or all three. Pass rate on that suite is the regression test.
Offline evals are CI. Benchmarks rank a base model on a public task. Human eval catches taste and harm. Online evals and A/B catch drift. Property tests travel. Golden essays overfit. The harness is where requirements live.
Version 2 of 4 · regression: a prompt change broke citation formatting, CI would fail this PR
Drag the slider to step through CI runs. Watch which property breaks, and whether the PR would pass.
A measurement procedure needs some way to turn model behavior into a number or a verdict. The three below are the concrete mechanics behind that judgment, and they compose: most real harnesses use more than one.
Reference-based
Code checks a concrete property directly: does the output parse against a schema, does the cited source actually exist, did the model refuse when it should have. Cheap, deterministic, and blind to anything it was not written to check for.
Comparative
Judgments, from a person or a cheaper model, compare two outputs and say which is better. Elo, the same rating system chess uses, or a Bradley-Terry fit turns a stream of those pairwise comparisons into a single ranking, the procedure behind Chatbot Arena (Zheng et al., 2023).
Statistical
Two variants split live traffic, and a significance test on the resulting metric decides whether the gap is real or noise. A sequential test, or a bandit algorithm that shifts traffic toward the better arm as evidence accumulates, reaches that answer without waiting on a fixed sample size.
Paper
A public argument that a single aggregate score is insufficient. Complements Zheng et al. on LLM-as-a-judge if the grader itself must be examined.
Book
Offline versus online evaluation, what to log, and how evaluation sits next to product metrics. Pair with Hamel Husain’s writing on evals as the product.
Course
A short on-ramp for owning a harness. After the course, freeze fixtures for the path under study.
A system is known to work only through a measurement procedure. The following are that procedure in different settings.
A frozen set of cases with scorers for schema, citation, refusal, or invented APIs. Pass rate on that suite is the test that a prompt or model change must survive.
Properties that code cannot score are rated by a person or by a cheaper model. The grader is itself a source of error and has to be checked.
A/B and production logs catch drift that a frozen fixture will miss. An incident that was not promoted into the suite will recur.
Freeze a JSONL of cases for the riskiest path you own. Scorers for schema, citation or refusal, and invented-API detection. Fail the PR if pass rate drops. Break a prompt on purpose and watch CI go red. Promote one real incident into a fixture.